Design Python-based Databricks pipelines for ingestion, transformation, enrichment, validation and publication of DER/VPP data.
Build scalable batch and near-real-time workloads using Spark, Delta Lake, notebooks, jobs and workflows.
Implement data-quality, reconciliation, lineage, auditability and observability controls.
Optimise partitioning, throughput, query performance, reliability, maintainability and cloud cost.
Support feature engineering and productionisation of telemetry, dispatch and platform analytics.
Deliver Python packages, tests, validation checks, workflow definitions and dashboard-ready datasets.
Must have skills:
Python; Databricks; PySpark/Apache Spark; Delta Lake; SQL; ETL/ELT; batch and streaming pipelines; data quality; lineage; performance tuning; Azure; Git/CI/CD; PostgreSQL
Pay: From $120,000.00 per year
Work Location: Hybrid remote in Perth WA
📌 Python / Databricks Engineer Perth
🏢 Duxxel
📍 Perth