We are seeking a
Senior Data Engineer (Databricks, Data Modeling)
.
You will model large-scale batch and streaming data into governed Bronze, Silver and Gold layers to support regulatory reporting data products.
You will build on Databricks using Spark, Python and SQL, partner with stakeholders on definitions and use AI-assisted tools to speed up delivery, testing and documentation.
Responsibilities
Design trusted Bronze, Silver and Gold layers using medallion architecture principles
Model large-scale batch and streaming datasets into reusable, governed data models for reporting use cases
Build Databricks pipelines using Delta Lake, Delta Live Tables or Lakeflow aligned to quality and lineage needs
Develop Spark workloads using PySpark and Spark SQL for performance, reliability and maintainability
Implement streaming ingestion and processing using Spark Structured Streaming and Kafka or similar
Partner with business and regulatory stakeholders to translate reporting needs into data products and model definitions
Strengthen governance with cataloging, lineage and data quality checks using Unity Catalog or equivalent patterns
Contribute to CI/CD for data pipelines, Git-based workflows,



testing and release practices
Requirements
Data modeling background across conceptual, logical and physical models using dimensional, Data Vault or similar approaches
Databricks experience, including Delta Lake plus governance concepts such as Unity Catalog
Apache Spark skill sets using PySpark and Spark SQL in production environments
Streaming data experience using Spark Structured Streaming, Kafka or similar technologies
Python capability for pipeline development and orchestration patterns
Advanced SQL skill sets for transformation logic, validation and performance tuning
Data governance experience covering quality, lineage, controls and well-defined data contracts
AI-assisted development tooling exposure, with Claude preferred, to accelerate build, testing and documentation
Nice to have
Regulatory reporting data exposure such as EMIR, MiFIR or ASIC derivative reporting
Data mesh experience delivering domain-owned data products
Data Vault 2.0 experience in large-scale enterprise models
Apache Kafka experience and solid understanding of streaming platform operations

📌 Senior Data Engineer (Databricks, Data Modeling) (New South Wales)
🏢 EPAM Systems
📍 New South Wales