Senior Databricks Engineer
12-month contract
$1000 + super
Australian Citizen
The opportunity
A large, complex organisation is undertaking a significant modernisation of its enterprise data platform and data governance architecture. As part of this transformation, they are seeking an experienced Senior Databricks Data Engineer to lead the development, implementation and operationalisation of a modern Azure Databricks lakehouse environment.
This is a senior hands-on engineering position combining Databricks development, data engineering, dimensional modelling, platform operations and technical leadership.
You will initially support a major data transformation program focused on modernising existing data assets and migrating them towards a scalable cloud-based architecture. Working alongside data architects, engineers, analysts, governance specialists and business stakeholders, you will design and implement prioritised data use cases using Azure Databricks as the core data processing platform.
Your new role
As the Senior Databricks Data Engineer, you will take technical ownership of key aspects of the Databricks environment and provide leadership to other engineers working across the platform.
You will be responsible for designing, building and maintaining robust data ingestion and transformation pipelines while helping establish engineering practices that improve platform scalability, reliability, security and maintainability.
Key responsibilities will include:
- Design, build and maintain scalable data ingestion and transformation pipelines within Azure Databricks.
- Lead the engineering delivery of new and existing data workloads across the Databricks environment.
- Integrate new data sources into the lakehouse platform from a variety of upstream systems.
- Develop transformation logic and implement complex business rules to produce trusted, reusable data assets.
- Build new datasets, measures and metrics that provide users with improved analytical insight and support important business decisions.
- Modernise existing data assets and move legacy data processing workloads towards the target-state Databricks architecture.
- Work closely with business and industry-facing teams to translate data requirements into scalable technical solutions.
- Design and develop new data models using Kimball dimensional modelling principles, including fact and dimension structures.
- Contribute to the ongoing development of the organisation's lakehouse architecture and target-state data platform.
- Extend existing job orchestration frameworks and reusable engineering patterns where required.
- Develop automated and repeatable processes for ingestion, transformation, scheduling, monitoring and deployment.
- Use platform and vendor tooling to monitor and improve data quality,
security and governance across the Databricks environment.
- Maintain, monitor and troubleshoot existing Databricks pipelines, jobs, clusters and environments.
- Investigate pipeline failures, performance bottlenecks and data quality issues and implement sustainable fixes.
- Optimise Databricks workloads for performance, reliability, scalability and cost efficiency.
- Contribute to the implementation of appropriate logging, monitoring, alerting and operational support practices.
- Establish sustainable BAU operational processes for the ongoing support of the platform.
- Apply solid software engineering practices, including modular development, code reviews, version control, testing and deployment standards.
- Act as a senior technical SME for the data platform and provide guidance on engineering design decisions.
- Mentor and provide technical guidance to data engineers across both project and BAU teams.
- Support knowledge transfer and uplift the Databricks capability of the broader data engineering function.
- Work closely with data architects and technical stakeholders to ensure solutions align with the organisation's target-state architecture.
What you'll need to succeed
To be successful in this position, you will bring strong hands-on experience delivering modern enterprise data platforms, particularly within the Microsoft Azure and Databricks ecosystem.
You will ideally demonstrate:
- Extensive commercial experience as a Senior Data Engineer, Databricks Engineer, Lead Data Engineer or similar.
- Strong hands-on experience with Azure Databricks in complex enterprise environments.
- Demonstrated experience designing and developing production-grade data ingestion and transformation pipelines.
- Strong SQL skills and experience working with large, complex datasets.
- Experience developing and supporting ETL/ELT data pipelines.
- Strong understanding of lakehouse and modern data platform architecture.
- Experience integrating multiple source systems into cloud-based data platforms.
- Strong understanding of data transformation, business rules and analytical data structures.
- Demonstrated experience with dimensional data modelling, preferably using Kimball methodology.
- Experience developing fact tables, dimension tables, star schemas and analytical data models.
- Experience with data pipeline orchestration, scheduling and dependency management.
- Strong understanding of data quality,
data governance and data security within enterprise data platforms.
- Experience monitoring, troubleshooting and optimising production Databricks workloads.
- Understanding of operational support practices for business-critical data platforms.
- Experience contributing to engineering standards, reusable frameworks and technical design principles.
- Strong software engineering discipline, including Git/version control, testing, CI/CD and code review practices.
- Experience working across both project delivery and BAU/production support environments.
- Strong stakeholder engagement skills and the ability to communicate technical concepts to both technical and non-technical audiences.
- Demonstrated experience mentoring engineers and providing technical leadership within a data engineering team.
Highly desirable experience
Candidates with experience across the following areas will be particularly well suited:
- Azure Databricks
- Apache Spark / PySpark
- SQL
- Python
- Delta Lake
- Azure Data Lake Storage
- ETL / ELT pipeline development
- Databricks Workflows / job orchestration
- Lakehouse architecture
- Kimball dimensional modelling
- Data quality frameworks
- Data governance and security
- CI/CD and DevOps practices
- Git-based development
- Large-scale data platform modernisation or migration programs
Leadership responsibilities
This position requires more than hands-on development. You will operate as a senior technical member of the data engineering function and will be expected to:
- Lead and mentor other data engineers.
- Provide technical direction and engineering guidance.
- Review solution designs and engineering approaches.
- Promote consistency in coding and development standards.
- Establish reusable engineering patterns rather than point solutions.
- Support knowledge transfer from project teams into BAU.
- Help uplift engineering capability across the wider data function.
- Champion reliable, secure and maintainable engineering practices.
- Act as a technical escalation point for complex Databricks and data pipeline issues.
What you'll get in return
This is an opportunity to play a key role in a large-scale enterprise data transformation, working on the modernisation of critical data assets and the establishment of a contemporary Azure Databricks platform.
You will have the opportunity to combine hands-on Databricks engineering with technical leadership, architecture, platform improvement and mentoring responsibilities while helping shape how a modern data engineering capability operates in BAU.
The position would particularly suit an experienced Senior or Lead Databricks Data Engineer who enjoys solving complex data problems while setting engineering standards and supporting the development of other engineers.
📌 Senior Data Engineer (Sydney)
🏢 Hays
📍 Sydney