Data Engineer – Azure & Databricks

Experience

3–8 years of experience in Data Engineering, with solid hands-on experience in Azure and Databricks.

Job Summary

We are looking for an experienced Data Engineer with robust hands-on expertise in Microsoft Azure, Azure Databricks, and data engineering technologies. The candidate will be responsible for designing, developing, and maintaining scalable data pipelines and data solutions on Azure.

Key Responsibilities
Design, develop, and maintain data pipelines using Azure Data Engineering services.
Develop and optimize data processing solutions using Azure Databricks and PySpark.
Build and maintain ETL/ELT pipelines for batch and real-time data processing.
Develop data ingestion workflows using Azure Data Factory (ADF).
Work with Azure Data Lake Storage Gen2 (ADLS) for scalable data storage.
Implement data transformation and cleansing using PySpark, Python, and SQL.
Design and implement data models for analytical and reporting requirements.
Optimize Databricks jobs, Spark workloads, SQL queries,



and data pipelines for performance and cost.
Implement data quality, validation, monitoring, and error-handling processes.
Work with Delta Lake and implement Delta tables, schema evolution, partitioning, and optimization.
Integrate data from multiple sources including databases, APIs, files, and cloud applications.
Implement CI/CD processes for data pipelines using Azure DevOps or GitHub.
Monitor production pipelines and troubleshoot data processing and performance issues.
Collaborate with data architects, analysts, developers, and business stakeholders.

Required Technical Skills
Robust experience in Data Engineering
Hands-on experience with Microsoft Azure
Robust experience with Azure Databricks
Advanced PySpark
Robust Python and SQL
Experience with Azure Data Factory (ADF)
Experience with Azure Data Lake Storage Gen2 (ADLS Gen2)
Strong understanding of ETL/ELT concepts
Experience with Delta Lake

📌 Data Engineer Sydney
🏢 Acira consultancy
📍 Sydney