13 Aug
|
Acira Consultancy
|
Sydney
13 Aug
Acira Consultancy
Sydney
Data Engineer – Azure & Databricks
Experience
3–8 years of experience in Data Engineering, with strong hands-on experience in Azure and Databricks.
Job Summary
We are looking for an experienced Data Engineer with robust hands-on expertise in Microsoft Azure, Azure Databricks, and data engineering technologies. The candidate will be responsible for designing, developing, and maintaining scalable data pipelines and data solutions on Azure.
Key Responsibilities
- Design, develop, and maintain data pipelines using Azure Data Engineering services.
- Develop and optimize data processing solutions using Azure Databricks and PySpark.
- Build and maintain ETL/ELT pipelines for batch and real-time data processing.
- Develop data ingestion workflows using Azure Data Factory (ADF).
- Work with Azure Data Lake Storage Gen2 (ADLS) for scalable data storage.
- Implement data transformation and cleansing using PySpark, Python, and SQL.
- Design and implement data models for analytical and reporting requirements.
- Optimize Databricks jobs, Spark workloads, SQL queries,
and data pipelines for performance and cost.
- Implement data quality, validation, monitoring, and error-handling processes.
- Work with Delta Lake and implement Delta tables, schema evolution, partitioning, and optimization.
- Integrate data from multiple sources including databases, APIs, files, and cloud applications.
- Implement CI/CD processes for data pipelines using Azure DevOps or GitHub.
- Monitor production pipelines and troubleshoot data processing and performance issues.
- Collaborate with data architects, analysts, developers, and business stakeholders.
Required Technical Skills
- Strong experience in Data Engineering
- Hands-on experience with Microsoft Azure
- Strong experience with Azure Databricks
- Advanced PySpark
- Strong Python and SQL
- Experience with Azure Data Factory (ADF)
- Experience with Azure Data Lake Storage Gen2 (ADLS Gen2)
- Strong understanding of ETL/ELT concepts
- Experience with Delta Lake
📌 Data Engineer (Sydney)
🏢 Acira Consultancy
📍 Sydney