Job Description

n

Role -Databricks Data Engineer Senior Analyst

n

About the Role

n

We are seeking experienced Databricks Data Engineers to join our Data & AI. You will play a pivotal role in designing, developing, and maintaining scalable data pipelines and transformation workflows on Azure Databricks, enabling advanced analytics and data engineering initiatives.

n

Key Responsibilities

n

n
- n

Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache Spark.

n
- n

Build and maintain Databricks notebooks using PySpark for data preparation, transformation, and enrichment to support Data Science use cases.

n
- n

Collaborate closely with Data Scientists to understand data requirements and ensure timely delivery of clean, structured datasets.

n
- n

Implement data quality checks, validation rules, and monitoring mechanisms.

n
- n

Integrate data from various sources including structured, semi-structured, and unstructured formats.

n
- n

Ensure data security, governance, and compliance with banking regulations.

n
- n

Optimize Spark jobs for performance and cost-efficiency in Azure.

n
- n

Participate in code reviews, design discussions, and agile ceremonies.

n
- n

Document data pipelines, workflows, and technical decisions.

n

n

.

n

Required Skills & Qualifications

n

Technical Skills

n

n
- n

Experience in data engineering, with at least 2+ years hands-on experience in Azure Databricks.

n
- n

Strong proficiency in Apache Spark,



especially PySpark for distributed data processing.

n
- n

Experience with Azure Data Factory, Azure Synapse, Azure Blob Storage, and Delta Lake.

n
- n

Solid understanding of ETL/ELT processes, data modeling, and data warehousing concepts.

n
- n

Proficiency in Python and/or Scala for data processing.

n
- n

Experience with CI/CD pipelines using tools like Azure DevOps, Git, and Terraform (preferred).

n
- n

Familiarity with SQL, NoSQL, and data lake architectures.

n
- n

Knowledge of data governance, security, and compliance in financial services.

n

n

Soft Skills

n

n
- n

Robust analytical and problem-solving skills.

n
- n

Excellent communication and stakeholder management abilities.

n
- n

Ability to work collaboratively in cross-functional teams.

n
- n

Agile mindset and experience working in Scrum/Agile environments.

n

n

Preferred Qualifications

n

n
- n

Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or related field.

n
- n

Databricks Certified Data Engineer Associate/Professional certification is a plus.

n
- n

Experience working in banking or financial services domain is highly desirable.

n

n

n

n

n

n

n

n

n

n

n

n

n

n

n

n

n

n

Australian citizenship is required, and candidates must be eligible to obtain security clearance.

n

Location is flexible across Brisbane, Sydney, and Melbourne.

n

n

n

n

n

n

n

n

n

n

n

n

n

n

n

n

n

.

n

📌 Databricks Data Engineer Senior Analyst (Queensland)
🏢 Avanade
📍 Queensland