- Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache Spark
- Build and maintain Databricks notebooks using PySpark for data preparation, transformation, and enrichment
- Collaborate with Data Scientists to understand data requirements and deliver clean, structured datasets
- Implement data quality checks, validation rules, and monitoring mechanisms
- Integrate structured, semi-structured, and unstructured data sources
- Ensure data security, governance, and compliance with banking regulations
- Optimize Spark jobs for performance and cost-efficiency in Azure
- Participate in code reviews, design discussions, and agile ceremonies
- Document data pipelines, workflows, and technical decisions

Requirements

- At least 2+ years of hands-on experience in Azure Databricks
- Solid proficiency in Apache Spark, especially PySpark
- Experience with Azure Data Factory, Azure Synapse, Azure Blob Storage, and Delta Lake
- Solid understanding of ETL/ELT processes, data modeling, and data warehousing concepts
- Proficiency in Python and/or Scala for data processing
- Experience with CI/CD pipelines using Azure DevOps, Git, and Terraform preferred
- Familiarity with SQL, NoSQL, and data lake architectures
- Knowledge of data governance, security, and compliance in financial services
- Strong analytical and problem-solving skills
- Excellent communication and stakeholder management abilities
- Ability to work collaboratively in cross-functional teams
- Agile mindset and experience in Scrum/Agile environments




- Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or a related field preferred
- Databricks Certified Data Engineer Associate/Professional certification is a plus
- Experience in banking or financial services is highly desirable
- Australian citizenship required
- Must be eligible to obtain security clearance

Core Competencies

Demonstrates expertise in designing and optimizing data ingestion pipelines using Azure Data Factory and Databricks, with a strong focus on data governance and compliance in financial services. Proficient in data processing with PySpark and Python, ensuring high-quality data delivery and collaboration with cross-functional teams.

Highest-signal resume keywords

- Azure Databricks
- Apache Spark
- Data Governance
- ETL/ELT Processes
- CI/CD Pipelines

ATS Optimization Keywords

Hard Skills

- Data Ingestion Pipelines
- PySpark
- Azure Data Factory
- SQL
- NoSQL
- Data Modeling
- Data Warehousing
- Python
- Scala
- Terraform

Soft Skills

- Analytical Skills
- Problem-Solving Skills
- Communication Skills
- Stakeholder Management
- Collaboration

Certifications & Qualifications

- Databricks Certified Data Engineer Associate
- Databricks Certified Data Engineer Professional

Industry Keywords

- Data Governance
- Data Security
- Compliance
- Banking Regulations
- Financial Services

Tools & Technologies

- Azure Synapse
- Azure Blob Storage
- Delta Lake
- Azure DevOps
- Git

#J-18808-Ljbffr

📌 Senior Data Engineer, Databricks (Brisbane City)
🏢 Jobtailor
📍 Brisbane City