Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache Spark
Build and maintain Databricks notebooks using PySpark for data preparation, transformation, and enrichment
Collaborate with Data Scientists to understand data requirements and deliver clean, structured datasets
Implement data quality checks, validation rules, and monitoring mechanisms
Integrate structured, semi-structured, and unstructured data sources
Ensure data security, governance, and compliance with banking regulations
Optimize Spark jobs for performance and cost-efficiency in Azure
Participate in code reviews, design discussions, and agile ceremonies
Document data pipelines, workflows, and technical decisions
Requirements
At least 2+ years of hands-on experience in Azure Databricks
Strong proficiency in Apache Spark, especially PySpark
Experience with Azure Data Factory, Azure Synapse, Azure Blob Storage, and Delta Lake
Solid understanding of ETL/ELT processes, data modeling, and data warehousing concepts
Proficiency in Python and/or Scala for data processing
Experience with CI/CD pipelines using Azure DevOps, Git, and Terraform preferred
Familiarity with SQL, NoSQL, and data lake architectures
Knowledge of data governance, security, and compliance in financial services
Strong analytical and problem-solving skills
Excellent communication and stakeholder management abilities
Ability to work collaboratively in cross-functional teams




Agile mindset and experience in Scrum/Agile environments
Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or a related field preferred
Databricks Certified Data Engineer Associate/Skilled certification is a plus
Experience in banking or financial services is highly desirable
Australian citizenship required
Must be eligible to obtain security clearance
Core Competencies
Demonstrates expertise in designing and optimizing data ingestion pipelines using Azure Data Factory and Databricks, with a strong focus on data governance and compliance in financial services.
Proficient in data processing with PySpark and Python, ensuring high-quality data delivery and collaboration with cross-functional teams.
Highest-signal resume keywords
Azure Databricks
Apache Spark
Data Governance
ETL/ELT Processes
CI/CD Pipelines
ATS Optimization Keywords
Hard Skills
Data Ingestion Pipelines
PySpark
Azure Data Factory
SQL
NoSQL
Data Modeling
Data Warehousing
Python
Scala
Terraform
Soft Skills
Analytical Skills
Problem-Solving Skills
Communication Skills
Stakeholder Management
Collaboration
Certifications & Qualifications
Databricks Certified Data Engineer Associate
Databricks Certified Data Engineer Professional
Industry Keywords
Data Governance
Data Security
Compliance
Banking Regulations
Financial Services
Tools & Technologies
Azure Synapse
Azure Blob Storage
Delta Lake
Azure DevOps
Git
#J-*****-Ljbffr

📌 Senior Data Engineer, Databricks (Brisbane)
🏢 Jobtailor
📍 Brisbane