Job Description
Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache Spark
n
Build and maintain Databricks notebooks using PySpark for data preparation, transformation, and enrichment
n
Collaborate with Data Scientists to understand data requirements and deliver clean, structured datasets
n
Implement data quality checks, validation rules, and monitoring mechanisms
n
Integrate structured, semi-structured, and unstructured data sources
n
Ensure data security, governance, and compliance with banking regulations
n
Optimize Spark jobs for performance and cost-efficiency in Azure
n
Participate in code reviews, design discussions, and agile ceremonies
n
Document data pipelines, workflows, and technical decisions
n
Requirements
n
At least 2+ years of hands-on experience in Azure Databricks
n
Robust proficiency in Apache Spark, especially PySpark
n
Experience with Azure Data Factory, Azure Synapse, Azure Blob Storage, and Delta Lake
n
Solid understanding of ETL/ELT processes, data modeling, and data warehousing concepts
n
Proficiency in Python and/or Scala for data processing
n
Experience with CI/CD pipelines using Azure DevOps, Git, and Terraform preferred
n
Familiarity with SQL, NoSQL, and data lake architectures
n
Knowledge of data governance, security, and compliance in financial services
n
Strong analytical and problem-solving skills
n
Excellent communication and stakeholder management abilities
n
Ability to work collaboratively in cross-functional teams
n
Agile mindset and experience in Scrum/Agile environments
n




Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or a related field preferred
n
Databricks Certified Data Engineer Associate/Professional certification is a plus
n
Experience in banking or financial services is highly desirable
n
Australian citizenship required
n
Must be eligible to obtain security clearance
n
Core Competencies
n
Demonstrates expertise in designing and optimizing data ingestion pipelines using Azure Data Factory and Databricks, with a strong focus on data governance and compliance in financial services. Proficient in data processing with PySpark and Python, ensuring high-quality data delivery and collaboration with cross-functional teams.
n
Highest-signal resume keywords
n
Azure Databricks
n
Apache Spark
n
Data Governance
n
ETL/ELT Processes
n
CI/CD Pipelines
n
ATS Optimization Keywords
n
Hard Skills
n
Data Ingestion Pipelines
n
PySpark
n
Azure Data Factory
n
SQL
n
NoSQL
n
Data Modeling
n
Data Warehousing
n
Python
n
Scala
n
Terraform
n
Soft Skills
n
Analytical Skills
n
Problem-Solving Skills
n
Communication Skills
n
Stakeholder Management
n
Collaboration
n
Certifications & Qualifications
n
Databricks Certified Data Engineer Associate
n
Databricks Certified Data Engineer Professional
n
Industry Keywords
n
Data Governance
n
Data Security
n
Compliance
n
Banking Regulations
n
Financial Services
n
Tools & Technologies
n
Azure Synapse
n
Azure Blob Storage
n
Delta Lake
n
Azure DevOps
n
Git
#J-*****-Ljbffr

📌 Senior Data Engineer, Databricks (Queensland)
🏢 Jobtailor
📍 Queensland