Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache SparkBuild and maintain Databricks notebooks using PySpark for data preparation, transformation, and enrichmentCollaborate with Data Scientists to understand data requirements and deliver clean, structured datasetsImplement data quality checks, validation rules, and monitoring mechanismsIntegrate structured, semi-structured, and unstructured data sourcesEnsure data security, governance, and compliance with banking regulationsOptimize Spark jobs for performance and cost-efficiency in AzureParticipate in code reviews, design discussions, and agile ceremoniesDocument data pipelines, workflows, and technical decisionsRequirementsAt least 2+ years of hands-on experience in Azure DatabricksStrong proficiency in Apache Spark, especially PySparkExperience with Azure Data Factory, Azure Synapse, Azure Blob Storage, and Delta LakeSolid understanding of ETL/ELT processes, data modeling, and data warehousing conceptsProficiency in Python and/or Scala for data processingExperience with CI/CD pipelines using Azure DevOps, Git, and Terraform preferredFamiliarity with SQL, NoSQL, and data lake architecturesKnowledge of data governance, security,



and compliance in financial servicesStrong analytical and problem-solving skillsExcellent communication and stakeholder management abilitiesAbility to work collaboratively in cross-functional teamsAgile mindset and experience in Scrum/Agile environmentsBachelor's or Master's degree in Computer Science, Information Systems, Engineering, or a related field preferredDatabricks Certified Data Engineer Associate/Skilled certification is a plusExperience in banking or financial services is highly desirableAustralian citizenship requiredMust be eligible to obtain security clearanceCore CompetenciesDemonstrates expertise in designing and optimizing data ingestion pipelines using Azure Data Factory and Databricks, with a strong focus on data governance and compliance in financial services.
Proficient in data processing with PySpark and Python, ensuring high-quality data delivery and collaboration with cross-functional teams.Highest-signal resume keywordsAzure DatabricksApache SparkData GovernanceETL/ELT ProcessesCI/CD PipelinesATS Optimization KeywordsHard SkillsData Ingestion PipelinesPySparkAzure Data FactorySQLNoSQLData ModelingData WarehousingPythonScalaTerraformSoft SkillsAnalytical SkillsProblem-Solving SkillsCommunication SkillsStakeholder ManagementCollaborationCertifications & QualificationsDatabricks Certified Data Engineer AssociateDatabricks Certified Data Engineer ProfessionalIndustry KeywordsData GovernanceData SecurityComplianceBanking RegulationsFinancial ServicesTools & TechnologiesAzure SynapseAzure Blob StorageDelta LakeAzure DevOpsGit
#J-*****-Ljbffr

📌 Senior Data Engineer, Databricks (Brisbane)
🏢 Jobtailor
📍 Brisbane