Status:
Valid visa with full working rights
Key Responsibilities
Framework Development:
Design and build metadata-driven, generic data ingestion frameworks (batch, streaming, and CDC) to automate the onboarding of new data sources.
Architecture & Design:
Define architectural standards for the Medallion architecture (Bronze, Silver, Gold layers) utilizing Delta Lake.
Technical Leadership:
Lead technical design discussions, mentor junior/mid-level engineers, and conduct rigorous code reviews.
Pipeline Optimization:
Optimize Spark jobs for performance, scalability, and cost reduction across the Databricks platform.
CI/CD & DevOps:
Implement and mature CI/CD pipelines (Git, automated testing) for data platforms and enforce DataOps best practices.
Governance & Security:
Integrate ingestion pipelines with Unity Catalog for enterprise-grade data governance and lineage tracking.
Required Qualifications
Experience:
8+ years in Data Engineering,



with at least 3–4 years of hands-on technical leadership experience leading engineering pods.
Core Databricks & Spark:
Advanced proficiency in PySpark and Databricks SQL.
Ingestion Expertise:
Proven experience building custom or highly configurable data ingestion frameworks for structured and unstructured data.
Cloud Platforms:
Solid experience working with AWS, Azure, or GCP cloud-native services (e.g., S3, ADLS).
Engineering Standards:
Deep understanding of data modeling (star/snowflake schemas), SCD (Slowly Changing Dimensions), and software engineering principles in a data context.
Soft Skills:
Excellent stakeholder management and the ability to translate complex technical architectures into business value.
Preferred Qualifications
Experience with Kafka, dbt, or Delta Live Tables (DLT).
Databricks Certified Data Engineer Associate/Professional credentials
To read about our privacy policy
#J-*****-Ljbffr

📌 Data Lead (Sydney)
🏢 FR Consultancy
📍 Sydney