We are seeking a seasoned Databricks Tech Lead to design, build, and scale our next-generation data ingestion and processing framework. In this role, you will lead a team of data engineers, champion best practices in software development, and build reusable, highly scalable pipelines to ingest and transform data from diverse enterprise sources into our Lakehouse platform.

Key Responsibilities

- Framework Development: Design and build metadata-driven, generic data ingestion frameworks (batch, streaming, and CDC) to automate the onboarding of new data sources.
- Architecture & Design: Define architectural standards for the Medallion architecture (Bronze, Silver, Gold layers) utilizing Delta Lake.
- Technical Leadership: Lead technical design discussions, mentor junior/mid-level engineers, and conduct rigorous code reviews.
- Pipeline Optimization: Optimize Spark jobs for performance, scalability, and cost reduction across the Databricks platform.
- CI/CD & DevOps: Implement and mature CI/CD pipelines (Git, automated testing) for data platforms and enforce DataOps best practices.
- Governance & Security:



Integrate ingestion pipelines with Unity Catalog for enterprise-grade data governance and lineage tracking.

Required Qualifications

- Experience: 8+ years in Data Engineering, with at least 3–4 years of hands‑on technical leadership experience leading engineering pods.
- Core Databricks & Spark: Advanced proficiency in PySpark and Databricks SQL.
- Ingestion Expertise: Proven experience building custom or highly configurable data ingestion frameworks for structured and unstructured data.
- Cloud Platforms: Strong experience working with AWS, Azure, or GCP cloud‑native services (e.g., S3, ADLS).
- Engineering Standards: Deep understanding of data modeling (star/snowflake schemas), SCD (Slowly Changing Dimensions), and software engineering principles in a data context.
- Soft Skills: Excellent stakeholder management and the ability to translate complex technical architectures into business value.

Preferred Qualifications

- Experience with Kafka, dbt, or Delta Live Tables (DLT).
- Databricks Certified Data Engineer Associate/Qualified credentials

#J-18808-Ljbffr

📌 Databricks Tech Lead (City of Sydney)
🏢 Nityo Infotech
📍 City of Sydney