Job Title: Databricks Engineer

Location: Melbourne, VIC (Hybrid)

Experience: 5–8 Years

About the Role

Viable Solutions is seeking an experienced Databricks Engineer to join our team and deliver scalable, high-performance data engineering and analytics solutions for enterprise clients. You will be responsible for designing, developing, and optimising data pipelines and lakehouse architectures on the Databricks platform. This is a great chance for someone with deep Databricks expertise who enjoys working on large-scale data processing, machine learning pipelines, and cloud-native data solutions.

Key Responsibilities

Databricks Platform Development

- Design, develop, and maintain data pipelines and workflows on the Databricks Lakehouse Platform
- Build and optimise Apache Spark jobs for large-scale data processing and transformation
- Develop and maintain Delta Lake tables — schema management, optimisation, and time travel
- Implement Databricks Workflows and Delta Live Tables (DLT) for pipeline orchestration
- Manage and optimise Databricks clusters — configuration, autoscaling, and cost management
- Develop notebooks and reusable libraries using Python, Scala, or SQL

Data Engineering & Pipelines

- Design and implement ELT/ETL pipelines for ingesting, transforming, and loading data at scale
- Work with structured, semi-structured, and unstructured data sources
- Implement lakehouse architecture patterns — Bronze, Silver, and Gold layers
- Integrate Databricks with upstream and downstream systems — databases, APIs, and data warehouses
- Implement data quality checks, validation, and observability across pipelines
- Manage Unity Catalog for data governance, lineage, and access control

Machine Learning & Analytics

- Build and manage MLflow experiments, model tracking, and model registry




- Support data scientists in operationalising ML models on Databricks
- Develop feature engineering pipelines for ML workloads
- Implement Databricks AutoML and experiment management best practices

Cloud & DevOps

- Deploy and manage Databricks workspaces on AWS, Azure, or GCP
- Implement infrastructure as code for Databricks — Terraform or Databricks Asset Bundles
- Build and maintain CI/CD pipelines for Databricks workloads — GitHub Actions, Azure DevOps, or Jenkins
- Implement GitOps practices for notebook and pipeline version control
- Monitor and optimise Databricks workloads for performance and cost efficiency

Governance & Security

- Implement Unity Catalog for data governance, metadata management, and access control
- Ensure data lineage, traceability, and compliance across all data assets
- Apply row-level and column-level security across Delta Lake tables
- Document data models, pipeline architectures, and operational runbooks

Required Skills & Experience

- 5–8 years of experience in data engineering or a related role
- Strong hands-on experience with Databricks Lakehouse Platform (mandatory)
- Strong proficiency in Apache Spark — PySpark, Spark SQL, and Spark Structured Streaming (mandatory)
- Strong proficiency in Python — data engineering and pipeline development (mandatory)
- Experience with Delta Lake — table management, optimisation, ACID transactions, and time travel
- Experience with Delta Live Tables (DLT) and Databricks Workflows
- Strong SQL skills — complex querying and data transformation
- Experience with Unity Catalog — data governance and access control
- Experience with MLflow — experiment tracking and model registry
- Hands-on experience with cloud platforms — AWS, Azure, or GCP
- Experience with CI/CD tools — GitHub Actions, Azure DevOps, or Jenkins
- Experience with Terraform for infrastructure as code
- Experience working in Agile / Scrum delivery environments

📌 Data Engineer (Melbourne)
🏢 Viable Solutions
📍 Melbourne