Job Title: Databricks Engineer
Location: Melbourne, VIC (Hybrid)
Experience: 5–8+ Years
About the Role
Viable Solutions is seeking an experienced
Databricks Engineer
to join our team and deliver scalable, high-performance data engineering and analytics solutions for enterprise clients. You will be responsible for designing, developing, and optimising data pipelines and lakehouse architectures on the Databricks platform. This is a great opportunity for someone with deep Databricks expertise who enjoys working on large-scale data processing, machine learning pipelines, and cloud-native data solutions.
Key Responsibilities
Databricks Platform Development
Design, develop, and maintain data pipelines and workflows on the
Databricks Lakehouse Platform
Build and optimise
Apache Spark
jobs for large-scale data processing and transformation
Develop and maintain
Delta Lake
tables — schema management, optimisation, and time travel
Implement
Databricks Workflows
and
Delta Live Tables (DLT)
for pipeline orchestration
Manage and optimise
Databricks clusters
— configuration, autoscaling, and cost management
Develop notebooks and reusable libraries using
Python, Scala, or SQL
Data Engineering & Pipelines
Design and implement
ELT/ETL pipelines
for ingesting, transforming, and loading data at scale
Work with structured, semi-structured, and unstructured data sources
Implement
lakehouse architecture
patterns — Bronze, Silver, and Gold layers
Integrate Databricks with upstream and downstream systems — databases, APIs, and data warehouses
Implement
data quality checks, validation, and observability
across pipelines
Manage
Unity Catalog
for data governance, lineage, and access control
Machine Learning & Analytics
Build and manage
MLflow
experiments, model tracking, and model registry
Support data scientists in operationalising ML models on Databricks




Develop feature engineering pipelines for ML workloads
Implement
Databricks AutoML
and experiment management best practices
Cloud & DevOps
Deploy and manage Databricks workspaces on
AWS, Azure, or GCP
Implement infrastructure as code for Databricks —
Terraform or Databricks Asset Bundles
Build and maintain
CI/CD pipelines
for Databricks workloads — GitHub Actions, Azure DevOps, or Jenkins
Implement
GitOps practices
for notebook and pipeline version control
Monitor and optimise Databricks workloads for performance and cost efficiency
Governance & Security
Implement
Unity Catalog
for data governance, metadata management, and access control
Ensure data lineage, traceability, and compliance across all data assets
Apply row-level and column-level security across Delta Lake tables
Document data models, pipeline architectures, and operational runbooks
Required Skills & Experience
5–8+ years of experience in data engineering or a related role
Strong hands-on experience with
Databricks Lakehouse Platform
(mandatory)
Strong proficiency in
Apache Spark
— PySpark, Spark SQL, and Spark Structured Streaming
(mandatory)
Strong proficiency in
Python
— data engineering and pipeline development
(mandatory)
Experience with
Delta Lake
— table management, optimisation, ACID transactions, and time travel
Experience with
Delta Live Tables (DLT)
and
Databricks Workflows
Solid
SQL
skills — complex querying and data transformation
Experience with
Unity Catalog
— data governance and access control
Experience with
MLflow
— experiment tracking and model registry
Hands-on experience with cloud platforms —
AWS, Azure, or GCP
Experience with
CI/CD tools
— GitHub Actions, Azure DevOps, or Jenkins
Experience with
Terraform
for infrastructure as code
Experience working in Agile / Scrum delivery environments
#J-*****-Ljbffr

📌 Data Engineer (Melbourne)
🏢 Viable Solutions
📍 Melbourne