Job Description
Job Title: Databricks Engineer
n
Location: Melbourne, VIC (Hybrid)
n
Experience: 5–8+ Years
n
About the Role
n
Viable Solutions is seeking an experienced Databricks Engineer to join our team and deliver scalable, high-performance data engineering and analytics solutions for enterprise clients. You will be responsible for designing, developing, and optimising data pipelines and lakehouse architectures on the Databricks platform. This is a great opportunity for someone with deep Databricks expertise who enjoys working on large-scale data processing, machine learning pipelines, and cloud-native data solutions.
n
Key Responsibilities
n
Databricks Platform Development
n
n
Design, develop, and maintain data pipelines and workflows on the Databricks Lakehouse Platform
n
Build and optimise Apache Spark jobs for large-scale data processing and transformation
n
Develop and maintain Delta Lake tables — schema management, optimisation, and time travel
n
Implement Databricks Workflows and Delta Live Tables (DLT) for pipeline orchestration
n
Manage and optimise Databricks clusters — configuration, autoscaling, and cost management
n
Develop notebooks and reusable libraries using Python, Scala, or SQL
n
n
Data Engineering & Pipelines
n
n
Design and implement ELT/ETL pipelines for ingesting, transforming, and loading data at scale
n
Work with structured, semi-structured, and unstructured data sources
n
Implement lakehouse architecture patterns — Bronze, Silver, and Gold layers
n
Integrate Databricks with upstream and downstream systems — databases, APIs, and data warehouses
n
Implement data quality checks, validation, and observability across pipelines
n
Manage Unity Catalog for data governance, lineage, and access control
n
n
Machine Learning & Analytics
n
n
Build and manage MLflow experiments, model tracking, and model registry
n




Support data scientists in operationalising ML models on Databricks
n
Develop feature engineering pipelines for ML workloads
n
Implement Databricks AutoML and experiment management best practices
n
n
Cloud & DevOps
n
n
Deploy and manage Databricks workspaces on AWS, Azure, or GCP
n
Implement infrastructure as code for Databricks — Terraform or Databricks Asset Bundles
n
Build and maintain CI/CD pipelines for Databricks workloads — GitHub Actions, Azure DevOps, or Jenkins
n
Implement GitOps practices for notebook and pipeline version control
n
Monitor and optimise Databricks workloads for performance and cost efficiency
n
n
Governance & Security
n
n
Implement Unity Catalog for data governance, metadata management, and access control
n
Ensure data lineage, traceability, and compliance across all data assets
n
Apply row-level and column-level security across Delta Lake tables
n
Document data models, pipeline architectures, and operational runbooks
n
n
Required Skills & Experience
n
n
5–8+ years of experience in data engineering or a related role
n
Strong hands-on experience with Databricks Lakehouse Platform (mandatory)
n
Strong proficiency in Apache Spark — PySpark, Spark SQL, and Spark Structured Streaming (mandatory)
n
Strong proficiency in Python — data engineering and pipeline development (mandatory)
n
Experience with Delta Lake — table management, optimisation, ACID transactions, and time travel
n
Experience with Delta Live Tables (DLT) and Databricks Workflows
n
Robust SQL skills — complex querying and data transformation
n
Experience with Unity Catalog — data governance and access control
n
Experience with MLflow — experiment tracking and model registry
n
Hands-on experience with cloud platforms — AWS, Azure, or GCP
n
Experience with CI/CD tools — GitHub Actions, Azure DevOps, or Jenkins
n
Experience with Terraform for infrastructure as code
n
Experience working in Agile / Scrum delivery environments
n
n
#J-*****-Ljbffr

📌 Data Engineer (Victoria)
🏢 Viable Solutions
📍 Victoria