Job Description

Job Title: Databricks Engineer

n

Location: Melbourne, VIC (Hybrid)

n

Experience: 5–8+ Years

n

About the Role

n

Viable Solutions is seeking an experienced Databricks Engineer to join our team and deliver scalable, high-performance data engineering and analytics solutions for enterprise clients. You will be responsible for designing, developing, and optimising data pipelines and lakehouse architectures on the Databricks platform. This is a great opportunity for someone with deep Databricks expertise who enjoys working on large-scale data processing, machine learning pipelines, and cloud-native data solutions.

n

Key Responsibilities

n

Databricks Platform Development

n

n
- Design, develop, and maintain data pipelines and workflows on the Databricks Lakehouse Platform
n
- Build and optimise Apache Spark jobs for large-scale data processing and transformation
n
- Develop and maintain Delta Lake tables — schema management, optimisation, and time travel
n
- Implement Databricks Workflows and Delta Live Tables (DLT) for pipeline orchestration
n
- Manage and optimise Databricks clusters — configuration, autoscaling, and cost management
n
- Develop notebooks and reusable libraries using Python, Scala, or SQL
n

n

Data Engineering & Pipelines

n

n
- Design and implement ELT/ETL pipelines for ingesting, transforming, and loading data at scale
n
- Work with structured, semi-structured, and unstructured data sources
n
- Implement lakehouse architecture patterns — Bronze, Silver, and Gold layers
n
- Integrate Databricks with upstream and downstream systems — databases, APIs, and data warehouses
n
- Implement data quality checks, validation, and observability across pipelines
n
- Manage Unity Catalog for data governance, lineage, and access control
n

n

Machine Learning & Analytics

n

n
- Build and manage MLflow experiments, model tracking, and model registry
n




- Support data scientists in operationalising ML models on Databricks
n
- Develop feature engineering pipelines for ML workloads
n
- Implement Databricks AutoML and experiment management best practices
n

n

Cloud & DevOps

n

n
- Deploy and manage Databricks workspaces on AWS, Azure, or GCP
n
- Implement infrastructure as code for Databricks — Terraform or Databricks Asset Bundles
n
- Build and maintain CI/CD pipelines for Databricks workloads — GitHub Actions, Azure DevOps, or Jenkins
n
- Implement GitOps practices for notebook and pipeline version control
n
- Monitor and optimise Databricks workloads for performance and cost efficiency
n

n

Governance & Security

n

n
- Implement Unity Catalog for data governance, metadata management, and access control
n
- Ensure data lineage, traceability, and compliance across all data assets
n
- Apply row-level and column-level security across Delta Lake tables
n
- Document data models, pipeline architectures, and operational runbooks
n

n

Required Skills & Experience

n

n
- 5–8+ years of experience in data engineering or a related role
n
- Strong hands-on experience with Databricks Lakehouse Platform (mandatory)
n
- Solid proficiency in Apache Spark — PySpark, Spark SQL, and Spark Structured Streaming (mandatory)
n
- Strong proficiency in Python — data engineering and pipeline development (mandatory)
n
- Experience with Delta Lake — table management, optimisation, ACID transactions, and time travel
n
- Experience with Delta Live Tables (DLT) and Databricks Workflows
n
- Strong SQL skills — complex querying and data transformation
n
- Experience with Unity Catalog — data governance and access control
n
- Experience with MLflow — experiment tracking and model registry
n
- Hands-on experience with cloud platforms — AWS, Azure, or GCP
n
- Experience with CI/CD tools — GitHub Actions, Azure DevOps, or Jenkins
n
- Experience with Terraform for infrastructure as code
n
- Experience working in Agile / Scrum delivery environments
n

n

#J-18808-Ljbffr

📌 Data Engineer (Victoria)
🏢 Viable Solutions
📍 Victoria