Job Description
Viable Solutions is seeking an experienced Databricks Engineer to join our team and deliver scalable, high-performance data engineering and analytics solutions for enterprise clients.
You will be responsible for designing, developing, and optimising data pipelines and lakehouse architectures on the Databricks platform.
This is a great prospect for someone with deep Databricks expertise who enjoys working on large-scale data processing, machine learning pipelines, and cloud-native data solutions.
Key Responsibilities
Databricks Platform Development
Design, develop, and maintain data pipelines and workflows on the Databricks Lakehouse Platform
Build and optimise Apache Spark jobs for large-scale data processing and transformation
Develop and maintain Delta Lake tables — schema management, optimisation, and time travel
Implement Databricks Workflows and Delta Live Tables (DLT) for pipeline orchestration
Manage and optimise Databricks clusters — configuration, autoscaling, and cost management
Develop notebooks and reusable libraries using Python, Scala, or SQL
Design and implement ELT/ETL pipelines for ingesting, transforming, and loading data at scale
Work with structured, semi-structured, and unstructured data sources
Implement lakehouse architecture patterns — Bronze, Silver, and Gold layers
Integrate Databricks with upstream and downstream systems — databases, APIs, and data warehouses
Implement data quality checks, validation, and observability across pipelines
Manage Unity Catalog for data governance, lineage, and access control
Build and manage MLflow experiments, model tracking, and model registry
Support data scientists in operationalising ML models on Databricks
Develop feature engineering pipelines for ML workloads




Implement Databricks AutoML and experiment management best practices
Deploy and manage Databricks workspaces on AWS, Azure, or GCP
Implement infrastructure as code for Databricks — Terraform or Databricks Asset Bundles
Build and maintain CI/CD pipelines for Databricks workloads — GitHub Actions, Azure DevOps, or Jenkins
Implement GitOps practices for notebook and pipeline version control
Monitor and optimise Databricks workloads for performance and cost efficiency
Governance & Security
Implement Unity Catalog for data governance, metadata management, and access control
Ensure data lineage, traceability, and compliance across all data assets
Apply row-level and column-level security across Delta Lake tables
Document data models, pipeline architectures, and operational runbooks
Required Skills & Experience
5–8+ years of experience in data engineering or a related role
Strong hands‐on experience with Databricks Lakehouse Platform (mandatory)
Strong proficiency in Apache Spark — PySpark, Spark SQL, and Spark Structured Streaming (mandatory)
Strong proficiency in Python — data engineering and pipeline development (mandatory)
Experience with Delta Lake — table management, optimisation, ACID transactions, and time travel
Experience with Delta Live Tables (DLT) and Databricks Workflows
Strong SQL skills — complex querying and data transformation
Experience with Unity Catalog — data governance and access control
Experience with MLflow — experiment tracking and model registry
Hands‐on experience with cloud platforms — AWS, Azure, or GCP
Experience with CI/CD tools — GitHub Actions, Azure DevOps, or Jenkins
Experience with Terraform for infrastructure as code
Experience working in Agile / Scrum delivery environments

📌 Data Engineer (Victoria)
🏢 Viable Solutions
📍 Victoria