Job Title: Databricks Engineer
Location: Melbourne, VIC (Hybrid)
Experience: 5–8+ Years
About the Role
Viable Solutions is seeking an experiencedDatabricks Engineerto join our team and deliver scalable, high-performance data engineering and analytics solutions for enterprise clients.
You will be responsible for designing, developing, and optimising data pipelines and lakehouse architectures on the Databricks platform.
This is a great opportunity for someone with deep Databricks expertise who enjoys working on large-scale data processing, machine learning pipelines, and cloud-native data solutions.
Key Responsibilities
Databricks Platform Development
Design, develop, and maintain data pipelines and workflows on theDatabricks Lakehouse Platform
Build and optimiseApache Sparkjobs for large-scale data processing and transformation
Develop and maintainDelta Laketables — schema management, optimisation, and time travel
ImplementDatabricks WorkflowsandDelta Live Tables (DLT)for pipeline orchestration
Manage and optimiseDatabricks clusters— configuration, autoscaling, and cost management
Develop notebooks and reusable libraries usingPython, Scala, or SQL
Data Engineering & Pipelines
Design and implementELT/ETL pipelinesfor ingesting, transforming, and loading data at scale
Work with structured, semi-structured, and unstructured data sources
Implementlakehouse architecturepatterns — Bronze, Silver, and Gold layers
Integrate Databricks with upstream and downstream systems — databases, APIs, and data warehouses
Implementdata quality checks, validation, and observabilityacross pipelines
ManageUnity Catalogfor data governance, lineage, and access control
Machine Learning & Analytics
Build and manageMLflowexperiments, model tracking, and model registry




Support data scientists in operationalising ML models on Databricks
Develop feature engineering pipelines for ML workloads
ImplementDatabricks AutoMLand experiment management best practices
Cloud & DevOps
Deploy and manage Databricks workspaces onAWS, Azure, or GCP
Implement infrastructure as code for Databricks —Terraform or Databricks Asset Bundles
Build and maintainCI/CD pipelinesfor Databricks workloads — GitHub Actions, Azure DevOps, or Jenkins
ImplementGitOps practicesfor notebook and pipeline version control
Monitor and optimise Databricks workloads for performance and cost efficiency
Governance & Security
ImplementUnity Catalogfor data governance, metadata management, and access control
Ensure data lineage, traceability, and compliance across all data assets
Apply row-level and column-level security across Delta Lake tables
Document data models, pipeline architectures, and operational runbooks
Required Skills & Experience
5–8+ years of experience in data engineering or a related role
Strong hands-on experience withDatabricks Lakehouse Platform(mandatory)
Solid proficiency inApache Spark— PySpark, Spark SQL, and Spark Structured Streaming(mandatory)
Strong proficiency inPython— data engineering and pipeline development(mandatory)
Experience withDelta Lake— table management, optimisation, ACID transactions, and time travel
Experience withDelta Live Tables (DLT)andDatabricks Workflows
StrongSQLskills — complex querying and data transformation
Experience withUnity Catalog— data governance and access control
Experience withMLflow— experiment tracking and model registry
Hands-on experience with cloud platforms —AWS, Azure, or GCP
Experience withCI/CD tools— GitHub Actions, Azure DevOps, or Jenkins
Experience withTerraformfor infrastructure as code
Experience working in Agile / Scrum delivery environments
#J-*****-Ljbffr

📌 Data Engineer (Melbourne)
🏢 Viable Solutions
📍 Melbourne