Job Description

We are seeking an experienced Databricks Architect to design, implement and optimise scalable enterprise data platforms, distributed processing solutions and AI/ML architectures using the Databricks platform.

n

Key Responsibilities

n

n
- Lead the end-to-end architecture and deployment of large-scale data platforms using Databricks, Apache Spark, Delta Lake and Unity Catalog.
n
- Design scalable batch and real-time data processing solutions and ETL/ELT pipelines.
n
- Develop medallion and lakehouse architecture patterns for enterprise data workloads.
n
- Architect secure and resilient Databricks environments across Azure, AWS or GCP.
n
- Establish data governance, access controls, lineage and compliance standards through Unity Catalog.
n
- Design integrations between Databricks and cloud storage, databases, APIs, BI platforms and enterprise applications.
n
- Define platform security standards covering identity, networking, encryption, secrets management and role-based access.
n
- Implement CI/CD, infrastructure-as-code and automated deployment practices for Databricks workloads.
n
- Optimise Spark jobs, clusters, SQL workloads and resource utilisation for performance and cost efficiency.
n
- Support the architecture and operationalisation of machine learning and generative AI solutions.
n
- Troubleshoot complex platform, integration and performance issues.
n
- Create reusable architecture patterns, standards and technical documentation.
n
- Collaborate with data engineers, AI/ML teams, security specialists, cloud engineers and business stakeholders.
n
- Provide technical leadership, architecture assurance and mentoring to engineering teams.
n
- Develop platform roadmaps and drive enterprise adoption of Databricks.
n

n

Mandatory Skills and Experience

n

n
- Extensive experience designing enterprise data platforms and lakehouse architectures.
n
- Strong hands-on Databricks architecture and implementation experience.
n
- Advanced knowledge of Apache Spark, PySpark, Spark SQL and performance optimisation.
n
- Strong experience with Delta Lake and Unity Catalog.
n




- Experience designing ETL/ELT pipelines and distributed data-processing solutions.
n
- Solid cloud architecture experience across Azure, AWS or GCP.
n
- Experience with cloud storage, networking, identity and security services.
n
- Knowledge of data governance, lineage, access controls and regulatory compliance.
n
- Experience implementing CI/CD and infrastructure-as-code using tools such as Terraform, Azure DevOps, GitHub Actions or Jenkins.
n
- Strong understanding of SQL, Python and data-modelling principles.
n
- Experience integrating Databricks with enterprise data and analytics platforms.
n
- Strong stakeholder-management and technical-leadership capabilities.
n

n

Highly Desirable

n

n
- Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional.
n
- Databricks Certified Generative AI Engineer or related architecture certification.
n
- Experience with MLflow, Feature Store, Model Serving and MLOps.
n
- Experience designing AI, machine-learning, GenAI or agentic AI solutions.
n
- Knowledge of streaming technologies such as Kafka, Structured Streaming or Event Hubs.
n
- Experience delivering regulated, government or large-enterprise data platforms.
n

n

Desired Skills and Experience

n

n
- Databricks Architect
n
- Led the architecture, design, and implementation of enterprise-scale Databricks Lakehouse platforms leveraging Apache Spark, Delta Lake, and Unity Catalog.
n
- Designed and delivered scalable batch and real-time data processing solutions, enabling high-performance analytics and business intelligence capabilities.
n
- Developed enterprise data architectures using Medallion and Lakehouse design patterns to support advanced data engineering, analytics, and AI workloads.
n
- Architected secure, resilient,



and scalable Databricks environments across Azure, AWS, and GCP cloud platforms.
n
- Established enterprise data governance frameworks, implementing access controls, lineage, compliance standards, and data security through Unity Catalog.
n
- Designed integrations between Databricks, cloud storage services, databases, APIs, enterprise applications, and BI platforms.
n
- Defined platform security standards covering identity management, networking, encryption, secrets management, and role-based access controls (RBAC).
n
- Implemented CI/CD pipelines and Infrastructure-as-Code solutions using Terraform, Azure DevOps, GitHub Actions, and Jenkins to automate deployments and platform management.
n
- Optimized Apache Spark workloads, cluster configurations, and SQL performance to improve processing efficiency and reduce operational costs.
n
- Supported the architecture and operationalization of Machine Learning, MLOps, and Generative AI solutions using MLflow, Feature Store, and Model Serving capabilities.
n
- Provided technical leadership, architectural governance, mentoring, and best-practice guidance to cross-functional engineering teams.
n
- Collaborated with data engineers, AI/ML specialists, cloud engineers, security teams, and business stakeholders to deliver strategic data initiatives.
n
- Troubleshot and resolved complex platform, integration, and performance challenges across enterprise environments.
n
- Developed reusable architecture standards, reference models, and technical documentation to accelerate delivery and support platform adoption.
n
- Created strategic platform roadmaps and drove enterprise-wide adoption of Databricks technologies and modern data architectures.
n

n

Paxus values diversity and welcomes applications from Indigenous Australians, people from diverse cultural and linguistic backgrounds and people living with a disability. If you require an adjustment to the recruitment process, including the application form in an alternate format, please contact me on the above contact details.

n

#J-18808-Ljbffr

📌 Databricks Architect (Victoria)
🏢 Paxus
📍 Victoria