We are seeking an experienced Databricks Architect to design, implement and optimise scalable enterprise data platforms, distributed processing solutions and AI/ML architectures using the Databricks platform.

Key Responsibilities

- Lead the end-to-end architecture and deployment of large-scale data platforms using Databricks, Apache Spark, Delta Lake and Unity Catalog.
- Design scalable batch and real-time data processing solutions and ETL/ELT pipelines.
- Develop medallion and lakehouse architecture patterns for enterprise data workloads.
- Architect secure and resilient Databricks environments across Azure, AWS or GCP.
- Establish data governance, access controls, lineage and compliance standards through Unity Catalog.
- Design integrations between Databricks and cloud storage, databases, APIs, BI platforms and enterprise applications.
- Define platform security standards covering identity, networking, encryption, secrets management and role-based access.
- Implement CI/CD, infrastructure-as-code and automated deployment practices for Databricks workloads.
- Optimise Spark jobs, clusters, SQL workloads and resource utilisation for performance and cost efficiency.
- Support the architecture and operationalisation of machine learning and generative AI solutions.
- Troubleshoot complex platform, integration and performance issues.
- Create reusable architecture patterns, standards and technical documentation.
- Collaborate with data engineers, AI/ML teams, security specialists, cloud engineers and business stakeholders.
- Provide technical leadership, architecture assurance and mentoring to engineering teams.
- Develop platform roadmaps and drive enterprise adoption of Databricks.

Mandatory Skills and Experience





- Extensive experience designing enterprise data platforms and lakehouse architectures.
- Strong hands-on Databricks architecture and implementation experience.
- Advanced knowledge of Apache Spark, PySpark, Spark SQL and performance optimisation.
- Strong experience with Delta Lake and Unity Catalog.
- Experience designing ETL/ELT pipelines and distributed data-processing solutions.
- Strong cloud architecture experience across Azure, AWS or GCP.
- Experience with cloud storage, networking, identity and security services.
- Knowledge of data governance, lineage, access controls and regulatory compliance.
- Experience implementing CI/CD and infrastructure-as-code using tools such as Terraform, Azure DevOps, GitHub Actions or Jenkins.
- Strong understanding of SQL, Python and data-modelling principles.
- Experience integrating Databricks with enterprise data and analytics platforms.
- Strong stakeholder-management and technical-leadership capabilities.

Highly Desirable

- Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Skilled.
- Databricks Certified Generative AI Engineer or related architecture certification.
- Experience with MLflow, Feature Store, Model Serving and MLOps.
- Experience designing AI, machine-learning, GenAI or agentic AI solutions.
- Knowledge of streaming technologies such as Kafka, Structured Streaming or Event Hubs.
- Experience delivering regulated, government or large-enterprise data platforms.

Paxus values diversity and welcomes applications from Indigenous Australians, people from diverse cultural and linguistic backgrounds and people living with a disability. If you require an adjustment to the recruitment process, including the application form in an alternate format, please contact me on the above contact details.

#J-18808-Ljbffr

📌 Databricks Architect (City of Sydney)
🏢 Paxus - Technology + Digital Talent
📍 City of Sydney