Job Description
We are seeking an experienced Databricks Architect to design, implement and optimise scalable enterprise data platforms, distributed processing solutions and AI/ML architectures using the Databricks platform.
n
Key Responsibilities
n
n
Lead the end-to-end architecture and deployment of large-scale data platforms using Databricks, Apache Spark, Delta Lake and Unity Catalog.
n
Design scalable batch and real-time data processing solutions and ETL/ELT pipelines.
n
Develop medallion and lakehouse architecture patterns for enterprise data workloads.
n
Architect secure and resilient Databricks environments across Azure, AWS or GCP.
n
Establish data governance, access controls, lineage and compliance standards through Unity Catalog.
n
Design integrations between Databricks and cloud storage, databases, APIs, BI platforms and enterprise applications.
n
Define platform security standards covering identity, networking, encryption, secrets management and role-based access.
n
Implement CI/CD, infrastructure-as-code and automated deployment practices for Databricks workloads.
n
Optimise Spark jobs, clusters, SQL workloads and resource utilisation for performance and cost efficiency.
n
Support the architecture and operationalisation of machine learning and generative AI solutions.
n
Troubleshoot complex platform, integration and performance issues.
n
Create reusable architecture patterns, standards and technical documentation.
n
Collaborate with data engineers, AI/ML teams, security specialists, cloud engineers and business stakeholders.
n
Provide technical leadership, architecture assurance and mentoring to engineering teams.
n
Develop platform roadmaps and drive enterprise adoption of Databricks.
n
n
Mandatory Skills and Experience
n
n




Extensive experience designing enterprise data platforms and lakehouse architectures.
n
Strong hands-on Databricks architecture and implementation experience.
n
Advanced knowledge of Apache Spark, PySpark, Spark SQL and performance optimisation.
n
Strong experience with Delta Lake and Unity Catalog.
n
Experience designing ETL/ELT pipelines and distributed data-processing solutions.
n
Robust cloud architecture experience across Azure, AWS or GCP.
n
Experience with cloud storage, networking, identity and security services.
n
Knowledge of data governance, lineage, access controls and regulatory compliance.
n
Experience implementing CI/CD and infrastructure-as-code using tools such as Terraform, Azure DevOps, GitHub Actions or Jenkins.
n
Strong understanding of SQL, Python and data-modelling principles.
n
Experience integrating Databricks with enterprise data and analytics platforms.
n
Strong stakeholder-management and technical-leadership capabilities.
n
n
Highly Desirable
n
n
Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional.
n
Databricks Certified Generative AI Engineer or related architecture certification.
n
Experience with MLflow, Feature Store, Model Serving and MLOps.
n
Experience designing AI, machine-learning, GenAI or agentic AI solutions.
n
Knowledge of streaming technologies such as Kafka, Structured Streaming or Event Hubs.
n
Experience delivering regulated, government or large-enterprise data platforms.
n
n
Paxus values diversity and welcomes applications from Indigenous Australians, people from diverse cultural and linguistic backgrounds and people living with a disability. If you require an adjustment to the recruitment process, including the application form in an alternate format, please contact me on the above contact details.
#J-*****-Ljbffr

📌 Databricks Architect (New South Wales)
🏢 Paxus - Technology + Digital Talent
📍 New South Wales