Job Role: Senior Databricks & AI Architect
Location:Sydney, Australia
Employment Type:Contract
Experience:8+ Years
Role Overview
We are looking for aSenior Databricks & AI Architectto lead the design and implementation of modern data and AI solutions.
The role will focus on Databricks architecture, data engineering, platform optimisation, FinOps, and emerging AI capabilities including Agentic AI, LLMs, and RAG.
The ideal candidate will have robust hands-on Databricks experience, excellent knowledge of Spark and Delta Lake, and experience delivering data and AI solutions within Insurance or Financial Services.
Key Responsibilities
Design and govern end-to-end data architectures using theDatabricks Data Intelligence Platform.
Lead the design of scalableSpark, Delta Lake, and Databricks SQLsolutions.
Develop and standardiseMedallion Architectureand reusable ETL/ELT frameworks.
Design data models usingStar Schema, Snowflake Schema, and Data Vault 2.0.
Architect AI solutions usingGenerative AI, Agentic AI, LLMs, and RAGpatterns.
Design multi-agent workflows and LLM orchestration usingLangChain, LlamaIndex, or similar frameworks.
Develop AI solutions for insurance use cases such asclaims, underwriting, policy management, and risk assessment.
Drive Databricksperformance optimisation and FinOps, including cluster policies, instance selection, Photon, Z-Ordering, and Liquid Clustering.
Implement cost monitoring, attribution, and optimisation strategies across Databricks workloads.
Ensure data governance, security, access control, and lineage usingDatabricks Unity Catalog.
Work closely with business, engineering, data science, security, and architecture teams to deliver scalable solutions.




Provide technical leadership, architectural guidance, and mentoring to data engineering and AI teams.
Required Skills & Experience
Bachelor's or Master's degree inComputer Science, Data Science, Information Technology, Engineering, or a related discipline.
Databricks Certified Data Engineer ProfessionalorDatabricks Certified Machine Learning Professionalis preferred.
8+ yearsof experience in Data Engineering, Data Architecture, or Solution Architecture.
3+ years of hands-on Databricks experience, including technical/architectural leadership.
Strong expertise inApache Spark, PySpark and/or Scala.
Strong hands-on experience withDatabricks, Delta Lake, and Databricks SQL.
Experience withMedallion Architecture and enterprise data modelling.
Strong knowledge ofStar Schema, Snowflake Schema, and/or Data Vault 2.0.
Proven experience deliveringAI/ML and Generative AI solutions.
Strong understanding ofAgentic AI, LLM orchestration, and RAG.
Experience withLangChain and/or LlamaIndex.
Strong knowledge of DatabricksFinOps, performance tuning, and cost optimisation.
Hands-on experience withPhoton, Z-Ordering, Liquid Clustering, cluster policies, and workload optimisation.
Strong understanding ofUnity Catalog, data governance, security, and access controls.
Experience working withstructured and unstructured data.
Strong communication and stakeholder management skills.
Domain Experience
Experience inInsurance or Financial Servicesis highly preferred.
Understanding of insurance data and processes such aspolicies, premiums, claims, underwriting, actuarial models, and risk assessmentis desirable.
Experience working in regulated enterprise environments is an advantage.
#J-*****-Ljbffr

📌 Senior Databricks & Ai Architect (New South Wales)
🏢 Recmatrix Consulting
📍 New South Wales