Job Role: Senior Databricks & AI Architect
Location:
Sydney, Australia
Employment Type:
Contract
Experience:
8+ Years
Role Overview
We are looking for a
Senior Databricks & AI Architect
to lead the design and implementation of modern data and AI solutions. The role will focus on Databricks architecture, data engineering, platform optimisation, FinOps, and emerging AI capabilities including Agentic AI, LLMs, and RAG.
The ideal candidate will have strong hands-on Databricks experience, excellent knowledge of Spark and Delta Lake, and experience delivering data and AI solutions within Insurance or Financial Services.
Key Responsibilities
Design and govern end-to-end data architectures using the
Databricks Data Intelligence Platform
.
Lead the design of scalable
Spark, Delta Lake, and Databricks SQL
solutions.
Develop and standardise
Medallion Architecture
and reusable ETL/ELT frameworks.
Design data models using
Star Schema, Snowflake Schema, and Data Vault 2.0
.
Architect AI solutions using
Generative AI, Agentic AI, LLMs, and RAG
patterns.
Design multi-agent workflows and LLM orchestration using
LangChain, LlamaIndex
, or similar frameworks.
Develop AI solutions for insurance use cases such as
claims, underwriting, policy management, and risk assessment
.
Drive Databricks
performance optimisation and FinOps
, including cluster policies, instance selection, Photon, Z-Ordering, and Liquid Clustering.
Implement cost monitoring, attribution, and optimisation strategies across Databricks workloads.
Ensure data governance, security, access control, and lineage using
Databricks Unity Catalog
.
Work closely with business, engineering, data science, security, and architecture teams to deliver scalable solutions.
Provide technical leadership,



architectural guidance, and mentoring to data engineering and AI teams.
Required Skills & Experience
Bachelor's or Master's degree in
Computer Science, Data Science, Information Technology, Engineering
, or a related discipline.
Databricks Certified Data Engineer Professional
or
Databricks Certified Machine Learning Professional
is preferred.
8+ years
of experience in Data Engineering, Data Architecture, or Solution Architecture.
3+ years of hands-on Databricks experience
, including technical/architectural leadership.
Robust expertise in
Apache Spark, PySpark and/or Scala
.
Strong hands-on experience with
Databricks, Delta Lake, and Databricks SQL
.
Experience with
Medallion Architecture and enterprise data modelling
.
Strong knowledge of
Star Schema, Snowflake Schema, and/or Data Vault 2.0
.
Proven experience delivering
AI/ML and Generative AI solutions
.
Strong understanding of
Agentic AI, LLM orchestration, and RAG
.
Experience with
LangChain and/or LlamaIndex
.
Strong knowledge of Databricks
FinOps, performance tuning, and cost optimisation
.
Hands-on experience with
Photon, Z-Ordering, Liquid Clustering, cluster policies, and workload optimisation
.
Strong understanding of
Unity Catalog, data governance, security, and access controls
.
Experience working with
structured and unstructured data
.
Strong communication and stakeholder management skills.
Domain Experience
Experience in
Insurance or Financial Services
is highly preferred.
Understanding of insurance data and processes such as
policies, premiums, claims, underwriting, actuarial models, and risk assessment
is desirable.
Experience working in regulated enterprise environments is an advantage.
#J-*****-Ljbffr

📌 Senior Databricks & Ai Architect (New South Wales)
🏢 Recmatrix Consulting
📍 New South Wales