We are seeking a Data and AI Architect to design shared data architecture for Agentic AI solutions on Databricks. You will shape secure ingestion, modeling and governance for IoT and time-series data and enable RAG retrieval patterns using Delta Lake, Unity Catalog, Spark and Vector Search. You simplify complexity and guide teams toward scalable, reusable designs.

You will define the shared data architecture for Agentic AI solutions built on Databricks. Your focus is creating a secure, scalable and reusable foundation for ingestion, modeling, governance and retrieval patterns that enable AI-ready data products.

You bring a pragmatic systems mindset, and you can translate complex data and AI requirements into explicit architecture decisions. You collaborate across engineering, product and security teams, challenge assumptions constructively and manage risks early.

Responsibilities
Design shared data and AI architecture on Databricks across Agentic AI solutions
Define data modeling standards for IoT and time-series domains aligned to analytics and RAG needs
Architect ingestion and processing pipelines using Spark and Delta Lake
Establish governance, lineage and access controls with Unity Catalog
Define secure data access patterns for internal teams and downstream consumption




Design RAG retrieval patterns including indexing strategy and datasets for Vector Search
Produce architecture decision records, reference designs and reusable templates
Partner with stakeholders to align scope, manage risks and drive implementation quality

Requirements
Databricks architecture experience in production settings
Data modeling expertise including event, time-series, dimensional and canonical modeling patterns
Ingestion pipeline experience across batch and streaming patterns
IoT and time-series data experience including scale, retention and query optimization
Delta Lake expertise including table design, performance tuning and lifecycle management
Databricks Unity Catalog experience covering governance, lineage and fine-grained access control
Apache Spark expertise for large-scale processing and optimization
Data governance and security experience including policy design and secure-by-default architecture

Nice to have
Experience implementing Vector Search and managing embedding lifecycle
Familiarity with retrieval quality, evaluation signals and RAG tuning considerations
Experience enabling shared data products across multiple teams

📌 Data And Ai Architect Melbourne
🏢 EPAM Systems
📍 Melbourne