Job Description

Job Description n
n

Lead AI & Data Platform Engineer

n
n

n
n
- Location: Melbourne (Hybrid Work Model)
n
n
- Salary: $180,000 – $210,000 + Super
n
n
- Industry: Financial Services
n
n
- Contract Type: Permanent / Full-Time
n
n

n
n

About the Role

n
n

We are seeking a high-caliber Lead AI & Data Platform Engineer to spearhead the evolution of our next-generation data architecture. In this role, you will bridge the gap between advanced Data Engineering and production-grade MLOps, designing resilient, petabyte-scale systems within a highly regulated Financial Services environment.

n
n

You will be our internal Databricks Expert , driving the strategy and deployment of AI Agentic workloads, advanced analytics pipelines, and secure cloud infrastructure.

n
n

Key Responsibilities

n
n

1. Databricks Platform Excellence

n
n

n
n
- Architect and maintain our enterprise Lakehouse infrastructure utilising Delta Lake and Medallion topology (Bronze/Silver/Gold layers).
n
n
- Enforce robust data governance and secure discovery across the enterprise using Unity Catalog .
n
n
- Build, monitor, and scale production pipelines via Delta Live Tables .
n
n

n
n

2. Production AI & MLOps Infrastructure

n
n

n
n
- Deploy, monitor, and scale production-grade ML and AI Agentic workloads.
n
n
- Operationalise LLM and Generative AI pipelines utilizing Databricks Mosaic AI , MLflow , Langchain , and Vector Databases .
n
n
- Manage corporate feature stores and optimize model serving endpoints.
n
n

n
n

3. AWS Cloud Architecture & Infrastructure as Code (IaC)

n
n

n
n
- Design secure,



high-performance cloud environments using Terraform.
n
n
- Manage intricate AWS configurations including S3 optimization, VPC peering, AWS PrivateLink, and granular IAM roles.
n
n
- Integrate cloud-native AI services like AWS Bedrock seamlessly into our data ecosystem.
n
n

n
n

4. Performance Tuning & Streaming

n
n

n
n
- Conduct advanced Spark optimization to handle high-concurrency, petabyte-scale data pipelines.
n
n
- Tune GPU clusters specifically optimized for heavy AI/ML training and inference workloads.
n
n
- Architect real-time streaming pipelines leveraging Kinesis or Kafka.
n
n

n
n

Your Profile

n
n

To be successful in this role, you will bring a blend of elite engineering skills and a deep understanding of modern data-ops.

n
n

Technical Requirements:

n
n

n
n
- Core Languages: Mastery of PySpark , Python , SQL , and Scala .
n
n
- AI Frameworks: Strong familiarity with PyTorch , TensorFlow , or Hugging Face .
n
n
- DataOps & CI/CD: Proven experience embedding automated testing and CI/CD pipelines into data deployments.
n
n
- Industry Experience: Prior experience within Financial Services or a similarly regulated environment is highly desirable (understanding of data privacy, compliance, and risk frameworks).
n
n

n
n

What’s on Offer

n
n

n
n
- Competitive Compensation: $180K – $210K base salary, depending on experience.
n
n
- Hybrid Flexibility: A healthy balance of remote work and collaborative time in our up-to-date Melbourne CBD office.
n
n
- Cutting-Edge Tech Stack: Total backing to utilize the absolute latest in Generative AI, Mosaic AI, and cloud-native tools.
n
n

n

📌 New Job Opening: Principal Data Engineer in Docklands, VIC (Melbourne)
🏢 The Professional Search Group
📍 Melbourne