Job Description
Job Description
n
n
Lead AI & Data Platform Engineer
n
n
n
n
Location:
Melbourne (Hybrid Work Model)
n
n
Salary:
$180,000 – $210,000 + Super
n
n
Industry:
Financial Services
n
n
Contract Type:
Permanent / Full-Time
n
n
n
n
About the Role
n
n
We are seeking a high-caliber
Lead AI & Data Platform Engineer
to spearhead the evolution of our next-generation data architecture. In this role, you will bridge the gap between advanced Data Engineering and production-grade MLOps, designing resilient, petabyte-scale systems within a highly regulated Financial Services environment.
n
n
You will be our internal
Databricks Expert
, driving the strategy and deployment of AI Agentic workloads, advanced analytics pipelines, and secure cloud infrastructure.
n
n
Key Responsibilities
n
n
1. Databricks Platform Excellence
n
n
n
n
Architect and maintain our enterprise
Lakehouse infrastructure
utilising Delta Lake and Medallion topology
(Bronze/Silver/Gold
layers).
n
n
Enforce robust data governance and secure discovery across the enterprise using
Unity Catalog
.
n
n
Build, monitor, and scale production pipelines via
Delta Live Tables
.
n
n
n
n
2. Production AI & MLOps Infrastructure
n
n
n
n
Deploy, monitor, and scale production-grade ML and AI Agentic workloads.
n
n
Operationalise LLM and Generative AI pipelines utilizing
Databricks Mosaic AI
,
MLflow
,
Langchain
, and
Vector Databases
.
n
n
Manage corporate feature stores and optimize model serving endpoints.
n
n
n
n
3. AWS Cloud Architecture & Infrastructure as Code (IaC)
n
n
n
n
Design secure,



high-performance cloud environments using Terraform.
n
n
Manage intricate AWS configurations including S3 optimization, VPC peering, AWS PrivateLink, and granular IAM roles.
n
n
Integrate cloud-native AI services like AWS Bedrock seamlessly into our data ecosystem.
n
n
n
n
4. Performance Tuning & Streaming
n
n
n
n
Conduct advanced Spark optimization to handle high-concurrency, petabyte-scale data pipelines.
n
n
Tune GPU clusters specifically optimized for heavy AI/ML training and inference workloads.
n
n
Architect real-time streaming pipelines leveraging Kinesis or Kafka.
n
n
n
n
Your Profile
n
n
To be successful in this role, you will bring a blend of elite engineering skills and a deep understanding of modern data-ops.
n
n
Technical Requirements:
n
n
n
n
Core Languages:
Mastery of
PySpark
,
Python
,
SQL
, and
Scala
.
n
n
AI Frameworks:
Strong familiarity with
PyTorch
,
TensorFlow
, or
Hugging Face
.
n
n
DataOps & CI/CD:
Proven experience embedding automated testing and CI/CD pipelines into data deployments.
n
n
Industry Experience:
Prior experience within Financial Services or a similarly regulated workplace is highly desirable (understanding of data privacy, compliance, and risk frameworks).
n
n
n
n
What's on Offer
n
n
n
n
Competitive Compensation:
$180K – $210K base salary, depending on experience.
n
n
Hybrid Flexibility:
A healthy balance of remote work and collaborative time in our modern Melbourne CBD office.
n
n
Cutting-Edge Tech Stack:
Total backing to utilize the absolute latest in Generative AI, Mosaic AI, and cloud-native tools.
n
n
n

📌 New Job Opening: Principal Data Engineer In Docklands, Vic (Melbourne)
🏢 The Professional Search Group
📍 Melbourne