A leading Australian brand is building out the streaming side of a major platform program, and they need a data engineer who can make real time data behave. Think high volume pipelines that have to stay right while the platform around them changes. You'll build and run streaming pipelines on Databricks and AWS, from ingestion through to trusted, analytics ready data.
What you'll do
Build and own streaming pipelines using Databricks Structured Streaming, Auto Loader and Delta Lake
Ingest from Kafka (MSK) and Kinesis into a Bronze, Silver, Gold lakehouse
Keep data correct: dedup, late and out-of-order events, schema changes and secure replays
Monitor and tune pipelines for latency, cost and reliability
What you bring
Strong hands-on Databricks streaming experience in production, not just batch
Solid Python coding experience (software)
Real experience with Kafka or Kinesis
AWS data services such as S3, Glue, Lambda, Athena, EMR and CloudWatch
Comfort in CI/CD and infrastructure as code (Terraform or CDK)
The details
Type: 6 month contract with high multiple extension possibility
Location: Melbourne CBD
#J-*****-Ljbffr