Permanent | Canberra or Melbourne | Up to $140k incl. super | Baseline clearance | December start
One of Australia's most active Data & AI practices is growing its sovereign delivery team and needs a streaming specialist for secure, cleared work. Amazon Kinesis is the primary streaming backbone on this program, so this is a hands-on role moving high-volume, real-time data into a Databricks Lakehouse.
The role
You'll build and run the real-time layer: Kinesis Data Streams and Firehose feeding Databricks through Spark Structured Streaming into Delta tables. You'll own latency, reliability and cost, build monitoring in CloudWatch, and work alongside the batch engineers so streaming and batch pipelines fit together cleanly.
- Build and support real-time pipelines on Kinesis Data Streams and Kinesis Data Firehose
- Develop streaming solutions in Databricks using Spark Structured Streaming
- Ingest and manage streaming data in Delta tables
- Tune pipelines for performance, latency, reliability and cost
- Build monitoring and alerting with AWS CloudWatch
- Write clean, testable Python and SQL, and troubleshoot production issues
What you'll bring
- Hands-on production experience with Kinesis Data Streams and/or Kinesis Data Firehose (essential)
- Spark Structured Streaming in production
- Hands-on Databricks, including Delta Lake
- Robust Python and SQL
- AWS fundamentals, particularly IAM, S3 and CloudWatch
- Sound design instincts for scalable, reliable pipelines and strong troubleshooting skills
- Australian citizen with active Baseline clearance (NV1 also welcome)
Good to have
- Unity Catalog
- Kafka or MSK
What's on offer
- Up to $140k including super, Consultant level
- Permanent role, starting 1 December
- Canberra or Melbourne
- Structured career development, local and international career paths, and flexible work arrangements