Lead Data Engineer – GCP

Location: Sydney, NSW preferred | Melbourne, VIC also considered
Employment Type: 12-Month Fixed-Term Contract (FTC)
Industry: Financial Services / FSI
Work Rights: Australian Citizens or candidates with full working rights in Australia will be considered

About the Role

We are looking for an experienced Lead Data Engineer to join a large-scale Financial Services project focused on building and modernising a Transactions Ingestion Data Platform supporting both real-time and batch data processing.

This is a hands-on technical leadership role, combining solution architecture, data engineering, customer engagement and technical delivery.

Key Responsibilities

- Lead technical architecture and solution design for large-scale transaction data platforms.
- Design and develop real-time and batch data ingestion and processing pipelines.
- Build solutions using GCP Dataflow, BigQuery, Cloud Spanner, Pub/Sub and Confluent Kafka.
- Develop event-driven pipelines from Kafka/GCS into Spanner and BigQuery.
- Design batch ingestion pipelines from files, databases, APIs and other sources.
- Provide technical guidance and best-practice recommendations to customer data engineering teams.
- Define solution architecture, detailed designs and hands-on implementations.
- Perform pair programming and knowledge transfer with customer engineers.
- Lead sprint planning, backlog management, scope, risks and technical delivery.
- Work as a workstream or overall technical lead depending on project scale.
- Develop and maintain CI/CD pipelines using Cloud Build, Artifact Registry and Terraform.
- Develop Cloud Composer DAGs and orchestrate data workflows using Cloud Workflows.
- Optimise BigQuery queries and data processing performance.
- Support data governance initiatives using GCP governance tools such as Dataplex and Data Catalog.




- Contribute to enterprise Generative AI initiatives and integrations.

Essential Skills & Experience

- Strong hands-on experience with GCP data and analytics technologies.
- Experience designing large-scale transaction data platforms.
- Strong experience with:
- GCP Dataflow
- BigQuery
- Cloud Spanner
- Pub/Sub
- Cloud Composer
- Cloud Workflows
- Confluent Kafka
- Cloud Run
- Cloud Build
- Terraform
- Strong programming skills in Python and Java.
- Experience building both real-time/event-driven and batch data pipelines.
- Experience with cloud data platform migrations and/or greenfield GCP implementations.
- Experience with CI/CD and infrastructure-as-code for data workloads.
- Strong solution design and technical architecture capabilities.
- Excellent communication and customer-facing consulting skills.

Desirable Experience

- dbt or Dataform.
- Hadoop, Spark, Cloudera/Hortonworks and other open-source data platforms.
- HBASE or MongoDB.
- Spanner, Bigtable, Cloud SQL and database migration projects.
- Legacy code/data platform modernisation.
- Generative AI technologies and enterprise integrations.
- GCP Data Engineering certification.

What We're Looking For

We are looking for someone who is hands-on technically but also comfortable leading architecture, engaging directly with customers and guiding other engineers.

If you have strong GCP Data Engineering experience and have worked on large-scale transaction or financial data platforms, we’d like to hear from you.

Apply now with your updated CV to discuss the chance.

#GCP #GoogleCloud #DataEngineering #DataEngineer #LeadDataEngineer #BigQuery #Dataflow #CloudSpanner #PubSub #Kafka #Terraform #Python #Java #DataArchitecture #FinancialServices #FSI #SydneyJobs #MelbourneJobs #AustraliaJobs

Pay: $96,200.00 – $150,000.00 per year

Work Location: In person

📌 Lead Data Engineer – GCP (Sydney)
🏢 Shoolin Consultancy
📍 Sydney