- Lead technical architecture and solution design for a large FSI customer Transactions Ingestion Data Platform Real Time and Batch
- Interact with customer's data engineering team to provide opinionated guidance on Transactions Ingestion Data platform architecture, best practices, schema designs (Spanner/BigQuery), and solution development
- Independently develop and unit test ingestion and processing pipelines (e.g., Kafka to Spanner/BigQuery via Dataflow, GCS to Spanner/BigQuery), and handover developed solutions to customer's technical team
- Perform pair programming with customer's data engineers
- Project manage sprint-based engagements, leading sprint planning, backlog management, scope definition, risks/issues, and timelines for successful MVP delivery
- Collaborate with cross-disciplinary project team, play the role of workstream or overall project tech lead depending on size and scale of project / program

Minimum Qualifications

- In-depth knowledge and experience of GCP data and analytics technologies
- Designing Large scale Transaction Data Platform architecture for (a) migrating open source or public cloud data platforms to GCP cloud native services (b) designing greenfield data platforms on GCP
- Define solution architecture and detailed design, and perform hands-on implementation for
- data pipelines for batch and event driven ingestion and processing of data from a variety of sources such as on-prem files, on-prem databases, APIs, etc.




- real-time data ingestion and event-driven processing pipelines from Confluent Kafka and GCS to Spanner and BigQuery
- Automatic transpilation of legacy code (HIVE, Teradata, python logic etc.) to BQ SQL
- CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform
- Orchestration setup and Cloud Composer DAG development for data pipeline workflows
- Data governance solutioning using GCP governance tooling (Dataplex, Data Catalog) Experience applying Generative AI technologies and integrations in enterprise environments.

Tools and languages experience Required

Must have:

- Must have: programming knowledge and willingness to be hands-on - Python, Java

Good to Have:

- Experience with dbt or Dataform, Terraform in context of data pipelines
- Experience with open source ecosystem and distributions such as Hadoop, Spark, Cloudera/Hortonworks and frameworks and tech such as SPARK, Oozie, Kafka, HBASE
- Understanding and experience with NoSQL databases such as HBASE, MongoDB
- Knowledge of cloud databases such as Spanner, BigTable, Cloud SQL, DB migrations Certification
- Good to have GCP Data engineering

LTM is proud to be an equal chance employer. We are committed to equal employment opportunity regardless of race, ethnicity, nationality, gender, gender-identity, gender expression, language, age, sexual orientation, religion, marital status, veteran status, socio-economic status, disability or any other characteristic protected by applicable law.

#J-18808-Ljbffr

📌 Lead Data Engineer (Sydney)
🏢 LTM
📍 Sydney