Lead technical architecture and solution design for a large FSI customer Transactions Ingestion Data Platform Real Time and Batch
Interact with customer's data engineering team to provide opinionated guidance on Transactions Ingestion Data platform architecture, best practices, schema designs (Spanner/BigQuery), and solution development
Independently develop and unit test ingestion and processing pipelines (e.g., Kafka to Spanner/BigQuery via Dataflow, GCS to Spanner/BigQuery), and handover developed solutions to customer's technical team
Perform pair programming with customer's data engineers
Project manage sprint-based engagements, leading sprint planning, backlog management, scope definition, risks/issues, and timelines for successful MVP delivery
Collaborate with cross-disciplinary project team, play the role of workstream or overall project tech lead depending on size and scale of project / program
Minimum Qualifications
In-depth knowledge and experience of GCP data and analytics technologies
Designing Large scale Transaction Data Platform architecture for (a) migrating open source or public cloud data platforms to GCP cloud native services (b) designing greenfield data platforms on GCP
Define solution architecture and detailed design, and perform hands-on implementation for
data pipelines for batch and event driven ingestion and processing of data from a variety of sources such as on-prem files, on-prem databases, APIs, etc.




real-time data ingestion and event-driven processing pipelines from Confluent Kafka and GCS to Spanner and BigQuery
Automatic transpilation of legacy code (HIVE, Teradata, python logic etc.) to BQ SQL
CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform
Orchestration setup and Cloud Composer DAG development for data pipeline workflows
Data governance solutioning using GCP governance tooling (Dataplex, Data Catalog) Experience applying Generative AI technologies and integrations in enterprise environments.
Tools and languages experience Required
Must have:
Must have: programming knowledge and willingness to be hands-on - Python, Java
Good to Have:
Experience with dbt or Dataform, Terraform in context of data pipelines
Experience with open source ecosystem and distributions such as Hadoop, Spark, Cloudera/Hortonworks and frameworks and tech such as SPARK, Oozie, Kafka, HBASE
Understanding and experience with NoSQL databases such as HBASE, MongoDB
Knowledge of cloud databases such as Spanner, BigTable, Cloud SQL, DB migrations Certification
Positive to have GCP Data engineering
LTM is proud to be an equal opportunity employer. We are committed to equal employment prospect regardless of race, ethnicity, nationality, gender, gender-identity, gender expression, language, age, sexual orientation, religion, marital status, veteran status, socio-economic status, disability or any other characteristic protected by applicable law.

📌 Lead Data Engineer (New South Wales)
🏢 LTM
📍 New South Wales