Role Description & Requirements:

● Lead technical architecture and solution design for a large FSI customer Transactions Ingestion Data Platform Real Time and Batch

● Interact with customer’s data engineering team to provide opinionated guidance on Transactions Ingestion Data platform architecture, best practices, schema designs (Spanner/BigQuery), and solution development

● Independently develop and unit test ingestion and processing pipelines (e.g., Kafka to Spanner/BigQuery via Dataflow, GCS to Spanner/BigQuery), and handover developed solutions to customer’s technical team

● Perform pair programming with customer’s data engineers

● Project manage sprint-based engagements, leading sprint planning, backlog management, scope definition, risks/issues, and timelines for successful MVP delivery

● Collaborate with cross-disciplinary project team, play the role of workstream or overall project tech lead depending on size and scale of project / program

Minimum Qualifications

● Expertise Required

○ In-depth knowledge and experience of GCP data and analytics technologies

○ Designing Large scale Transaction Data Platform architecture for (a) migrating open source or public cloud data platforms to GCP cloud native services (b)



designing greenfield data platforms on GCP

○ Define solution architecture and detailed design, and perform hands-on implementation for

■ data pipelines for batch and event driven ingestion and processing of data from a variety of sources such as on-prem files, on-prem databases, APIs, etc.

■ real-time data ingestion and event-driven processing pipelines from

Confluent Kafka and GCS to Spanner and BigQuery

○ Automatic transpilation of legacy code (HIVE, Teradata, python logic etc.) to BQ SQL

○ BQ query performance optimisation

○ CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform

○ Orchestration setup and Cloud Composer DAG development for data pipeline workflows

○ Data governance solutioning using GCP governance tooling (Dataplex, Data Catalog)

○ Experience applying Generative AI technologies and integrations in enterprise settings.

Tools and languages experience Required

Must have

● GCP Dataflow, Cloud Spanner, BigQuery, Pub/Sub, Cloud Composer, Cloud Workflows, Confluent Kafka, Cloud Run, Cloud Build, Terraform

● Must have: programming knowledge and willingness to be hands-on - Python, Java

📌 Senior Data Engineer Sydney
🏢 Nityo Infotech
📍 Sydney