Hi all,

Hope you are doing well.

This is Selvi from Reninfo Systems!.

We do have a new role of with our client, If you are interested please let me know and send me your updated resume.

Role:Data Engineer-Lead

Location:Sydney/Melbourne

Duration:FTC(12 months)

Role 1: Lead Data Engineer

Role Description & Requirements:

- Lead technical architecture and solution design for a large FSI customer Transactions Ingestion Data Platform Real Time and Batch
- Interact with customer’s data engineering team to provide opinionated guidance on Transactions Ingestion Data platform architecture, best practices, schema designs (Spanner/BigQuery), and solution development
- Independently develop and unit test ingestion and processing pipelines (e.g., Kafka to Spanner/BigQuery via Dataflow, GCS to Spanner/BigQuery), and handover developed solutions to customer’s technical team
- Perform pair programming with customer’s data engineers
- Project manage sprint-based engagements, leading sprint planning, backlog management, scope definition, risks/issues, and timelines for successful MVP delivery
- Collaborate with cross-disciplinary project team, play the role of workstream or overall project tech lead depending on size and scale of project / program

Minimum Qualifications

Expertise Required

In-depth knowledge and experience of GCP data and analytics technologies

- Designing Large scale Transaction Data Platform architecture for (a) migrating open source or public cloud data platforms to GCP cloud native services (b) designing greenfield data platforms on GCP
- Define solution architecture and detailed design, and perform hands-on implementation for –





data pipelines for batch and event driven ingestion and processing of data from a variety of sources such as on-prem files, on-prem databases, APIs, etc. real-time data ingestion and event-driven processing pipelines from Confluent Kafka and GCS to Spanner and BigQuery
- Automatic transpilation of legacy code (HIVE, Teradata, python logic etc.) to BQ SQL
- BQ query performance optimisation
- CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform
- Orchestration setup and Cloud Composer DAG development for data pipeline workflows
- Data governance solutioning using GCP governance tooling (Dataplex, Data Catalog)
- Experience applying Generative AI technologies and integrations in enterprise environments.

Tools and languages experience Required

Must have

- GCP Dataflow, Cloud Spanner, BigQuery, Pub/Sub, Cloud Composer, Cloud Workflows, Confluent Kafka, Cloud Run, Cloud Build, Terraform
- Must have: programming knowledge and willingness to be hands-on - Python, Java

Good to Have

- Experience with dbt or Dataform, Terraform in context of data pipelines
- Experience with open source ecosystem and distributions such as Hadoop, Spark, Cloudera/Hortonworks and frameworks and tech such as SPARK, Oozie, Kafka, HBASE
- Understanding and experience with NoSQL databases such as HBASE, MongoDB
- Knowledge of cloud databases such as Spanner, BigTable, Cloud SQL, DB migrations

Certification

1. Positive to have GCP Data engineering

Best Regards

Selvi

Sr Recruitment Consultant

Renaissance Info Systems

Suite 508, Level 5/343 Little Collins St, Melbourne VIC 3000, Australia

E-Mail: [email protected]

Linkedin: https://www.linkedin.com/in/muthamilselvisubramanian/

📌 Senior Data Engineer (Sydney)
🏢 Renaissance InfoSystems
📍 Sydney