10 Nov
|
Avance Consulting
|
Sydney
10 Nov
Avance Consulting
Sydney
We are looking for an experienced GCP Data Engineer with solid hands-on expertise in designing, developing, and supporting scalable data solutions on Google Cloud Platform (GCP).
Key Responsibilities
Design and develop scalable ETL/ELT data pipelines on GCP.
Build batch and streaming data solutions using BigQuery, Cloud Storage, Dataflow, Pub/Sub, and Dataproc.
Develop data transformation and processing solutions using Python, SQL, and Apache Spark.
Design and optimize BigQuery datasets, tables, partitioning, clustering, and complex SQL queries.
Build ingestion frameworks to integrate data from databases, APIs, files, SaaS platforms, and event streams.
Implement workflow orchestration using Cloud Composer / Apache Airflow.
Develop robust data quality,
reconciliation, logging, monitoring, and exception-handling frameworks.
Implement incremental data processing, CDC, and high-volume data ingestion patterns.
Work with CI/CD pipelines and infrastructure automation for deploying GCP data solutions.
Implement appropriate IAM, security, encryption, and data governance controls.
Troubleshoot performance and production issues across data pipelines.
Collaborate with data architects, analysts, business stakeholders, and downstream reporting teams.
📌 Data Engineer Sydney
🏢 Avance Consulting
📍 Sydney