Data Engineer – Spark & Kafka

We are looking for a Data Engineer with experience in Spark and Kafka to operate with one of our large Federal Government Clients. As a Data Engineer, you will have the chance to work with cutting‑edge technologies and collaborate with cross‑functional teams to ensure successful data integration.

Responsibilities

- Maintain and develop Spark pipelines to process large‑scale data sets.
- Ingest tables into our data lake from Kafka using batch processing, micro‑batching, or real‑time processing techniques.
- Run Spark jobs for ingestion, curation, and extraction of data.
- Alter schemas and configs to meet business requirements and ensure data quality.
- Attend meetings regarding data ingestion and collaborate with cross‑functional teams to ensure successful data integration.
- Troubleshoot and resolve bugs in the data processing pipelines.
- Monitor performance and troubleshoot issues using logging and monitoring tools.
- Document best practices and develop standards for data processing pipelines.
- Stay up‑to‑date with emerging technologies and industry trends in big data processing and analytics.

Requirements

- Bachelor’s or Master’s degree in Computer Science, Software Engineering,



or a related field.
- 3+ years of experience in data engineering, software development, or a related field.
- Strong programming skills in Scala, Python, or Java.
- Expertise in Spark, Kafka, and big data processing concepts.
- Experience with data storage solutions such as Hadoop HDFS or Amazon S3, and data serialization formats.
- Knowledge of database concepts, data modelling, schema design, and normalisation.
- Familiarity with monitoring and logging tools, and workflow management tools such as Apache Airflow.
- Experience with version control systems such as Git.
- Security clearance required – Baseline (NV1 preferred).

Attributes and Aptitude

- Strong communication, problem‑solving, and collaboration abilities.
- Ability to work independently and in a team‑oriented, collaborative environment.

Benefits

We offer competitive compensation packages and benefits, along with opportunities for growth and advancement within the company. As a member of our team, you will have the chance to work on challenging projects, develop your skills, and make a real impact on our business.

#J-18808-Ljbffr

📌 Data Engineer (Canberra)
🏢 Nuvento
📍 Canberra