Data Engineer with using Apache Spark, Python, Scala.

/n Location: Melbourne

/n Position Type: Contract/n /n

Data Engineer/n As a Data Engineer, you will be responsible for designing, developing, and maintaining robust data infrastructure and pipelines. You will work closely with cross-functional teams to understand business requirements, optimize data workflows, and ensure the reliability and performance of our data systems./n Key Responsibilities:/n /n

- Design, build, and optimize data processing pipelines using Apache Spark, Python, Scala.

/n

- Develop and maintain data ingestion and extraction processes, including streaming data pipelines using Kafka and batch processing workflows.

/n

- Implement performance tuning techniques to optimize data processing and query performance, ensuring scalability and efficiency.

/n

- Collaborate with DevOps teams to deploy and manage data infrastructure on NetApp S3 (very similar to AWS S3), Kubernetes.

/n

- Containerize data applications using Docker and orchestrate deployment using Kubernetes for scalability and reliability.

/n

- Develop and maintain unit tests using frameworks like pytest, junit, to ensure the quality and reliability of data pipelines.

/n

- Implement and adhere to best practices for data governance, security, and compliance.

/n

- Utilize Behavior-Driven Development (BDD) tools like Cucumber and Lettuce to write and execute test scenarios for dataworkflows.

/n





- Stay up to date with emerging technologies and industry trends and evaluate their potential impact on our data infrastructure and processes.

/n Requirements:/n /n

- Proven experience in designing and building scalable data pipelines using Apache Spark, Python, and Scala.

/n

- Strong understanding of data warehousing concepts, ETL processes, and data modelling techniques.

/n

- Experience with performance tuning and optimization of Spark jobs and SQL queries.

/n

- Familiarity with stream processing frameworks like Kafka and messaging systems.

/n

- Hands-on experience with containerization and orchestration tools like Docker and Kubernetes.

/n

- Experience with unit testing frameworks like pytest, junit, and BDD tools like Cucumber and Lettuce.

/n

- Excellent problem-solving skills and attention to detail.

/n

- Robust communication and collaboration skills, with the ability to work effectively in a team environment

/n /n

/n ​​​​​​​When you apply, you voluntarily consent to the disclosure, collection and use of your personal data for employment/recruitment and related purposes in accordance with the Tech Aalto Privacy Policy, a copy of which is published at Tech Aalto’s website (https://www.techaalto.com/privacy/)/n

Confidentiality is assured, and only shortlisted candidates will be notified for interviews./n

/n /n

📌 Data Engineer with using Apache Spark, Python, Scala. (Melbourne)
🏢 Tech Aalto
📍 Melbourne