17 Nov
|
ALOIS UK
|
Perth
About The Role
We are looking for a Senior Data Engineer to lead the design and delivery of secure, scalable and cost-effective data solutions across on-premises and cloud platforms.
You will provide technical leadership across complex data engineering initiatives, designing batch and near-real-time pipelines and integrating data into trusted data assets for reporting, analytics and downstream applications. You will also help establish reusable engineering patterns, improve platform reliability and mentor other data engineers.
Key responsibilities
- Lead the design and delivery of complex, production-grade data pipelines and data assets across Azure, Databricks and on-premises big-data platforms.
- Translate product priorities, business outcomes and technical requirements into practical engineering solutions.
- Own or lead solution design for ingestion, transformation, modelling, storage and data consumption.
- Ensure solutions meet non-functional requirements, including security, reliability, performance, scalability, maintainability and usability.
- Optimise cloud and platform usage to meet performance requirements while managing compute and operating costs.
- Establish and promote reusable data engineering patterns, standards and practices.
- Review technical designs, source-to-target mappings, test strategies, implementation plans and operational documentation.
- Lead data profiling, wrangling and analysis for recent and complex datasets.
- Work with subject-matter experts to understand business rules and ensure data assets are fit for purpose.
- Design data products appropriate to different consumption patterns, including reporting, analytics and downstream applications.
- Coordinate planning, governance, architecture, engineering, testing and release activities across multiple teams.
- Ensure pipelines meet agreed service levels, dependencies, monitoring and operational requirements.
- Lead the resolution of complex production issues and drive permanent improvements.
- Provide technical mentoring,
coaching and peer review for data engineers.
- Contribute to engineering roadmaps, delivery estimates, risk management and continuous improvement.
- Promote secure, governed and high-quality data practices across the delivery community.
What you will bring
Essential Experience And Skills
- Seven or more years of experience in data engineering, data warehousing or a related field.
- Strong experience designing and delivering end-to-end data engineering solutions.
- Advanced SQL and Python, with practical experience using Spark or equivalent distributed processing technologies.
- At least two years of experience designing and delivering solutions with Azure Databricks.
- Experience delivering cloud-based analytics or reporting pipelines.
- Strong experience with batch processing and practical exposure to near-real-time or streaming solutions.
- Experience with data profiling, source-to-target mapping, ETL/ELT development, testing, implementation and SQL optimisation.
- Experience with orchestration tools such as Airflow, Control‑M or equivalent.
- Experience applying CI/CD, version control, automated testing and release-management practices.
- Experience designing solutions for structured and unstructured data, including JSON, XML and Avro.
- Demonstrated ability to provide technical leadership, mentor engineers and make sound engineering decisions.
Desirable experience
- Azure data services and cloud data warehouse technologies.
- Cloudera, Hive, Impala, HDFS, HBase or Hue.
- Kafka, Spark Streaming or other streaming frameworks.
- REST APIs, microservices or Data-as-a-Service architectures.
- Experience with high-volume batch or near-real-time processing.
- Experience delivering data solutions across complex business domains.
- Experience with financial or management-reporting processes.
- Relevant Azure, Databricks or other industry certifications.
Qualifications
A degree in Computer Science, Mathematics, Statistics, Information Management, Finance, Economics or a related discipline is desirable. Equivalent practical expe
📌 Senior Data Engineer (Perth)
🏢 ALOIS UK
📍 Perth