A Bachelor's or Higher Degree is the minimum entry required for the position
Proficiency with
AWS SageMaker Unified Studio
, including its various components
Discover
,
Build
and
Govern
.
Experience with development
in Sagemaker IDE
and
Application Jupyter lab
, Spaces and Partner AI Apps). Data Analysis and Integrations ( Query Editor, Visual ETL Jobs and Data Processing jobs)
Orchestration of workflows and ML Pipelines
Understanding and expertise to work with
ML
and
Gen AI tools
available in unified studio are ad on values.
Setting up projects and data governance in
SageMaker Unified studio Advanced Data Ingestion
&
Processing
Real time & Batch): Framework Development: Proven ability to design, develop, and implement highly reusable and adaptable data ingestion frameworks capable of handling diverse source types (e.g., databases, APIs, message queues, file systems).
Low Latency Real time: Deep expertise in building real time data pipelines with stringent sub second latency requirements, utilizing services like (
AWS Kinesis Data Streams/Firehose
),
Apache Kafka,
or similar streaming technologies. Batch Processing: Experience with robust batch ingestion patterns and tools (e.g.,
AWS Glue, Apache Spark
) for efficient processing of larger datasets.
Data Transformation: Robust skills in
designing
and
implementing
efficient data transformation
logic for both streaming and batch data.
Programming and Scripting: Advanced proficiency in
Python
, particularly for developing scalable data ingestion and export frameworks, API integration, and extensive use of the
AWS SDK
Experience with performance optimization techniques for Python applications in data intensive environments.
Familiarity with other relevant languages (e.g., Scala, Spark) for high performance streaming applications is beneficial.
AWS Services
for Data and Infrastructure: In
depth knowledge
of core




AWS services: Lambda, S3, DynamoDB, CloudWatch, SQS, SNS,
and
API Gateway
.
Strong understanding of AWS networking
VPC, security groups, private endpoints) and IAM for secure, fine grained access control.
Databases Relational and
No SQL
Expertise with
Amazon RDS (
Relational Database Service) for both real time data ingestion and efficient batch export. This includes optimising database performance, connection pooling, and transaction management for high throughput, low latency operations.
Proficiency with
Amazon Redshift
for large scale data warehousing, including data loading strategies (e.g., COPY command), query optimisation, and managing
Redshift clusters
for analytical workloads and batch export.
Experience with
No SQ
L databases such as
Amazon DynamoDB, MongoDB
for high performance, low latency data storage and retrieval, particularly for real time applications and feature serving.
Orchestration and Workflow Management:
Experience with
AWS Managed Apache Airflow
for orchestrating
complex data pipelines
, scheduling batch jobs, and managing dependencies between ingestion, processing, and export tasks. Ability to write, deploy, and manage
Airflow
DAGs
Directed Acyclic Graphs) for robust workflow
automation. Monitoring
and
Observability
Real time Heartbeat Export): Ability to design and implement comprehensive real time monitoring solutions, including custom metrics, detailed logging, and tracing.
Experience with AWS CloudWatch for collecting, analysing, and acting on operational data, specifically for generating and exporting "heartbeat" signals to external systems or dashboards. Knowledge of setting up proactive alerts and automated notifications for system health, performance degradation, and data pipeline anomalies.
Software Development Practices & Architecture: Strong understanding of software engineering principles, design patterns, and architectural best practices
#J-*****-Ljbffr

📌 Aws Data Engineer (New South Wales)
🏢 Tech Mahindra
📍 New South Wales