Job Description

We are seeking a highly skilled LLM Pre-training & Distributed Systems Engineer. This role is essential for orchestrating large-scale machine learning training runs and optimizing distributed infrastructure. The ideal candidate will have a deep understanding of GPU clusters and extensive experience in system engineering to ensure efficient and reliable training processes.

n
n

Responsibilities:

n
n

n
n
- Orchestrate distributed training runs across 1,000+ GPUs using PyTorch, DeepSpeed, or Megatron-LM.
n
n
- Optimize networking (InfiniBand/RDMA) and memory management to prevent out-of-memory errors.
n
n
- Automate checkpointing and failure recovery during month-long training runs.
n
n

n
n

Required Skills:

n
n

n
n
- Deep expertise in 3D parallelism (Data, Tensor, Pipeline).
n
n
- Experience managing SLURM or Kubernetes-based GPU clusters.
n
n
- Solid systems engineering background (C++, CUDA, Python).
n
n

n

📌 LLM Pre-training & Distributed Engineer (AI Infrastructure) (Melbourne)
🏢 Hyphen Connect
📍 Melbourne