Job Description

Artificial Intelligence Engineer n

This range is provided by Tribus. Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.

Base pay range n

A$180,000.00/yr - A$200,000.00/yr

n

Direct message the job poster from Tribus

n

Founder | Recruiter @ tribus - connecting leading technology experts with top financial services firms across APAC | part time recruiter, full-time…

n

Software Engineer - AI

n

LLM | Python | AWS

n

We’re partnering with a fast-growing software company building AI‑driven products used in high‑stakes, real‑world workflows.

n

The focus is on production‑quality AI: systems that must be reliable, measurable, and safe at scale.

n

They’re looking for a Software Engineer with AI experience to join a team responsible for the core AI platform, with a particular emphasis on LLM evaluation, observability, and reliability.

n

This is a hands‑on engineering role, sitting close to product and domain experts, where your work directly influences how AI quality is defined, measured, and enforced in production.

What you’ll work on n

n
- Building and operating LLM evaluation pipelines that assess model quality, robustness, and safety
n
- Defining test sets, metrics, and evaluation workflows, including human‑in‑the‑loop processes where required
n
- Translating product and domain constraints into concrete, testable evaluation criteria
n
- Running and orchestrating distributed evaluation workloads on AWS, including monitoring compute usage
n
- Analysing evaluation results, identifying failure modes, and collaborating on mitigations (prompt changes, data updates,



model selection or fine‑tuning)
n
- Integrating and assessing open‑source and vendor evaluation frameworks, writing glue code where needed
n
- Contributing to the evolution of the AI evaluation and platform architecture
n

What they’re looking for nn
- Experience monitoring and evaluating LLM‑based applications
n
- Hands‑on exposure to LLM evaluation tools, benchmarks, and metrics
n
- Understanding of common LLM failure modes (e.g. hallucination, bias, toxicity, prompt injection)
n
- Experience with cloud ML infrastructure, ideally AWS
n
- Familiarity with distributed workloads (e.g. Ray, AWS Lambda, or similar)
n
- Comfort working with an evolving LLM observability and evaluation stack
n
- Ability to work with non‑ML stakeholders and convert qualitative requirements into quantitative tests
n

Working environment & benefits nn
- Flexible hybrid setup, with twice‑weekly collaboration in a modern CBD office
n
- Strong learning and career development opportunities in a scaling business
n
- Wellness focus including additional leave and gym membership
n
- Collaborative team culture with regular social events
n
- Pool table, snacks, and a genuinely supportive environment
n

n

This role is well suited to engineers who care about AI reliability and correctness, and who want to work on systems where evaluation and safeguards genuinely matter.

n

Must be based in Sydney with full working rights. Remote working or sponsorship is not available for this role.

Seniority level n

Mid‑Senior level

Employment type n

Full‑time

Job function n

Engineering and Information Technology

Industries n

Technology, Information and Media

n

#J-18808-Ljbffr

📌 Artificial Intelligence Engineer (Sydney)
🏢 Tribus
📍 Sydney