Job Description

Artificial Intelligence Engineer

n

This range is provided by Tribus. Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.

n

Base pay range

n

A$180,000.00/yr - A$200,000.00/yr

n

Direct message the job poster from Tribus

n

Founder | Recruiter @ tribus - connecting leading technology experts with top financial services firms across APAC | part-time recruiter, full-time...

n

Software Engineer - AI

n

LLM | Python | AWS

n

We're partnering with a fast-growing software company building AI‐driven products used in high‐stakes, real‐world workflows.

n

The focus is on production‐quality AI: systems that must be reliable, measurable, and safe at scale.

n

They're looking for a Software Engineer with AI experience to join a team responsible for the core AI platform, with a particular emphasis on LLM evaluation, observability, and reliability.

n

This is a hands‐on engineering role, sitting close to product and domain experts, where your work directly influences how AI quality is defined, measured, and enforced in production.

n

What you'll work on

n

n
- Building and operating LLM evaluation pipelines that assess model quality, robustness, and safety
n
- Defining test sets, metrics, and evaluation workflows, including human‐in‐the‐loop processes where required
n
- Translating product and domain constraints into concrete, testable evaluation criteria
n
- Running and orchestrating distributed evaluation workloads on AWS, including monitoring compute usage
n
- Analysing evaluation results, identifying failure modes, and collaborating on mitigations (prompt changes, data updates, model selection or fine‐tuning)
n




- Integrating and assessing open‐source and vendor evaluation frameworks, writing glue code where needed
n
- Contributing to the evolution of the AI evaluation and platform architecture
n

n

What they're looking for

n

n
- Experience monitoring and evaluating LLM‐based applications
n
- Hands‐on exposure to LLM evaluation tools, benchmarks, and metrics
n
- Understanding of common LLM failure modes (e.g. hallucination, bias, toxicity, prompt injection)
n
- Experience with cloud ML infrastructure, ideally AWS
n
- Familiarity with distributed workloads (e.g. Ray, AWS Lambda, or similar)
n
- Comfort working with an evolving LLM observability and evaluation stack
n
- Ability to work with non‐ML stakeholders and convert qualitative requirements into quantitative tests
n

n

Working environment & benefits

n

n
- Flexible hybrid setup, with twice‐weekly collaboration in a modern CBD office
n
- Strong learning and career development opportunities in a scaling business
n
- Wellness focus including additional leave and gym membership
n
- Collaborative team culture with regular social events
n
- Pool table, snacks, and a genuinely supportive workplace
n

n

This role is well suited to engineers who care about AI reliability and correctness, and who want to work on systems where evaluation and safeguards genuinely matter.

n

Must be based in Sydney with full working rights. Remote working or sponsorship is not available for this role.

n

Seniority level

n

Mid‐Senior level

n

Employment type

n

Full‐time

n

Job function

n

Engineering and Information Technology

n

Industries

n

Technology, Information and Media

#J-18808-Ljbffr

📌 Artificial Intelligence Engineer (New South Wales)
🏢 Tribus
📍 New South Wales