Job Description
Artificial Intelligence Engineer
n
This range is provided by Tribus. Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.
n
Base pay range
n
A$180,******/yr - A$200,******/yr
n
Direct message the job poster from Tribus
n
Founder | Recruiter @ tribus - connecting leading technology experts with top financial services firms across APAC | part-time recruiter, full-time...
n
Software Engineer - AI
n
LLM | Python | AWS
n
We're partnering with a fast-growing software company building AI‐driven products used in high‐stakes, real‐world workflows.
n
The focus is on production‐quality AI: systems that must be reliable, measurable, and safe at scale.
n
They're looking for a Software Engineer with AI experience to join a team responsible for the core AI platform, with a particular emphasis on LLM evaluation, observability, and reliability.
n
This is a hands‐on engineering role, sitting close to product and domain experts, where your work directly influences how AI quality is defined, measured, and enforced in production.
n
What you'll work on
n
n
Building and operating LLM evaluation pipelines that assess model quality, robustness, and safety
n
Defining test sets, metrics, and evaluation workflows, including human‐in‐the‐loop processes where required
n
Translating product and domain constraints into concrete, testable evaluation criteria
n
Running and orchestrating distributed evaluation workloads on AWS, including monitoring compute usage
n
Analysing evaluation results, identifying failure modes, and collaborating on mitigations (prompt changes, data updates, model selection or fine‐tuning)
n




Integrating and assessing open‐source and vendor evaluation frameworks, writing glue code where needed
n
Contributing to the evolution of the AI evaluation and platform architecture
n
n
What they're looking for
n
n
Experience monitoring and evaluating LLM‐based applications
n
Hands‐on exposure to LLM evaluation tools, benchmarks, and metrics
n
Understanding of common LLM failure modes (e.g. hallucination, bias, toxicity, prompt injection)
n
Experience with cloud ML infrastructure, ideally AWS
n
Familiarity with distributed workloads (e.g. Ray, AWS Lambda, or similar)
n
Comfort working with an evolving LLM observability and evaluation stack
n
Ability to work with non‐ML stakeholders and convert qualitative requirements into quantitative tests
n
n
Working environment & benefits
n
n
Versatile hybrid setup, with twice‐weekly collaboration in a modern CBD office
n
Strong learning and career development opportunities in a scaling business
n
Wellness focus including additional leave and gym membership
n
Collaborative team culture with regular social events
n
Pool table, snacks, and a genuinely supportive environment
n
n
This role is well suited to engineers who care about AI reliability and correctness, and who want to work on systems where evaluation and safeguards genuinely matter.
n
Must be based in Sydney with full working rights. Remote working or sponsorship is not available for this role.
n
Seniority level
n
Mid‐Senior level
n
Employment type
n
Full‐time
n
Job function
n
Engineering and Information Technology
n
Industries
n
Technology, Information and Media
#J-*****-Ljbffr

📌 Artificial Intelligence Engineer (New South Wales)
🏢 Tribus
📍 New South Wales