Senior AI Test Analyst

? Initial 12-Month Contract (Strong Extension Potential)

? Competitive Daily Rate

? Brisbane Based | Hybrid Working Arrangements

? Cutting-Edge Enterprise AI Program

Are you a seasoned Test Analyst with hands-on experience evaluating AI, LLM, and machine learning solutions? This is a rare opportunity to help shape how enterprise AI is tested, governed, and safely deployed at scale. You'll work across a portfolio of high-profile AI initiatives, bridging traditional QA practices with contemporary AI evaluation methodologies.

Key Responsibilities:

- Design and maintain AI evaluation frameworks, golden datasets, and benchmark testing suites
- Assess AI outputs for accuracy, reasoning quality, hallucinations, bias, and compliance risks
- Conduct adversarial testing including prompt injection, jailbreak, and data leakage scenarios
- Build and manage automated AI evaluation pipelines using modern testing frameworks
- Perform root cause analysis across prompts, retrieval layers, integrations, and AI models
- Lead UAT activities and produce formal test evidence for project governance and stage gates

Key Skills:

- 7+ years' experience in software testing, QA, or test engineering within enterprise environments
- 2+ years evaluating AI, LLM, or machine learning solutions using structured AI testing methodologies




- Strong Python skills and experience with API testing and test automation practices
- Practical understanding of LLMs, embeddings, context windows, token limits, and model behaviour
- Experience translating technical test outcomes into actionable business recommendations
- Strong defect triage, root cause analysis, and governance documentation experience

Highly Desirable:

- ISTQB CT-AI certification or equivalent AI testing credentials
- LLM red teaming experience and familiarity with OWASP frameworks
- Experience with DeepEval, Promptfoo, Ragas, TruLens, Langfuse, or Arize
- Exposure to Jira, Zephyr Scale, Confluence, and CI/CD pipelines
- Experience within higher education environments, including Canvas or LTI integrations
- Accessibility testing knowledge aligned to WCAG 2.2 AA and Azure OpenAI environments

This role offers the chance to work on some of the most innovative AI initiatives currently being delivered, helping ensure solutions are safe, reliable, accessible, and fit for purpose before reaching thousands of users.

To apply or learn more, reach out at [email protected] or 0480 002 456.

Whizdom is a 100% Australian-owned recruitment company and a Veteran Employer of Choice (highest tier), proudly supporting government and commercial organisations across Australia.

📌 Test Analyst - AI (Brisbane)
🏢 Whizdom
📍 Brisbane