Initial 12-Month Contract (Strong Extension Potential)
Competitive Daily Rate
Brisbane Based | Hybrid Working Arrangements
Senior AI Test Analyst
Are you an experienced Test Analyst who understands the difference betweenusing AI to support testingandtesting AI itself?
We're seeking a Senior AI Test Analyst to join a major AI portfolio focused on evaluating enterprise AI solutions used at scale.
This role sits at the intersection of traditional software testing and AI evaluation, making it ideal for someone with strong QA foundations who can critically assess AI behaviour, reliability, safety, and performance in real-world scenarios.
This role is perfect for a Test Analyst who can evaluate AI as the product under test and enjoys investigating edge cases, challenging model behaviour, and identifying where AI systems can fail.
Key Responsibilities
Design and execute test strategies across both traditional software solutions and AI-enabled products
Develop AI evaluation frameworks, benchmark datasets, and structured testing approaches
Assess AI outputs for accuracy, reliability, consistency, safety, and hallucinations
Perform adversarial testing, prompt variation analysis, and AI guardrail validation
Conduct integration, regression, exploratory, and non-functional testing across enterprise environments
Facilitate UAT, defect triage, root cause analysis, and formal test reporting for governance activities
Key Skills
7+ years' experience in software testing, QA, or test engineering within complex enterprise environments




Demonstrated experience testing AI-enabled products, LLM applications, or Generative AI solutions
Solid understanding of risk-based testing, exploratory testing, and quality assurance methodologies
Experience evaluating AI outputs using experimentation, investigation, and structured assessment techniques
Ability to analyse failures across prompts, retrieval layers, application logic, and underlying AI models
Strong stakeholder engagement skills with the ability to communicate complex findings to technical and non-technical audiences
Highly Desirable
Experience testing LLM-based applications or enterprise AI platforms in production environments
Knowledge of hallucination testing, prompt injection testing, model drift, and AI safety evaluation
Exposure to AI evaluation tools such as DeepEval, Promptfoo, Ragas, TruLens, Langfuse, or Arize
Python, automation, API testing, and data analysis experience
ISTQB CT-AI certification or equivalent AI testing credentials
Higher education, Azure OpenAI, Canvas/LTI, or WCAG accessibility testing experience
This is a fantastic opportunity for a naturally curious tester who enjoys experimentation, challenging assumptions, and uncovering unexpected behaviours.
If you've tested AI products and know how to measure quality beyond a simple pass/fail outcome, we'd love to hear from you.
Whizdom is a 100% Australian-owned recruitment company and a Veteran Employer of Choice (highest tier), proudly supporting government and commercial organisations across Australia.
#J-*****-Ljbffr

📌 Test Analyst - Ai (Queensland)
🏢 Whizdom
📍 Queensland