Initial 12-Month Contract (Robust Extension Potential)Competitive Daily RateBrisbane Based | Hybrid Working ArrangementsSenior AI Test AnalystAre you an experienced Test Analyst who understands the difference between using AI to support testing and testing AI itself?
We're seeking a Senior AI Test Analyst to join a major AI portfolio focused on evaluating enterprise AI solutions used at scale.
This role sits at the intersection of traditional software testing and AI evaluation, making it ideal for someone with strong QA foundations who can critically assess AI behaviour, reliability, safety, and performance in real-world scenarios.This role is perfect for a Test Analyst who can evaluate AI as the product under test and enjoys investigating edge cases, challenging model behaviour, and identifying where AI systems can fail.Key ResponsibilitiesDesign and execute test strategies across both traditional software solutions and AI-enabled productsDevelop AI evaluation frameworks, benchmark datasets, and structured testing approachesAssess AI outputs for accuracy, reliability, consistency, safety, and hallucinationsPerform adversarial testing, prompt variation analysis, and AI guardrail validationConduct integration, regression, exploratory, and non-functional testing across enterprise environmentsFacilitate UAT, defect triage, root cause analysis, and formal test reporting for governance activitiesKey Skills7+ years' experience in software testing, QA,



or test engineering within complex enterprise environmentsDemonstrated experience testing AI-enabled products, LLM applications, or Generative AI solutionsStrong understanding of risk-based testing, exploratory testing, and quality assurance methodologiesExperience evaluating AI outputs using experimentation, investigation, and structured assessment techniquesAbility to analyse failures across prompts, retrieval layers, application logic, and underlying AI modelsStrong stakeholder engagement skills with the ability to communicate complex findings to technical and non-technical audiencesHighly DesirableExperience testing LLM-based applications or enterprise AI platforms in production environmentsKnowledge of hallucination testing, prompt injection testing, model drift, and AI safety evaluationExposure to AI evaluation tools such as DeepEval, Promptfoo, Ragas, TruLens, Langfuse, or ArizePython, automation, API testing, and data analysis experienceISTQB CT-AI certification or equivalent AI testing credentialsHigher education, Azure OpenAI, Canvas/LTI, or WCAG accessibility testing experienceThis is a fantastic opportunity for a naturally curious tester who enjoys experimentation, challenging assumptions, and uncovering unexpected behaviours.
If you've tested AI products and know how to measure quality beyond a simple pass/fail outcome, we'd love to hear from you.Whizdom is a 100% Australian-owned recruitment company and a Veteran Employer of Choice (highest tier), proudly supporting government and commercial organisations across Australia.
#J-*****-Ljbffr

📌 Test Analyst - Ai (Brisbane)
🏢 Whizdom
📍 Brisbane