Profectus is seeking an experienced AI Test and Evaluation and Assurance Specialist to support the safe and responsible adoption of AI-enabled capability within an Australian Government/Commonwealth context. The role provides independent, evidence based assessment of AI-enabled systems, covering performance, security, reliability and operational suitability, alongside the governance and assurance products that support capability, acquisition and operational decisions. The work includes assessing technical evidence and vendor claims, overseeing evaluation activities delivered by third parties, and advising senior stakeholders on risk and suitability.

Key Responsibilities

Support AI assurance and test, evaluation, verification and validation activities across AI-enabled systems

Assess technical evidence, performance claims, limitations and risks associated with AI-enabled systems

Contribute to the planning, coordination and oversight of evaluation activities delivered by industry, research organisations and other Commonwealth groups

Support the client to act as an informed customer and assurance authority for AI activities delivered by third parties

Develop, maintain and review AI governance and assurance artefacts, including use cases, system cards and model cards

Review AI related policies, submissions, technical papers and proposals and advise on their implications

Support the implementation and continuous improvement of AI governance and assurance arrangements

Analyse complex technical, operational and policy issues and provide evidence based advice to decision makers

Translate technical findings into clear and actionable advice, reports and decision support products for senior stakeholders

Identify risks, issues and opportunities associated with AI-enabled capability and recommend management strategies

Engage with stakeholders across Commonwealth, industry,



academia and government to support AI assurance and capability activities

Capture and transfer corporate knowledge and lessons learned to support workforce continuity

Required Experience

Tertiary qualification in engineering, computer science, artificial intelligence, data science, machine learning, systems engineering, mathematics, statistics or another relevant STEM discipline, or equivalent skilled experience

Minimum five years experience developing, implementing, managing, assuring, testing, evaluating or governing AI, machine learning or other advanced technology solutions

Experience applying engineering, systems engineering, risk, assurance, operational analysis or test and evaluation methods to complex real world problems

Experience planning, conducting, supporting or overseeing assurance, verification, validation, test and evaluation or experimentation activities

Sound understanding of AI and machine learning systems, including the ability to assess technical documentation, performance claims, evidence and risks

Experience developing, reviewing or applying governance, assurance, policy, standards or risk management approaches in a technology, engineering or capability setting

Ability to interpret policy, governance and regulatory requirements and translate them into practical implementation approaches and advice

Experience preparing reports, briefs, submissions or advice for senior stakeholders and decision makers

Experience engaging and collaborating with stakeholder groups across technical, operational and governance functions





Postgraduate qualification in artificial intelligence, machine learning, data science, systems engineering, test and evaluation or operational analysis

Experience delivering AI, machine learning, data science or advanced analytics within Commonwealth Government, Commonwealth industry or national security

Experience with generative AI, large language models, retrieval augmented generation, computer vision, autonomy or advanced analytics

Experience supporting verification, validation, assurance, certification, accreditation or test and evaluation activities

Experience acting as a customer representative, technical authority, assurance authority, evaluator or independent verification and validation function

Experience assessing vendor or supplier technical performance, capability claims and solution effectiveness

Experience developing or implementing AI governance, responsible AI, risk management, compliance, privacy, security or ethical AI approaches

Experience with cloud based AI and machine learning platforms such as Microsoft Azure AI, AWS or Google Cloud

Experience with AI and machine learning tools such as TensorFlow, PyTorch or Scikit-learn, and proficiency in Python or similar languages

Relevant industry certifications in AI, machine learning, cloud AI services, systems engineering or related disciplines

Security Clearance Requirements

Must be an Australian Citizen

Must have an active AGSVA NV1 clearance

Must comply with ITAR requirements

Working Arrangements

Onsite

Why Profectus Profectus partners with Australian Government agencies to deliver trusted ICT capability across complex delivery programs. We take a long term, relationship driven approach, supporting our people across the full lifecycle of their engagement while contributing to outcomes of national importance.

#J-18808-Ljbffr

📌 Test and Evaluation Specialist (Canberra)
🏢 Profectus
📍 Canberra