Profectus is seeking an experienced AI Test and Evaluation and Assurance Specialist to support the safe and responsible adoption of AI-enabled capability within an Australian Government/Commonwealth context. The role provides independent, evidence based assessment of AI-enabled systems, covering performance, security, reliability and operational suitability, alongside the governance and assurance products that support capability, acquisition and operational decisions. The work includes assessing technical evidence and vendor claims, overseeing evaluation activities delivered by third parties, and advising senior stakeholders on risk and suitability.

Key Responsibilities

- Support AI assurance and test, evaluation, verification and validation activities across AI-enabled systems
- Assess technical evidence, performance claims, limitations and risks associated with AI-enabled systems
- Contribute to the planning, coordination and oversight of evaluation activities delivered by industry, research organisations and other Commonwealth groups
- Support the client to act as an informed customer and assurance authority for AI activities delivered by third parties
- Develop, maintain and review AI governance and assurance artefacts, including use cases, system cards and model cards
- Review AI related policies, submissions, technical papers and proposals and advise on their implications
- Support the implementation and continuous improvement of AI governance and assurance arrangements
- Analyse complex technical, operational and policy issues and provide evidence based advice to decision makers
- Translate technical findings into explicit and actionable advice, reports and decision support products for senior stakeholders
- Identify risks, issues and opportunities associated with AI-enabled capability and recommend management strategies
- Engage with stakeholders across Commonwealth, industry,



academia and government to support AI assurance and capability activities
- Capture and transfer corporate knowledge and lessons learned to support workforce continuity

Required Experience

- Tertiary qualification in engineering, computer science, artificial intelligence, data science, machine learning, systems engineering, mathematics, statistics or another relevant STEM discipline, or equivalent professional experience
- Minimum five years experience developing, implementing, managing, assuring, testing, evaluating or governing AI, machine learning or other advanced technology solutions
- Experience applying engineering, systems engineering, risk, assurance, operational analysis or test and evaluation methods to complex real world problems
- Experience planning, conducting, supporting or overseeing assurance, verification, validation, test and evaluation or experimentation activities
- Sound understanding of AI and machine learning systems, including the ability to assess technical documentation, performance claims, evidence and risks
- Experience developing, reviewing or applying governance, assurance, policy, standards or risk management approaches in a technology, engineering or capability setting
- Ability to interpret policy, governance and regulatory requirements and translate them into practical implementation approaches and advice
- Experience preparing reports, briefs, submissions or advice for senior stakeholders and decision makers
- Experience engaging and collaborating with stakeholder groups across technical, operational and governance functions




- Postgraduate qualification in artificial intelligence, machine learning, data science, systems engineering, test and evaluation or operational analysis
- Experience delivering AI, machine learning, data science or advanced analytics within Commonwealth Government, Commonwealth industry or national security
- Experience with generative AI, large language models, retrieval augmented generation, computer vision, autonomy or advanced analytics
- Experience supporting verification, validation, assurance, certification, accreditation or test and evaluation activities
- Experience acting as a customer representative, technical authority, assurance authority, evaluator or independent verification and validation function
- Experience assessing vendor or supplier technical performance, capability claims and solution effectiveness
- Experience developing or implementing AI governance, responsible AI, risk management, compliance, privacy, security or ethical AI approaches
- Experience with cloud based AI and machine learning platforms such as Microsoft Azure AI, AWS or Google Cloud
- Experience with AI and machine learning tools such as TensorFlow, PyTorch or Scikit-learn, and proficiency in Python or similar languages
- Relevant industry certifications in AI, machine learning, cloud AI services, systems engineering or related disciplines

Security Clearance Requirements

- Must be an Australian Citizen
- Must have an active AGSVA NV1 clearance
- Must comply with ITAR requirements

Working Arrangements

- Onsite

Why Profectus

Profectus partners with Australian Government agencies to deliver trusted ICT capability across complex delivery programs. We take a long term, relationship driven approach, supporting our people across the full lifecycle of their engagement while contributing to outcomes of national importance.

#J-18808-Ljbffr

📌 Test and Evaluation Specialist (Canberra)
🏢 Profectus
📍 Canberra