Profectus is seeking an experienced AI Test and Evaluation and Assurance Specialist to support the safe and responsible adoption of AI-enabled capability within an Australian Government/Commonwealth context.
The role provides independent, evidence based assessment of AI-enabled systems, covering performance, security, reliability and operational suitability, alongside the governance and assurance products that support capability, acquisition and operational decisions.
The work includes assessing technical evidence and vendor claims, overseeing evaluation activities delivered by third parties, and advising senior stakeholders on risk and suitability.
Key Responsibilities
Support AI assurance and test, evaluation, verification and validation activities across AI-enabled systems
Assess technical evidence, performance claims, limitations and risks associated with AI-enabled systems
Contribute to the planning, coordination and oversight of evaluation activities delivered by industry, research organisations and other Commonwealth groups
Support the client to act as an informed customer and assurance authority for AI activities delivered by third parties
Develop, maintain and review AI governance and assurance artefacts, including use cases, system cards and model cards
Review AI related policies, submissions, technical papers and proposals and advise on their implications
Support the implementation and continuous improvement of AI governance and assurance arrangements
Analyse complex technical, operational and policy issues and provide evidence based advice to decision makers
Translate technical findings into clear and actionable advice, reports and decision support products for senior stakeholders
Identify risks, issues and opportunities associated with AI-enabled capability and recommend management strategies
Engage with stakeholders across Commonwealth, industry,



academia and government to support AI assurance and capability activities
Capture and transfer corporate knowledge and lessons learned to support workforce continuity
Required Experience
Tertiary qualification in engineering, computer science, artificial intelligence, data science, machine learning, systems engineering, mathematics, statistics or another relevant STEM discipline, or equivalent qualified experience
Minimum five years experience developing, implementing, managing, assuring, testing, evaluating or governing AI, machine learning or other advanced technology solutions
Experience applying engineering, systems engineering, risk, assurance, operational analysis or test and evaluation methods to complex real world problems
Experience planning, conducting, supporting or overseeing assurance, verification, validation, test and evaluation or experimentation activities
Sound understanding of AI and machine learning systems, including the ability to assess technical documentation, performance claims, evidence and risks
Experience developing, reviewing or applying governance, assurance, policy, standards or risk management approaches in a technology, engineering or capability setting
Ability to interpret policy, governance and regulatory requirements and translate them into practical implementation approaches and advice
Experience preparing reports, briefs, submissions or advice for senior stakeholders and decision makers
Experience engaging and collaborating with stakeholder groups across technical, operational and governance functions




Postgraduate qualification in artificial intelligence, machine learning, data science, systems engineering, test and evaluation or operational analysis
Experience delivering AI, machine learning, data science or advanced analytics within Commonwealth Government, Commonwealth industry or national security
Experience with generative AI, large language models, retrieval augmented generation, computer vision, autonomy or advanced analytics
Experience supporting verification, validation, assurance, certification, accreditation or test and evaluation activities
Experience acting as a customer representative, technical authority, assurance authority, evaluator or independent verification and validation function
Experience assessing vendor or supplier technical performance, capability claims and solution effectiveness
Experience developing or implementing AI governance, responsible AI, risk management, compliance, privacy, security or ethical AI approaches
Experience with cloud based AI and machine learning platforms such as Microsoft Azure AI, AWS or Google Cloud
Experience with AI and machine learning tools such as TensorFlow, PyTorch or Scikit-learn, and proficiency in Python or similar languages
Relevant industry certifications in AI, machine learning, cloud AI services, systems engineering or related disciplines
Security Clearance Requirements
Must be an Australian Citizen
Must have an active AGSVA NV1 clearance
Must comply with ITAR requirements
Working Arrangements
Onsite
Why Profectus
Profectus partners with Australian Government agencies to deliver trusted ICT capability across complex delivery programs.
We take a long term, relationship driven approach, supporting our people across the full lifecycle of their engagement while contributing to outcomes of national importance.
#J-*****-Ljbffr

📌 Test And Evaluation Specialist (Canberra)
🏢 Profectus
📍 Canberra