Key Responsibilities - Design,develop, and maintain prompt frameworks, including system prompts,few-shot examples, role-based prompts, and reasoning workflows forproduction-grade LLM applications.

- Build andmanage automated evaluation frameworks to measure model performance,accuracy, latency, and regression across releases.
- Conductstructured A/B testing across prompt variations, model versions, andconfiguration settings to optimize task-specific outcomes.
- Convertproduct requirements and edge-case scenarios into effective promptinstructions, personas, constraints, and guardrails.
- Partnerwith ML engineers and product teams to determine when prompt engineeringis sufficient versus when fine-tuning, RAG, or other AI architectures arerequired.
- Create andmaintain a centralized prompt repository with version control, documentation,and performance benchmarks for organizational reuse.
- Leadred-teaming and adversarial testing exercises to identify jailbreak risks,hallucinations, and model vulnerabilities.
- Defineevaluation criteria, annotation guidelines, and quality standards toensure consistency, safety, and reliability of AI-generated outputs.
- Mentorengineers and stakeholders on prompt engineering best practices,evaluation methodologies, and the capabilities and limitations of modernLLMs.
- Presentprompt strategies, benchmark results, and trade-off analyses to product,engineering, and leadership teams.
- Applyadvanced prompting techniques, including chain-of-thought, zero-shot,few-shot, and role-based prompting.
- Driveprompt testing,



evaluation, benchmarking, and continuous optimizationefforts.
- Improve AIresponse quality through systematic assessment, tuning, and refinement.
- Managecontext handling and prompt orchestration for complex AI workflows.

Technical

Skills - Strongprogramming and scripting skills in one or more up-to-date programminglanguages(C#, Python, Javascript).

- Experiencebuilding automation, evaluation pipelines, APIs, or AI-poweredapplications using enterprise-grade development practices.
- Hands-onexperience with LLM platforms, prompt engineering, model evaluation, andAI application development.
- Familiaritywith prompt orchestration frameworks, vector databases, RAG architectures,and AI agent workflows.
- Understandingof data analysis, experimentation, benchmarking, and performanceoptimization.
- Experiencewith version control systems, CI/CD pipelines, and cloud platforms.
- Strongknowledge of REST APIs, JSON, and system integration patterns.
- Ability tocollaborate effectively with software engineers, data scientists, andproduct teams to deliver production-ready AI solutions.

Experience Requirements - 4-7 yearsof combined experience in NLP, AI/ML products, software development,technical writing, or related fields.

- At least 2years of direct, hands‑on prompt engineering experience with productionLLM applications.
- Proventrack record of owning and managing prompt systems end-to-end, from designand implementation through monitoring and optimization in production.

📌 Prompt Engineer (City of Sydney)
🏢 BlackCube Labs
📍 City of Sydney