Job Description
Key Responsibilities
n
n
Design,develop, and maintain prompt frameworks, including system prompts,few-shot examples, role-based prompts, and reasoning workflows forproduction-grade LLM applications.
n
Build andmanage automated evaluation frameworks to measure model
performance,accuracy,
latency, and regression across releases.
n
Conductstructured A/B testing across prompt variations, model versions, andconfiguration settings to optimize task-specific outcomes.
n
Convertproduct requirements and edge-case scenarios into effective
promptinstructions,
personas, constraints, and guardrails.
n
Partnerwith ML engineers and product teams to determine when prompt engineeringis sufficient versus when fine-tuning, RAG, or other AI architectures arerequired.
n
Create andmaintain a centralized prompt repository with version control, documentation,and performance benchmarks for organizational reuse.
n
Leadred-teaming and adversarial testing exercises to identify jailbreak
risks,hallucinations,
and model vulnerabilities.
n
Defineevaluation criteria, annotation guidelines, and quality standards toensure consistency, safety, and reliability of AI-generated outputs.
n
Mentorengineers and stakeholders on prompt engineering best
practices,evaluation
methodologies, and the capabilities and limitations of modernLLMs.
n
Presentprompt strategies, benchmark results, and trade-off analyses to
product,engineering,
and leadership teams.
n
Applyadvanced prompting techniques, including chain-of-thought, zero-shot,few-shot, and role-based prompting.
n
Driveprompt testing, evaluation,



benchmarking, and continuous
optimizationefforts.
n
Improve AIresponse quality through systematic assessment, tuning, and refinement.
n
Managecontext handling and prompt orchestration for complex AI workflows.
n
n
Technical Skills
n
n
Strongprogramming and scripting skills in one or more contemporary
programminglanguages(C#,
Python, Javascript).
n
Experiencebuilding automation, evaluation pipelines, APIs, or AI-
poweredapplications
using enterprise-grade development practices.
n
Hands-onexperience with LLM platforms, prompt engineering, model evaluation, andAI application development.
n
Familiaritywith prompt orchestration frameworks, vector databases, RAG architectures,and AI agent workflows.
n
Understandingof data analysis, experimentation, benchmarking, and
performanceoptimization.
n
Experiencewith version control systems, CI/CD pipelines, and cloud platforms.
n
Strongknowledge of REST APIs, JSON, and system integration patterns.
n
Ability tocollaborate effectively with software engineers, data scientists, andproduct teams to deliver production-ready AI solutions.
n
n
Experience Requirements
n
n
4-7 yearsof combined experience in NLP, AI/ML products, software
development,technical
writing, or related fields.
n
At least 2years of direct, hands‐on prompt engineering experience with productionLLM applications.
n
Proventrack record of owning and managing prompt systems end-to-end, from designand implementation through monitoring and optimization in production.
n
#J-*****-Ljbffr

📌 Prompt Engineer (New South Wales)
🏢 BlackCube Labs
📍 New South Wales