Job Description

Key Responsibilities

n

n
- Design,develop, and maintain prompt frameworks, including system prompts,few-shot examples, role-based prompts, and reasoning workflows forproduction-grade LLM applications.
n
- Build andmanage automated evaluation frameworks to measure model performance,accuracy, latency, and regression across releases.
n
- Conductstructured A/B testing across prompt variations, model versions, andconfiguration settings to optimize task-specific outcomes.
n
- Convertproduct requirements and edge-case scenarios into effective promptinstructions, personas, constraints, and guardrails.
n
- Partnerwith ML engineers and product teams to determine when prompt engineeringis sufficient versus when fine-tuning, RAG, or other AI architectures arerequired.
n
- Create andmaintain a centralized prompt repository with version control, documentation,and performance benchmarks for organizational reuse.
n
- Leadred-teaming and adversarial testing exercises to identify jailbreak risks,hallucinations, and model vulnerabilities.
n
- Defineevaluation criteria, annotation guidelines, and quality standards toensure consistency, safety, and reliability of AI-generated outputs.
n
- Mentorengineers and stakeholders on prompt engineering best practices,evaluation methodologies, and the capabilities and limitations of modernLLMs.
n
- Presentprompt strategies, benchmark results, and trade-off analyses to product,engineering, and leadership teams.
n
- Applyadvanced prompting techniques, including chain-of-thought, zero-shot,few-shot, and role-based prompting.
n
- Driveprompt testing, evaluation,



benchmarking, and continuous optimizationefforts.
n
- Improve AIresponse quality through systematic assessment, tuning, and refinement.
n
- Managecontext handling and prompt orchestration for complex AI workflows.
n

n

Technical Skills

n

n
- Strongprogramming and scripting skills in one or more contemporary programminglanguages(C#, Python, Javascript).
n
- Experiencebuilding automation, evaluation pipelines, APIs, or AI-poweredapplications using enterprise-grade development practices.
n
- Hands-onexperience with LLM platforms, prompt engineering, model evaluation, andAI application development.
n
- Familiaritywith prompt orchestration frameworks, vector databases, RAG architectures,and AI agent workflows.
n
- Understandingof data analysis, experimentation, benchmarking, and performanceoptimization.
n
- Experiencewith version control systems, CI/CD pipelines, and cloud platforms.
n
- Strongknowledge of REST APIs, JSON, and system integration patterns.
n
- Ability tocollaborate effectively with software engineers, data scientists, andproduct teams to deliver production-ready AI solutions.
n

n

Experience Requirements

n

n
- 4-7 yearsof combined experience in NLP, AI/ML products, software development,technical writing, or related fields.
n
- At least 2years of direct, hands‐on prompt engineering experience with productionLLM applications.
n
- Proventrack record of owning and managing prompt systems end-to-end, from designand implementation through monitoring and optimization in production.
n

#J-18808-Ljbffr

📌 Prompt Engineer (New South Wales)
🏢 BlackCube Labs
📍 New South Wales