Key Responsibilities
Design,develop, and maintain prompt frameworks, including system prompts,few-shot examples, role-based prompts, and reasoning workflows forproduction-grade LLM applications.
Build andmanage automated evaluation frameworks to measure model performance,accuracy, latency, and regression across releases.
Conductstructured A/B testing across prompt variations, model versions, andconfiguration settings to optimize task-specific outcomes.
Convertproduct requirements and edge-case scenarios into effective promptinstructions, personas, constraints, and guardrails.
Partnerwith ML engineers and product teams to determine when prompt engineeringis sufficient versus when fine-tuning, RAG, or other AI architectures arerequired.
Create andmaintain a centralized prompt repository with version control, documentation,and performance benchmarks for organizational reuse.
Leadred-teaming and adversarial testing exercises to identify jailbreak risks,hallucinations, and model vulnerabilities.
Defineevaluation criteria, annotation guidelines, and quality standards toensure consistency, safety, and reliability of AI-generated outputs.
Mentorengineers and stakeholders on prompt engineering best practices,evaluation methodologies, and the capabilities and limitations of modernLLMs.
Presentprompt strategies, benchmark results, and trade-off analyses to product,engineering, and leadership teams.
Applyadvanced prompting techniques, including chain-of-thought, zero-shot,few-shot, and role-based prompting.
Driveprompt testing, evaluation,



benchmarking, and continuous optimizationefforts.
Improve AIresponse quality through systematic assessment, tuning, and refinement.
Managecontext handling and prompt orchestration for complex AI workflows.
Technical Skills
Strongprogramming and scripting skills in
one or more contemporary programminglanguages
(C#, Python, Javascript).
Experiencebuilding automation, evaluation pipelines, APIs, or AI-poweredapplications using enterprise-grade development practices.
Hands-onexperience with LLM platforms, prompt engineering, model evaluation, andAI application development.
Familiaritywith prompt orchestration frameworks, vector databases, RAG architectures,and AI agent workflows.
Understandingof data analysis, experimentation, benchmarking, and performanceoptimization.
Experiencewith version control systems, CI/CD pipelines, and cloud platforms.
Strongknowledge of REST APIs, JSON, and system integration patterns.
Ability tocollaborate effectively with software engineers, data scientists, andproduct teams to deliver production-ready AI solutions.
Experience Requirements
4-7 yearsof combined experience in NLP, AI/ML products, software development,technical writing, or related fields.
At least 2years of direct, hands‐on prompt engineering experience with productionLLM applications.
Proventrack record of owning and managing prompt systems end-to-end, from designand implementation through monitoring and optimization in production.
#J-*****-Ljbffr

📌 Prompt Engineer (New South Wales)
🏢 BlackCube Labs
📍 New South Wales