Key Responsibilities

- Design,
develop, and maintain prompt frameworks, including system prompts,
few-shot examples, role-based prompts, and reasoning workflows for
production-grade LLM applications.

- Build and
manage automated evaluation frameworks to measure model performance,
accuracy, latency, and regression across releases.

- Conduct
structured A/B testing across prompt variations, model versions, and
configuration settings to optimize task-specific outcomes.

- Convert
product requirements and edge-case scenarios into effective prompt
instructions, personas, constraints, and guardrails.

- Partner
with ML engineers and product teams to determine when prompt engineering
is sufficient versus when fine-tuning, RAG, or other AI architectures are
required.

- Create and
maintain a centralized prompt repository with version control, documentation,
and performance benchmarks for organizational reuse.

- Lead
red-teaming and adversarial testing exercises to identify jailbreak risks,
hallucinations, and model vulnerabilities.

- Define
evaluation criteria, annotation guidelines, and quality standards to
ensure consistency, safety, and reliability of AI-generated outputs.

- Mentor
engineers and stakeholders on prompt engineering best practices,
evaluation methodologies, and the capabilities and limitations of modern
LLMs.

- Present
prompt strategies, benchmark results, and trade-off analyses to product,
engineering, and leadership teams.

- Apply
advanced prompting techniques, including chain-of-thought, zero-shot,
few-shot, and role-based prompting.

- Drive




prompt testing, evaluation, benchmarking, and continuous optimization
efforts.

- Improve AI
response quality through systematic assessment, tuning, and refinement.

- Manage
context handling and prompt orchestration for complex AI workflows.

Technical Skills

- Strong
programming and scripting skills in one or more modern programming
languages(C#, Python, Javascript).

- Experience
building automation, evaluation pipelines, APIs, or AI-powered
applications using enterprise-grade development practices.

- Hands-on
experience with LLM platforms, prompt engineering, model evaluation, and
AI application development.

- Familiarity
with prompt orchestration frameworks, vector databases, RAG architectures,
and AI agent workflows.

- Understanding
of data analysis, experimentation, benchmarking, and performance
optimization.

- Experience
with version control systems, CI/CD pipelines, and cloud platforms.

- Robust
knowledge of REST APIs, JSON, and system integration patterns.

- Ability to
collaborate effectively with software engineers, data scientists, and
product teams to deliver production-ready AI solutions.

Experience Requirements

- 4-7 years
of combined experience in NLP, AI/ML products, software development,
technical writing, or related fields.

- At least 2
years of direct, hands-on prompt engineering experience with production
LLM applications.

- Proven
track record of owning and managing prompt systems end-to-end, from design
and implementation through monitoring and optimization in production.

📌 Prompt Engineer (Sydney)
🏢 Important Company of the Sector
📍 Sydney