Annotate unstructured data, including support chats, customer inquiries, reviews, call recordings, transcripts, images, and documents. Perform classification, named entity recognition (NER), intent extraction, sentiment analysis, and evaluation of conversations and LLM-generated responses Develop annotation guidelines, label taxonomies, and instructions covering examples and edge cases. Configure annotation projects, create golden datasets, and design quality control tasks Use LLMs (GPT, Claude, Gemini, and open-source models) for pre-labeling, automated quality evaluation (LLM-as-a-judge), and synthetic data generation. Compare AI-generated labels with human annotations and implement human-in-the-loop workflows Monitor annotation quality using inter-annotator agreement metrics (Cohen's/Fleiss' kappa, Krippendorff's alpha, agreement rate), golden set accuracy, annotation speed, and cost efficiency Coordinate annotators, conduct calibration sessions, provide feedback, and evaluate individual performance Manage external annotation vendors and platforms (Toloka, Scale AI, Labelbox, Appen, or similar), including vendor selection, pilot projects, requirements, SLAs, budgets, deadlines, quality control, and deliverable acceptance Manage datasets throughout their lifecycle, including versioning, storage, deduplication, class balancing, train/validation/test splits, and documentation (data cards) Ensure compliance with data privacy requirements, including anonymization, NDAs, and access management for external vendors Collaborate with Data Science and ML Engineering teams to define data requirements, analyze model errors,



and prioritize additional annotation tasks using active learning and hard examples Requirements 1.5-2+ years of experience in data annotation or preparing datasets for machine learning, including unstructured data such as text, audio, and images Hands-on experience with annotation tools such as Label Studio, CVAT, Prodigy, Doccano, or similar platforms, including project setup, annotation templates, and data export Experience managing crowdsourcing platforms or annotation vendors, including task assignment, quality assurance, deliverable acceptance, and budget management Experience using LLMs for data annotation through web interfaces and APIs, including prompt engineering, few-shot prompting, structured JSON outputs, and quality evaluation Knowledge of annotation quality and inter-annotator agreement metrics (kappa, alpha, precision/recall on golden datasets), with the ability to interpret results and make quality-related decisions Working knowledge of Python for scripting and automation, including Pandas, JSON/CSV/JSONL processing, API calls, and basic data validation SQL skills for querying, filtering, and aggregating data Ability to write clear, structured annotation guidelines and technical instructions English proficiency at B2 level or above for documentation,



communication with international vendors, and working with English-language datasets Nice to have experience preparing datasets for LLM training and evaluation, including SFT conversations, RLHF, preference data, red teaming, and AI assistant response evaluation; experience with data annotation in iGaming, betting, e-commerce, FinTech, or customer support environments; experience with speech transcription and annotation, including speaker diarization, ASR error analysis, and tools such as Whisper; understanding of fundamental ML concepts, including train/test splits, data leakage, and the impact of label noise on model performance; experience with active learning or weak supervision; experience annotating multilingual datasets or managing multilingual annotation teams The prospect to work on large-scale products and contribute to their development A flexible work environment built on autonomy, ownership, and trust, without micromanagement The option to work fully remotely or from a modern office Medical insurance coverage in countries where the company operates An annual $2,000 benefits allowance after the probation period, which can be used for health insurance upgrades, language learning, and sports activities Partial reimbursement for professional training, courses, and participation in industry conferences Access to a corporate digital library Company-funded weekly online sessions with a psychologist Relocation opportunities to countries where the company has offices, with relocation assistance available after the probation period, subject to approval

📌 Data Annotator IT (Brisbane)
🏢 01.tech
📍 Brisbane