Salary: up to $13 per hour

Job Type: Full-time

Annotate unstructured data, including support chats, customer inquiries, reviews, call recordings, transcripts, images, and documents. Perform classification, named entity recognition (NER), intent extraction, sentiment analysis, and evaluation of conversations and LLM-generated responsesDevelop annotation guidelines, label taxonomies, and instructions covering examples and edge cases. Configure annotation projects, create golden datasets, and design quality control tasksUse LLMs (GPT, Claude, Gemini, and open-source models) for pre-labeling, automated quality evaluation (LLM-as-a-judge), and synthetic data generation. Compare AI-generated labels with human annotations and implement human-in-the-loop workflowsMonitor annotation quality using inter-annotator agreement metrics (Cohen's/Fleiss' kappa, Krippendorff's alpha, agreement rate), golden set accuracy, annotation speed, and cost efficiencyCoordinate annotators, conduct calibration sessions, provide feedback, and evaluate individual performanceManage external annotation vendors and platforms (Toloka, Scale AI, Labelbox, Appen, or similar), including vendor selection, pilot projects, requirements, SLAs, budgets, deadlines, quality control, and deliverable acceptanceManage datasets throughout their lifecycle, including versioning, storage, deduplication, class balancing, train/validation/test splits, and documentation (data cards)Ensure compliance with data privacy requirements, including anonymization, NDAs, and access management for external vendorsCollaborate with Data Science and ML Engineering teams to define data requirements, analyze model errors,



and prioritize additional annotation tasks using active learning and hard examplesRequirements1.5-2+ years of experience in data annotation or preparing datasets for machine learning, including unstructured data such as text, audio, and imagesHands-on experience with annotation tools such as Label Studio, CVAT, Prodigy, Doccano, or similar platforms, including project setup, annotation templates, and data exportExperience managing crowdsourcing platforms or annotation vendors, including task assignment, quality assurance, deliverable acceptance, and budget managementExperience using LLMs for data annotation through web interfaces and APIs, including prompt engineering, few-shot prompting, structured JSON outputs, and quality evaluationKnowledge of annotation quality and inter-annotator agreement metrics (kappa, alpha, precision/recall on golden datasets), with the ability to interpret results and make quality-related decisionsWorking knowledge of Python for scripting and automation, including Pandas, JSON/CSV/JSONL processing, API calls, and basic data validationSQL skills for querying, filtering, and aggregating dataAbility to write clear, structured annotation guidelines and technical instructionsEnglish proficiency at B2 level or above for documentation,



communication with international vendors, and working with English-language datasetsNice to have experience preparing datasets for LLM training and evaluation, including SFT conversations, RLHF, preference data, red teaming, and AI assistant response evaluation; experience with data annotation in iGaming, betting, e-commerce, FinTech, or customer support environments; experience with speech transcription and annotation, including speaker diarization, ASR error analysis, and tools such as Whisper; understanding of fundamental ML concepts, including train/test splits, data leakage, and the impact of label noise on model performance; experience with active learning or weak supervision; experience annotating multilingual datasets or managing multilingual annotation teamsThe opportunity to work on large-scale products and contribute to their developmentA flexible work environment built on autonomy, ownership, and trust, without micromanagementThe option to work fully remotely or from a contemporary officeMedical insurance coverage in countries where the company operatesAn annual $2,000 benefits allowance after the probation period, which can be used for health insurance upgrades, language learning, and sports activitiesPartial reimbursement for professional training, courses, and participation in industry conferencesAccess to a corporate digital libraryCompany-funded weekly online sessions with a psychologistRelocation opportunities to countries where the company has offices, with relocation assistance available after the probation period, subject to approval #J-18808-Ljbffr

📌 Data Annotator IT & Data Development (Gold Coast)
🏢 01.tech
📍 Gold Coast