The candidate will have responsibilities across the following functions:
Data Sourcing and ETL architecture:
End-to-end pipeline design: extraction, normalisation, deduplication, validation, and load; plus cost, performance, and reliability of the whole layer.
Sourcing layer standards: coverage, accuracy, schema design; the rule-vs-LLM standard codified so the team applies it without you.
Cost and performance optimisation: cost ceilings with the token math behind them, incremental processing, recovery design, scheduling.
Extraction systems and frameworks:
Extractor framework: patterns, abstractions, and tooling other engineers ship into; recent extractors are fast to build and reliable to run.
Hard-source extractors: anti-bot defences, JS-heavy rendering, schema drift, low- quality structure; plus the proxy and IP rotation strategy behind them.
Agentic extraction pipelines: rule-based triage, LLM escalation, structured-output validation, retries, human-review queues.
LLM infrastructure and observability:
LLMs as production systems: versioned prompts, labelled eval sets, measured precision and recall, prompt versioning you can roll back, judges debugged on real data.
Eval and observability scaffolding: eval frameworks, prompt versioning, traces, drift detection when a vendor silently updates a model.
Model-choice playbook: cheap models for classification, stronger models for nuanced extraction, frontier models for hard edge cases; revised as model economics shift.
Skills library and orchestration:
Skills library: versioned SKILL.
MD specs for recurring extraction patterns, invocable by engineers and agents alike.
Orchestration: Airflow or equivalent patterns that scale with data volumes and source counts; dependency management, recovery, cost-aware scheduling.
Requirements:
6-10 years shipping production extraction, ETL,



or data pipeline systems in business-critical environments; deep web extraction at scale with experience designing around anti-bot defences, proxy architecture, JS rendering, schema drift, and recovery.
Shipped LLMs inside extraction pipelines as production systems: structured outputs, versioned prompts, labelled eval sets, logged traces, judges debugged on precision/recall, drift detected on vendor model updates.
You can show the repo.
Built agents and tool-calling pipelines: architected agent workflows, written SKILL.
MD specs others depend on, run tool-calling systems in production.
Sharp judgement on rules vs. LLMs: reach for deterministic logic first, defend the call either way, and have codified the standard for a team.
Expert Python: production-grade, performance-aware, comfortable with concurrency and scale; plus advanced SQL for complex transformations and performance optimisation.
Extensive Airflow or equivalent: orchestration, dependency management, recovery patterns, cost optimisation at production scale.
Shipped with agentic IDEs: Claude Code, Cursor, or equivalent as your default mode, with shipped extraction systems to show for it.
Architecture judgement and a product mindset: when to refactor, optimise, ship, or start again; you care about coverage and accuracy landing with customers, not uptime metrics.
You live and breathe AI tools.
Structured outputs, evals, traces, and LLM tracing.
Highly Valued:
Cloud data platforms: Snowflake, Redshift; AWS for pipeline deployment (Lambda, S3, ECS, Glue).
Vector databases, embeddings, or retrieval patterns for matching and deduplication.
Eval frameworks like Braintrust, Promptfoo, or Inspect; LLM tracing tooling.
Data quality frameworks with automated testing and anomaly detection at scale.
B2B data: firmographics, people data, company registries across markets.
Data privacy and compliance: GDPR, CCPA.
Startup or scaleup experience where you shipped fast and owned outcomes end to end.

📌 Lead Data Engineer (Melbourne)
🏢 Firmable
📍 Melbourne