The candidate will have responsibilities across the following functions:

Data Sourcing and ETL architecture:

- End-to-end pipeline design: extraction, normalisation, deduplication, validation, and load; plus cost, performance, and reliability of the whole layer.
- Sourcing layer standards: coverage, accuracy, schema design; the rule-vs-LLM standard codified so the team applies it without you.
- Cost and performance optimisation: cost ceilings with the token math behind them, incremental processing, recovery design, scheduling.

Extraction systems and frameworks:

- Extractor framework: patterns, abstractions, and tooling other engineers ship into; recent extractors are fast to build and reliable to run.
- Hard-source extractors: anti-bot defences, JS-heavy rendering, schema drift, low- quality structure; plus the proxy and IP rotation strategy behind them.
- Agentic extraction pipelines: rule-based triage, LLM escalation, structured-output validation, retries, human-review queues.

LLM infrastructure and observability:

- LLMs as production systems: versioned prompts, labelled eval sets, measured precision and recall, prompt versioning you can roll back, judges debugged on real data.
- Eval and observability scaffolding: eval frameworks, prompt versioning, traces, drift detection when a vendor silently updates a model.
- Model-choice playbook: cheap models for classification, stronger models for nuanced extraction, frontier models for hard edge cases; revised as model economics shift.

Skills library and orchestration:

- Skills library: versioned SKILL. MD specs for recurring extraction patterns, invocable by engineers and agents alike.
- Orchestration: Airflow or equivalent patterns that scale with data volumes and source counts; dependency management, recovery, cost-aware scheduling.

Requirements:

- 6-10 years shipping production extraction, ETL,



or data pipeline systems in business-critical environments; deep web extraction at scale with experience designing around anti-bot defences, proxy architecture, JS rendering, schema drift, and recovery.
- Shipped LLMs inside extraction pipelines as production systems: structured outputs, versioned prompts, labelled eval sets, logged traces, judges debugged on precision/recall, drift detected on vendor model updates. You can show the repo.
- Built agents and tool-calling pipelines: architected agent workflows, written SKILL. MD specs others depend on, run tool-calling systems in production.
- Sharp judgement on rules vs. LLMs: reach for deterministic logic first, defend the call either way, and have codified the standard for a team.
- Expert Python: production-grade, performance-aware, comfortable with concurrency and scale; plus advanced SQL for complex transformations and performance optimisation.
- Extensive Airflow or equivalent: orchestration, dependency management, recovery patterns, cost optimisation at production scale.
- Shipped with agentic IDEs: Claude Code, Cursor, or equivalent as your default mode, with shipped extraction systems to show for it.
- Architecture judgement and a product mindset: when to refactor, optimise, ship, or start again; you care about coverage and accuracy landing with customers, not uptime metrics.
- You live and breathe AI tools. Structured outputs, evals, traces, and LLM tracing.

Highly Valued:

- Cloud data platforms: Snowflake, Redshift; AWS for pipeline deployment (Lambda, S3, ECS, Glue).
- Vector databases, embeddings, or retrieval patterns for matching and deduplication.
- Eval frameworks like Braintrust, Promptfoo, or Inspect; LLM tracing tooling.
- Data quality frameworks with automated testing and anomaly detection at scale.
- B2B data: firmographics, people data, company registries across markets.
- Data privacy and compliance: GDPR, CCPA.
- Startup or scaleup experience where you shipped fast and owned outcomes end to end.

#J-18808-Ljbffr

📌 Lead Data Engineer (Melbourne)
🏢 Firmable
📍 Melbourne