Payrate:

$92.00 - $98.00/hr.

Summary:

We are seeking a hands-on Data Pipeline Engineer to build and operate Python-based workflows that transform large document and text datasets into high-quality data for AI model training. This role combines OCR processing, data cleaning and validation, scalable pipeline development, and Azure storage management.

Responsibilities

- Develop, maintain, and optimize Python pipelines for data ingestion, OCR processing, text extraction, cleaning, normalization, and deduplication.
- Run and monitor large-scale batch processing jobs, investigate failures, and implement reliable retry and recovery mechanisms.
- Build automated quality checks to identify missing or corrupted files, incomplete OCR outputs, duplicate content, formatting issues, and metadata inconsistencies.
- Prepare training-ready datasets through filtering, text segmentation, tokenization, and conversion into agreed data formats.
- Manage bulk uploads, downloads, and synchronization between local processing environments and Azure storage; verify transfer completeness and data integrity.
- Organize raw data, intermediate outputs, and released datasets with consistent naming, metadata, versioning, and source traceability.
- Optimize processing throughput, memory usage, storage consumption, and cloud costs using appropriate batch, parallel, or distributed processing approaches.
- Track dataset statistics, processing coverage, quality metrics, and exceptions; produce clear operational reports.
- Collaborate with applied scientists and model-training teams to translate data requirements into reproducible processing workflows.
- Maintain tested, documented, version-controlled code and follow established data access, security, and governance requirements.

Qualifications

- Demonstrated experience in data engineering, data pipeline development, or a data science role with substantial production programming responsibilities.
- Strong proficiency in Python, including writing reusable modules, debugging, testing, logging, and handling processing errors.




- Hands‑on experience processing large structured and unstructured datasets using memory‑efficient, parallel, or distributed techniques.
- Practical experience with Azure Blob Storage or Azure Data Lake Storage, including bulk data transfers, storage organization, and access troubleshooting.
- Familiarity with Python data‑processing libraries and formats such as pandas, NumPy, JSON/JSONL, CSV, and Parquet.
- Working knowledge of SQL, Git, Linux environments, and command‑line tools.
- Strong troubleshooting skills, attention to data quality, and ability to communicate technical issues clearly.

Skills

- Experience with OCR pipelines, scanned documents, multilingual text, or document‑layout processing.
- Experience with Azure Databricks, Azure Machine Learning, Azure Data Factory, or workflow orchestration tools.
- Familiarity with Hugging Face Datasets, tokenization, near‑duplicate detection, and training/evaluation data separation.
- Experience with dataset lineage, reproducibility, containerized environments, and CI/CD.

Pay Transparency:

The typical base pay for this role across the U.S. is:$92.00 - $98.00/hour. Non‑exempt positions are eligible for overtime at a rate of 1.5 times the base hourly rate for all hours worked in excess of 40 in a work week, or as required by state or local law. Final offer amounts, within the base pay set forth above, are determined by factors including your relevant skills, education and experience. Full‑time employees are eligible to select from different benefits packages. Packages may include medical, dental, and vision advantages, health savings accounts with qualified medical plan enrollment, 10 paid days off, 3 days paid bereavement leave, 401(k) plan participation with employer match, life and disability insurance, commuter benefits, dependent care flexible spending account, accident insurance, critical illness insurance, hospital indemnity insurance, accommodations and reimbursement for work travel, and discretionary performance or recognition bonus. Sick leave and mobile phone reimbursement provided based on state or local law.

#J-18808-Ljbffr

📌 AI Data Engineer (Redmond)
🏢 Aditi Consulting
📍 Redmond