20 Oct
|
WattleCore Technologies Pty
|
North Sydney
20 Oct
WattleCore Technologies Pty
North Sydney
The Opportunity
We are seeking an experienced DataOps / CI-CD Engineer to build and own the delivery pipeline for an enterprise Snowflake and dbt data platform program. You will design the branching strategy, automate validation and testing of dbt code, and deliver controlled, auditable promotion of changes across DEV, TEST, UAT and PROD. You will also automate Snowflake deployments using the Snowflake CLI and infrastructure-as-code, with secure handling of secrets and service accounts. This role suits an engineer who enjoys bringing engineering discipline, repeatability and governance to data delivery in a regulated environment.
Key Responsibilities
Pipeline ownership
- Design, build and own the end-to-end CI/CD pipeline for the Snowflake and dbt platform.
- Define and maintain the Git branching strategy (e.g. trunk-based or GitFlow), branch protection rules and merge conventions.
- Implement pull request validation, including automated dbt compile, dbt build/test, SQL linting (e.g. SQLFluff) and slim CI using state comparison and deferral.
- Automate promotion of code and objects across DEV, TEST, UAT and PROD with environment-specific configuration and gated quality checks.
- Monitor pipeline health, reduce build times and continuously improve developer experience for the data engineering team.
Snowflake deployment automation and infrastructure-as-code
- Automate Snowflake object deployment using the Snowflake CLI and complementary tooling (e.g. schemachange or native Git integration).
- Provision and manage Snowflake infrastructure – databases, schemas, warehouses, roles, grants, resource monitors and integrations – using Terraform (Snowflake provider)
or similar IaC.
- Manage state, drift detection and environment parity across all tiers.
- Implement secure secrets management using a vault service (e.g. Azure Key Vault, AWS Secrets Manager, HashiCorp Vault) and pipeline secret stores.
- Set up and govern service accounts and authentication for automation, including key-pair authentication, OAuth or workload identity, with least-privilege roles and credential rotation.
Release management and controls
- Configure release approvals, setting protection rules and segregation of duties for UAT and PROD deployments.
- Design and test rollback and recovery approaches, using techniques such as versioned releases, Snowflake zero-copy cloning and Time Travel.
- Ensure complete audit trails linking work items, commits, pull requests, approvals and deployments.
- Produce release notes and evidence to support change management and change advisory board (CAB) processes.
Ways of working
- Work closely with data engineers, enterprise architects, testers and platform and security teams within an agile scrum team.
- Document pipeline designs, runbooks and onboarding guides, and coach engineers on Git and CI/CD practices.
- Align pipeline and deployment patterns with enterprise architecture, security and risk standards.
Essential Skills and Experience
- Proven experience building and owning CI/CD pipelines for data platforms, ideally Snowflake and dbt.
- Strong Git skills: branching strategies, pull request workflows, branch protection and merge management.
- Hands-on experience with at least one of GitHub Actions, Azure DevOps Pipelines or GitLab CI, including multi-stage pipelines, environments and approval gates.
- Experience automating Snowflake deployments, including the Snowflake CLI and infrastructure-as-code with Terraform or similar.
- Solid understanding of secrets management, service accounts and secure authentication for automated pipelines.
- Working knowledge of dbt (Core or Cloud) – compile, build, test, selectors, environments and artifacts.
- Experience implementing release approvals, rollback strategies and audit trails.
- Scripting skills in Python, Bash and/or PowerShell.
- Clear communication skills and the ability to work with engineering, architecture and governance stakeholders.
Highly Desirable
- Banking or financial services experience, or other regulated change-control environments (e.g. ITIL change management, CAB processes, APRA CPS 234 / CPS 230 obligations).
- SnowPro Core certification; HashiCorp Terraform Associate; Azure DevOps, GitHub Actions or GitLab certifications.
- Experience with containerised build agents (Docker) and self-hosted runners.
- Exposure to data observability, automated data testing or quality gates in pipelines.
- Experience with ServiceNow or Jira integration for change and release traceability.
📌 DevOps / CI-CD Engineer (Snowflake dbt) (North Sydney)
🏢 WattleCore Technologies Pty
📍 North Sydney