data-pipelines

# Data Pipelines Pipelines fail on silent schema drift, partial writes, and unclear ownership. Design for at-least-once delivery, idempotent sinks, and observable stages. ## When to Offer This Workflow **Trigger conditions:** - Batch or streaming ingestion (Kafka, Fivetran, Airflow, Dagster, Spark, etc.) - Late data, backfills, or schema changes breaking jobs - SLA misses on freshness or row counts **Initial offer:** Use **six stages**: (1) requirements & SLAs, (2) source contracts, (3) transforms & idempotency, (4) orchestration & dependencies, (5) quality & monitoring, (6) lineage & operations). Confirm batch vs stream and cloud stack. --- ## Stage 1: Requirements & SLAs **Goal:** Freshness (latency), completeness expectations, cost ceiling, failure tolerance (quarantine vs stop-the-line). **Exit condition:** SLA table: pipeline → metric → threshold. --- ## Stage 2: Source Contracts **Goal:** Schema versioning; CDC vs snapshot pulls; API rate limits. ### Practices - Raw landing zone immutable; curated layers downstream --- ## Stage 3: Transforms & Idempotency **Goal:** Deterministic transforms; upsert keys; partition strategy for rewinds. ### Practices - Watermark progress for incremental loads --- ## Stage 4: Orchestration & Dependencies **Goal:** Clear DAG; retry policy; backfill without double counting; SLA miss alerts. --- ## Stage 5: Quality & Monitoring **Goal:** Data quality checks (null spikes, row bounds, referential checks); metrics on lag, duration, error rate. --- ## Stage 6: Lineage & Operations **Goal:** Column-level lineage where valuable; on-call runbook; ownership per pipeline. --- ## Final Review Checklist - [ ] SLAs and failure policy explicit - [ ] Source contracts and schema evolution path - [ ] Idempotent writes and checkpointing - [ ] Orchestration with retries and safe backfill - [ ] Data quality checks and alerts - [ ] Lineage and ownership documented ## Tips for Effective Guidance - Separate compute from storage cost awareness for large shuffles. - Pair with **etl-design** for batch patterns and **message-queues** for streaming handoffs. ## Handling Deviations - Single-script pipelines: still document inputs, outputs, and schedule.