testing-strategy

# Testing Strategy (Deep Workflow) Testing strategy answers: **what failures would hurt users**, **what’s cheap to catch**, and **what signals we trust in CI**. Coverage percentage alone is a weak proxy—**risk alignment** matters. ## When to Offer This Workflow **Trigger conditions:** - New service or major refactor; “what should we test?” - Flaky CI, long runtimes, or tests nobody trusts - Debate: unit vs integration vs e2e; QA headcount vs automation **Initial offer:** Use **six stages**: (1) risk & quality goals, (2) pyramid & layers, (3) design per layer, (4) data & environments, (5) CI & gates, (6) observability of test health. Confirm **release cadence** and **regulatory** needs. --- ## Stage 1: Risk & Quality Goals **Goal:** Connect tests to **user impact** and **business risk**. ### Questions 1. Worst **failure categories**: payments wrong, data leak, outage, wrong advice (AI)? 2. **SLO** for critical paths—what must never break silently? 3. **Change velocity**—how fast must PRs merge safely? ### Output **Risk register** → **test priorities** (not every line equally important). **Exit condition:** Top **5 risks** have explicit **test intent**. --- ## Stage 2: Pyramid & Layers **Goal:** **Many fast tests**, **some integration**, **few e2e**—proportion tuned to risk. ### Layers (typical) - **Unit**: pure logic, cheap, deterministic - **Integration**: DB, queue, real dependencies in containers—slower but valuable - **Contract**: between services—consumer-driven contracts when decoupled teams - **E2E**: full stack—expensive; **minimal happy path + critical regressions** ### Anti-patterns - **E2E-only** (slow, flaky) - **Mock everything** (misses real integration bugs) **Exit condition:** Written **policy**: what belongs in each layer for this codebase. --- ## Stage 3: Design Per Layer **Goal:** Tests are **readable**, **stable**, and **debuggable**. ### Unit - **Given/when/then** clarity; **avoid** testing implementation details - **Property-based** tests for tricky invariants (dates, money, parsers) ### Integration - **Testcontainers** or docker-compose in CI; **migrations** applied - **Parallel** safe—unique DB schemas or transactions ### E2E - **Stable selectors** (data-testid); **retry** policy disciplined—**fix flakes**, don’t hide them - **Seed data** minimal; **idempotent** setup **Exit condition:** **Flake** classification process exists (quarantine + ticket). --- ## Stage 4: Data & Environments **Goal:** **Representative** data without **PII** leakage. ### Practices - **Fixtures** versioned; **factories** for variations - **Anonymized** prod-like datasets for perf tests—**governance** for access - **Env parity**: staging behaves like prod enough for meaningful e2e **Exit condition:** Data **generation** documented; **secrets** not in tests. --- ## Stage 5: CI & Gates **Goal:** **Fast feedback** on PRs; **nightly** heavier suites if needed. ### Tiers - **PR**: lint, unit, fast integration subset - **Main**: full integration; **optional** e2e against ephemeral env - **Release**: smoke + canary in prod ### Metrics - **Flake rate**, **duration**, **quarantined** tests count—**visible** **Exit condition:** **Merge policy** tied to green checks; **exceptions** process defined. --- ## Stage 6: Test Health & Culture **Goal:** Tests are **owned** like features. ### Practices - **Ownership** per suite; **on-call** for CI when org size supports - **Delete** tests that don’t pay rent—**or** fix them --- ## Final Review Checklist - [ ] Risks mapped to test layers - [ ] Pyramid policy documented - [ ] Flake management process exists - [ ] CI tiers match team velocity - [ ] Data/fixture strategy safe and maintainable ## Tips for Effective Guidance - Recommend **testing seams**: boundaries where contracts are stable. - Warn against **snapshot abuse** for large UI—**diff noise** kills trust. - For **AI/LLM**, discuss **eval harnesses** beyond classic unit tests. ## Handling Deviations - **Legacy untestable code**: **characterization tests** then refactor seams. - **Startup speed**: **smoke** + **critical path** first; expand as pain appears.