load-testing

# Load Testing (Deep Workflow) Load tests answer **whether the system meets behavior under target load**—not “how many RPS the tool prints.” Tie every run to **SLOs**, **workload realism**, and **analysis** that engineers can act on. ## When to Offer This Workflow **Trigger conditions:** - Major launch, traffic spike season, infra resize - Latency/timeout under peak; need **evidence** for capacity decisions - Comparing architectures or **debottlenecking** **Initial offer:** Use **seven stages**: (1) goals & SLOs, (2) workload model, (3) scenarios & scripts, (4) environment & data, (5) run & observe, (6) analyze bottlenecks, (7) fixes & retest. Confirm **tool** (k6, Locust, Gatling, JMeter) and **environment** policy (prod-like staging vs synthetic). --- ## Stage 1: Goals & SLOs **Goal:** Define **success** in measurable terms. ### Questions 1. **Peak** RPS/users, **growth** assumption, **duration** of peak 2. **SLOs**: p95/p99 latency, error rate, throughput **per** critical endpoint 3. **Scope**: read-heavy vs write-heavy; **background** jobs interaction **Exit condition:** Numeric **targets** + **out of scope** (e.g., “third-party API mocked”). --- ## Stage 2: Workload Model **Goal:** **Representative** mix—not one URL forever. ### Practices - **Transaction mix** from analytics or access logs (proportions) - **Think time** between steps for user journeys - **Payload** size distribution; **auth** token behavior - **Spike** vs **soak** vs **step** ramp—match **real** failure modes **Exit condition:** **Workload profile** documented (table or script comments). --- ## Stage 3: Scenarios & Scripts **Goal:** **Deterministic**, **idempotent** load scripts where possible. ### Practices - **Correlate** virtual user with **trace/request id** for debugging - **Parameterize** data to avoid **cache** **fantasy** (every request hits same key) - **Order** operations to match **real** **causality** (login → browse → checkout) ### Pitfalls - **Client-side** bottleneck (single generator machine)—**distribute** load generators **Exit condition:** **Smoke** run at small k validates script **correctness**. --- ## Stage 4: Environment & Data **Goal:** **Fidelity** without **destroying** prod. ### Rules - **Staging** scale proportional; **feature flags** aligned - **Data volume** similar order-of-magnitude for **DB** **plans** - **External** deps: mock, **sandbox**, or **throttle** **awareness** **Exit condition:** **Safety** checklist: no prod writes unless explicitly planned and isolated. --- ## Stage 5: Run & Observe **Goal:** **System-wide** visibility during test. ### Instrumentation - **App**: latency histograms, error codes, **queue** depth - **Infra**: CPU, memory, **connections**, **GC**, **disk** IOPS - **DB**: slow queries, **locks**, **replication** lag - **Tracing** sample during test for **hot spans** **Exit condition:** **Dashboard** or **runbook** link for the test window. --- ## Stage 6: Analyze Bottlenecks **Goal:** Identify **dominant** constraint: **app**, **DB**, **network**, **dependency**. ### Process - **Utilization** vs **saturation** (e.g., CPU high but wait on locks—different fix) - **Compare** p95 vs **max**—**tail** often **separate** issue - **Reproduce** bottleneck with **smaller** experiment when unclear **Exit condition:** **Written** hypothesis with **evidence** (graphs, trace ids). --- ## Stage 7: Fixes & Retest **Goal:** **Controlled** changes with **retest** protocol. ### Practices - **One** major change per retest when debugging - **Document** **baseline** vs **after** for regression to **capacity** planning --- ## Final Review Checklist - [ ] SLO-aligned goals and workload mix - [ ] Realistic scenarios; distributed load if needed - [ ] Environment safe and representative enough - [ ] Full-stack observability during runs - [ ] Bottleneck analysis leads to actionable tickets ## Tips for Effective Guidance - **Warm** caches explicitly if prod is always warm—otherwise **misleading** **good** numbers. - **Throughput** without **latency** SLO is meaningless. - Call out **coordination** **overhead** (locks, **hot** **keys**) vs **raw** CPU. ## Handling Deviations - **Cannot** match prod data: **state** **assumptions** and test **directional** only. - **Serverless**: account for **cold** **start** and **account** **concurrency** limits in interpretation.