observability-slos

# Observability & SLOs (Deep Workflow) SLOs connect **engineering work** to **user-perceived reliability**. SLIs must be **measurable from systems** but **grounded in user journeys**. ## When to Offer This Workflow **Trigger conditions:** - Defining **99.9%** without defining **for what** - Too many pages or none; need **error budget** discipline - **Product** wants features while **stability** degrades **Initial offer:** Use **six stages**: (1) pick user journeys, (2) define SLIs, (3) set SLO targets & windows, (4) error budget policy, (5) alerting on budget burn, (6) review & iterate). Confirm **metric** stack and **dependency** SLOs from vendors. --- ## Stage 1: User Journeys **Goal:** **Critical paths** that matter if broken—checkout, login, API sync, not “CPU low”. ### Output 3–10 journeys ranked by **business impact** and **frequency**. **Exit condition:** One paragraph per journey: user intent + failure symptom. --- ## Stage 2: Define SLIs **Goal:** **Ratio** of good events over total over a window—**implementation** explicit. ### Examples - **Availability**: successful requests / valid requests (define “valid”) - **Latency**: proportion of requests faster than **T** ms ### Good SLIs - **Objective**, **low-cardinality** enough to measure reliably **Exit condition:** SLI formula + data source (metrics, logs, probes). --- ## Stage 3: SLO Targets & Windows **Goal:** **Target** (e.g., 99.9% monthly) implies **allowed** bad minutes—make it explicit. ### Practices - **Rolling** 30d common; align with **release** cadence - **Tier** services: not everything needs same SLO **Exit condition:** Published table: journey → SLI → target → window. --- ## Stage 4: Error Budget Policy **Goal:** **What we do** when budget is healthy vs exhausted. ### Policy ideas - Budget healthy → ship features; low → freeze risky changes, focus on reliability - **Escalation** when budget burns fast (multi-window alerts) **Exit condition:** Written policy with product sign-off. --- ## Stage 5: Alerting on Burn **Goal:** Page on **budget burn rate**, not every blip—**multi-window** **multi-burn-rate** pattern when using Google-style SLO alerting. ### Practices - **Fast burn** = page soon; **slow burn** = ticket/track **Exit condition:** Alert rules linked to runbooks. --- ## Stage 6: Review & Iterate **Goal:** SLOs **drift** with architecture—**quarterly** review; adjust targets with data. --- ## Final Review Checklist - [ ] Journeys and SLIs tied to real user pain - [ ] Targets realistic vs dependencies and cost - [ ] Error budget policy agreed with product - [ ] Alerts on burn, not noisy symptom spam - [ ] Review cadence scheduled ## Tips for Effective Guidance - Translate **99.9%** to **minutes/month** of allowed badness. - **SLA** (contract) vs **SLO** (internal)—don’t confuse. - Dependency SLO caps what you can promise—surface that early. ## Handling Deviations - **No metrics yet**: start with **proxy SLI** (synthetic probes) and improve instrumentation. - **Batch systems**: event processing lag as SLI instead of HTTP.