example-device-health

# Cisco IOS-XE Device Health Check Structured triage procedure for assessing Cisco IOS-XE device health. Produces a prioritized findings report with severity classifications and recommended actions. ## When to Use - Device is reported as slow, unresponsive, or dropping traffic - Scheduled health audit of IOS-XE routers or switches - Post-change verification after configuration or software updates - Capacity planning data collection for CPU, memory, and interface utilization - Incident response when a device is suspected as the fault domain ## Prerequisites - SSH or console access to the target IOS-XE device (privilege level 1 minimum) - Device running IOS-XE 16.x or 17.x (commands validated against 17.3+) - Network reachability confirmed (ping/traceroute to management IP succeeds) - Knowledge of the device's normal baseline (typical CPU, memory, traffic levels) - Change control approval if performing checks during a maintenance window ## Procedure Follow this sequence. Each step produces data for the final report. Do not skip steps unless the device is unresponsive (jump to Step 6 for crash recovery). ### Step 1: Establish Baseline Context Collect device identity and uptime to frame the health check. ``` show version | include uptime|Version|bytes of memory show inventory | include PID show clock ``` Record: hostname, software version, uptime, hardware model, current time. Flag if uptime is unexpectedly short — indicates recent reload or crash. ### Step 2: CPU Utilization Assessment ``` show processes cpu sorted | head 20 show processes cpu history show processes cpu platform sorted 5sec ``` Compare 5-second, 1-minute, and 5-minute averages against thresholds. If 5-second average exceeds 80%, identify the top process immediately. Key processes to watch: - **IP Input** — high values indicate traffic processing overload - **Crypto IKMP** — VPN negotiation storms - **SNMP ENGINE** — aggressive polling - **BGP Router** — large table churn or route oscillation - **IOSD** — general control plane congestion ### Step 3: Memory Utilization Assessment ``` show memory statistics show memory platform information show processes memory sorted | head 15 ``` Calculate used percentage: `(Total - Free) / Total * 100`. Check for memory fragmentation: compare Largest Free block to Total Free. If largest free block is less than 10% of total free, fragmentation is a concern. ### Step 4: Interface Health ``` show interfaces summary show interfaces counters errors show interfaces | include line protocol|drops|error|CRC|collision ``` For each interface with errors: - Calculate error rate: `errors / (input packets + output packets) * 100` - Error rate above 0.1% is warning, above 1% is critical - CRC errors suggest Layer 1 issues (cabling, optics, SFP) - Input errors with no CRC suggest buffer or overrun issues - Output drops indicate congestion — check QoS policy ### Step 5: Routing Table Health ``` show ip route summary show ip bgp summary (if BGP is configured) show ip ospf neighbor (if OSPF is configured) show ip eigrp neighbors (if EIGRP is configured) ``` Verify: expected number of routes present, no unexpected route withdrawals, all routing protocol neighbors in established/full state. Flag: neighbor state changes in the last hour, route count significantly different from baseline, any routes via unexpected next-hops. ### Step 6: Platform and Environment ``` show environment all show platform software status control-processor brief show logging | include %|Error|Warning|traceback (last 50 lines) ``` Check: power supply status, fan status, temperature readings. Any environmental alarm is an immediate escalation trigger. Review recent syslog for crash signatures (traceback, CPUHOG, MALLOCFAIL). ## Threshold Tables Reference: `references/threshold-tables.md` for detailed per-parameter thresholds. | Parameter | Normal | Warning | Critical | |-----------|--------|---------|----------| | CPU 5-min avg | < 40% | 40–70% | > 70% | | CPU 5-sec spike | < 80% | 80–90% | > 90% | | Memory used | < 70% | 70–85% | > 85% | | Memory fragmentation | > 10% largest/total | 5–10% | < 5% | | Interface error rate | < 0.01% | 0.01–0.1% | > 0.1% | | Interface output drops | < 100/hr | 100–1000/hr | > 1000/hr | | Routing neighbors | All established | Flapping | Down | | Temperature | Within spec | Within 5°C of max | At or above max | ## Decision Trees ### Triage Priority ``` Is the device reachable? ├── No → Escalate immediately. Check console access, power, environment. └── Yes ├── CPU critical? → Identify top process → Apply mitigation per process │ ├── IP Input → Check for traffic storm, ACL optimization │ ├── BGP Router → Check for route churn, peer flap, table size │ └── Other → Collect 'show tech-support' for TAC escalation ├── Memory critical? → Check for memory leak │ ├── Largest free < 5% of total → Likely fragmentation, schedule reload │ └── Steady growth over time → Memory leak, collect 'show mem alloc' ├── Interface errors? → Classify error type │ ├── CRC/input errors → Layer 1 (cable, optic, SFP) │ └── Output drops → QoS policy or congestion └── All within thresholds → Document clean health, schedule next check ``` ### Escalation Criteria Escalate to senior engineer or TAC when any of these conditions are met: - CPU sustained above 90% for more than 15 minutes with no identifiable cause - Memory below 15% free with no recent change to explain consumption - Traceback or CPUHOG messages in logs within last 24 hours - Environmental alarm (power, fan, temperature) present - More than 3 routing neighbor state changes in last hour ## Report Template Generate a structured report with these sections: ``` DEVICE HEALTH REPORT ==================== Device: [hostname] Model: [PID from inventory] Software: [version] Uptime: [uptime string] Check Time: [timestamp] Performed By: [operator/agent] SUMMARY: [HEALTHY | WARNING | CRITICAL] FINDINGS: 1. [Severity] [Component] — [Description] Observed: [metric value] Threshold: [normal/warning/critical range] Action: [recommended action] 2. ... RECOMMENDATIONS: - [Prioritized list of actions] NEXT CHECK: [scheduled date based on findings severity] ``` Severity levels for findings: - **INFO** — within normal thresholds, noted for baseline - **WARNING** — approaching threshold, monitor closely - **CRITICAL** — threshold exceeded, action required - **EMERGENCY** — device at risk of failure, immediate action ## Troubleshooting ### Device Unresponsive to SSH Try console access. If console is also unresponsive, check power and environment remotely (smart PDU, out-of-band management). If the device has crashed, collect crashinfo: `dir crashinfo:` after recovery. ### CPU Spikes During Health Check SNMP polling or show commands themselves can briefly spike CPU. Wait 30 seconds after connecting before collecting CPU data. Use `terminal length 0` to avoid paging pauses that extend session time. ### Inconsistent Memory Readings Memory values fluctuate during normal operation. Collect three samples at 30-second intervals and average them. Check `show memory dead` for memory that is allocated but unreachable (leak indicator). ### Interface Counter Interpretation Counters are cumulative since last clear. Use `show interfaces [name]` to see the last clear time. For rate calculations, collect counters twice with a known interval: `(counter2 - counter1) / interval_seconds`. ### Routing Protocol Neighbor Issues If OSPF neighbors are stuck in INIT/2WAY, check MTU mismatch and area configuration. If BGP peers show "Active" state, verify TCP connectivity on port 179 and check for ACL blocking. EIGRP stuck-in-active indicates a convergence problem downstream.