system-restoration

# System Restoration Comprehensive guide for restoring Advantage HPE's operational intelligence systems when they fail or go down. ## Investigation Workflow ### 1. System Status Assessment Before fixing anything, map out what's broken: **Core Intelligence Systems:** 1. **Zero Revenue Alerts** → #margin-alerts (Every 30 min) 2. **Morning Pulse** → #manager-nudges (Daily 6:35 AM) 3. **Live Nudges** → #manager-nudges (Every 15 min) 4. **Material Truth Report** → #material-intel-systems (Daily 7:00 AM) 5. **Friend-Zone Reformatter** → #live-ops (ServiceTitan email alerts) **Investigation Commands:** ```bash # Check LaunchD services launchctl list | grep ranger # Check cron jobs cron list # Check running processes ps aux | grep -E "(keel|pulse|margin|nudge)" | grep -v grep # Find system code find /Users/stephendobbins/.config/ranger -name "*.py" | grep -E "(pulse|margin|nudge)" find /Users/stephendobbins/.openclaw/workspace -name "*.py" | grep -E "(zero|revenue)" ``` ### 2. Locate Code & Determine Failure Cause **Common Locations:** - `/Users/stephendobbins/.config/ranger/scripts/` - Main operational scripts - `/Users/stephendobbins/.config/ranger/materials/` - Material intelligence - `/Users/stephendobbins/.openclaw/workspace/` - Recent scripts & fixes - `/Users/stephendobbins/Library/LaunchAgents/` - LaunchD service definitions **Common Failure Patterns:** - **LaunchD services unloaded** - Emergency shutdown or system restart - **Data source broken** - ServiceTitan API returning wrong data - **Scheduling missing** - Functions exist but no cron/LaunchD trigger - **Script errors** - Import failures, credential issues ## System-Specific Restoration ### Zero Revenue Alerts **Script:** `/Users/stephendobbins/.config/ranger/scripts/margin_alerts.py` **Channel:** #margin-alerts (C0A5L7MG60P) **Schedule:** Every 30 minutes **Restoration Steps:** 1. Verify script exists and posts to Slack 2. Load LaunchD service: `launchctl load /Users/stephendobbins/Library/LaunchAgents/com.ranger.margin-alerts.plist` 3. Test manually: `cd /Users/stephendobbins/.config/ranger/scripts && python3 margin_alerts.py` 4. Check logs: `tail /tmp/margin_alerts.log` ### Morning Pulse **Script:** `/Users/stephendobbins/.config/ranger/scripts/pulse_os_full.py` **Channel:** #manager-nudges (C0A5V9JL2KV) **Schedule:** Daily 6:35 AM CT **Restoration Steps:** 1. **If broken API data:** Check for `.bak` backup with working data sources 2. **Restore backup:** `cp pulse_os_full.py.bak pulse_os_full.py` 3. **Fix data sources:** Replace API calls with browser automation (see references/browser-data-sources.md) 4. Load LaunchD service: `launchctl load /Users/stephendobbins/Library/LaunchAgents/com.ranger.morning-pulse.plist` 5. Test: `python3 pulse_os_full.py pulse` ### Live Nudges **Script:** `/Users/stephendobbins/.config/ranger/scripts/pulse_os_full.py nudges` **Channel:** #manager-nudges **Schedule:** Every 15 minutes **Function:** `run_nudges()` on line 548-617 **Features:** 🚗 dispatched / 📍 arrived / ✅ completed alerts **Restoration Steps:** 1. Verify function exists: `grep -n "def run_nudges" pulse_os_full.py` 2. Create LaunchD service (see scripts/create-live-nudges-service.py) 3. Load service: `launchctl load /Users/stephendobbins/Library/LaunchAgents/com.ranger.live-nudges.plist` 4. Test: `python3 pulse_os_full.py nudges` ### Material Truth Report **Script:** `/Users/stephendobbins/.config/ranger/materials/reconciliation_report.py` **Channel:** #material-intel-systems (C0A5L7RB5EK) **Schedule:** Daily 7:00 AM CT **Restoration Steps:** 1. Test script: `cd /Users/stephendobbins/.config/ranger/materials && python3 reconciliation_report.py --no-email` 2. Create cron job with 7:00 AM schedule 3. Verify channel posting ## Data Source Repair ### ServiceTitan API vs UI Data **Problem:** ServiceTitan API often returns test/historical data instead of real operational data. **Solution:** Replace API calls with browser automation: 1. **Create browser data source module** (see scripts/browser_data_sources.py) 2. **Import in main script:** Replace parse functions with browser equivalents 3. **Preserve output format** - Same sections, different data source **Browser Data Functions:** - `get_browser_low_margin_jobs()` - `get_browser_stale_estimates()` - `get_browser_revenue_leaks()` - `get_browser_driver_incidents()` ### KEEL System Issues **Script:** `/Users/stephendobbins/.config/ranger/keel/keel_slack_bot.py` **Safe restart for field tech DM only:** 1. **Disable operational intelligence:** Set `OPERATIONAL_INTELLIGENCE_ENABLED = False` 2. **Restart process:** `cd /Users/stephendobbins/.config/ranger/keel && python3 keel_slack_bot.py &` 3. **Verify running:** `ps aux | grep keel_slack_bot` ## Service Management Commands ### LaunchD Services ```bash # List services launchctl list | grep ranger # Load service launchctl load /Users/stephendobbins/Library/LaunchAgents/com.ranger.<service>.plist # Unload service launchctl unload /Users/stephendobbins/Library/LaunchAgents/com.ranger.<service>.plist # Start service immediately launchctl start com.ranger.<service> # Check service logs tail /tmp/<service>.log tail /tmp/<service>.err ``` ### Cron Jobs (OpenClaw) ```bash # List jobs cron list # Add job cron add <job-definition> # Remove job cron remove <job-id> ``` ## Emergency Shutdown Recovery When systems are emergency-stopped due to bad data: 1. **Investigate root cause** - Usually ServiceTitan API data issues 2. **Fix data sources** - Switch to browser automation or correct API endpoints 3. **Test manually** - Verify data accuracy before re-enabling 4. **Restore services** - Load LaunchD services and cron jobs 5. **Monitor initially** - Check logs and channel posts for accuracy ## Resources ### scripts/ - `create-live-nudges-service.py` - Generate LaunchD plist for live nudges - `browser_data_sources.py` - Browser automation replacement for broken APIs ### references/ - `launchd-service-templates.md` - LaunchD plist templates for different schedules - `channel-ids.md` - Slack channel IDs for all operational intelligence channels - `troubleshooting-checklist.md` - Step-by-step debugging guide