Workflow

## Architecture Workflows live in `workflows/` with components and flows: ``` workflows/ ├── index.md # Inventory with tags ├── components/ │ ├── connections/ # Auth configs │ ├── nodes/ # Reusable operations │ └── triggers/ # Event sources └── flows/{name}/ ├── flow.md # Definition ├── config.yaml # Parameters ├── run.sh # Executable ├── state/ # Persistent between runs └── logs/ ``` ## Quick Reference | Topic | File | |-------|------| | Directory layout, naming, formats | `structure.md` | | Data passing between nodes | `data-flow.md` | | Cursor, seen set, checkpoint | `state.md` | | Retry, rollback, idempotency | `errors.md` | | Connections, nodes, triggers | `components.md` | | Create, test, update, archive | `lifecycle.md` | ## Requirements - **jq** — JSON processing - **yq** — YAML config parsing - **curl** — HTTP requests - **flock** — Lock files to prevent concurrent runs - Secrets in macOS Keychain (`security find-generic-password`) ## Data Storage - **Location:** `workflows/` in workspace root - **State:** `flows/{name}/state/` — cursor.json, seen.json, checkpoint.json - **Logs:** `flows/{name}/logs/` — JSONL per run - **Data:** `flows/{name}/data/` — intermediate files between nodes ## Core Rules ### 1. Data Flow Pattern Each node writes output to `data/{NN}-{name}.json`. Next node reads it. ```bash curl ... > data/01-fetch.json jq '...' data/01-fetch.json > data/02-filter.json ``` Breaking this pattern = nodes can't communicate. ### 2. Error Declaration Every node in flow.md MUST declare: - **On error:** retry(N) | fail | continue | alert - **On empty:** skip | continue | fail Undefined behavior = unpredictable workflow. ### 3. Lock Files Prevent concurrent runs: ```bash LOCKFILE="/tmp/workflow-${NAME}.lock" exec 200>"$LOCKFILE" flock -n 200 || exit 0 ``` ### 4. State Files | File | Purpose | |------|---------| | cursor.json | "Where did I leave off?" | | seen.json | "What have I processed?" | | checkpoint.json | "Multi-step recovery" | ### 5. Component Reuse Before creating new connections/nodes/triggers: ```bash ls workflows/components/connections/ ls workflows/components/nodes/ ``` Use existing. Update "Workflows Using This" when adding. --- **Related:** For LLM-driven multi-phase processes, see the `cycle` skill.