ComfyUI

# ComfyUI Runner ## Overview Run ComfyUI workflows on the local server (default 127.0.0.1:8188) using API-format JSON and return output images. ## Editing the workflow before running The run script only takes `--workflow <path>`. You must **inspect and edit the workflow JSON** before running, using your best knowledge of the ComfyUI API format. Do not assume fixed node IDs, `class_type` names, or `_meta.title` values — the user may have updated the default workflow or supplied a custom one. **For every run (including the default workflow):** 1. Read the workflow JSON (default: `skills/comfyui/assets/default-workflow.json`, or the path/file the user gave). 2. **Identify prompt-related nodes** by inspecting the graph: look for nodes that hold the main text prompt — e.g. `PrimitiveStringMultiline`, `CLIPTextEncode` (positive text), or any node with `_meta.title` or `class_type` suggesting "Prompt" / "positive" / "text". Update the corresponding input (e.g. `inputs.value`, or the text input to the encoder) to the image prompt you derived from the user (subject, style, lighting, quality). If the user didn’t ask for a custom image, you can leave the existing prompt or tweak only if needed. 3. **Optionally identify style/prefix nodes** — e.g. `StringConcatenate`, or a second string input that acts as style. Set them if the user asked for a specific style or to clear a default prefix. 4. **Optionally set a new seed** — find sampler-like nodes (e.g. `KSampler`, `BasicGuider`, or any node with a `seed` input) and set `seed` to a new random integer so each run can differ. 5. Write the modified workflow to a temp file (e.g. `skills/comfyui/assets/tmp-workflow.json`). Use `~/ComfyUI/venv/bin/python` for any inline Python; do not use bare `python`. 6. Run: `comfyui_run.py --workflow <path-to-edited-json>`. If the workflow structure is unclear or you can’t find prompt/sampler nodes, run the file as-is and only change what you can reliably identify. Same approach for arbitrary user-supplied JSON: inspect first, edit at your best knowledge, then run. ## Run script (single responsibility) ```bash ~/ComfyUI/venv/bin/python skills/comfyui/scripts/comfyui_run.py \ --workflow <path-to-workflow.json> ``` The script only queues the workflow and polls until done. It prints JSON with `prompt_id` and output `images`. All prompt/style/seed changes are done by you in the JSON beforehand. ## If the server isn’t reachable If the run script fails with a connection error (e.g. connection refused or timeout to 127.0.0.1:8188), ComfyUI may not be installed or not running. **Check:** Does `~/ComfyUI` exist and contain `main.py`? - **If not installed:** Install ComfyUI (e.g. clone the repo, create a venv, install dependencies, then start the server). Example: ```bash git clone https://github.com/comfyanonymous/ComfyUI.git ~/ComfyUI cd ~/ComfyUI python3 -m venv venv ~/ComfyUI/venv/bin/pip install -r requirements.txt ``` Then start the server (see below). Tell the user they may need to install model weights into `~/ComfyUI/models/` depending on the workflow. - **If installed but not running:** Start the ComfyUI server so the API is available on port 8188. Example: ```bash ~/ComfyUI/venv/bin/python ~/ComfyUI/main.py --listen 127.0.0.1 ``` Run in the background or in a separate terminal so it keeps running. Then retry the workflow run. Use `~` (or the user’s home) for paths so it works on their machine. ## Model weights from URLs When the user pastes or sends a **list of model weight URLs** (one per line, or comma-separated), download those files into the ComfyUI installation so the workflow can use them later. 1. **Normalize the list** — one URL per line; strip empty lines and comments (lines starting with `#`). 2. **Run the download script** with the ComfyUI base path (default `~/ComfyUI`). The script uses [pget](https://github.com/replicate/pget) for parallel downloads when available; if `pget` is not in PATH, it installs it to `~/.local/bin` automatically (no sudo). If pget cannot be installed (e.g. unsupported OS/arch), it falls back to a built-in download. Use the ComfyUI venv Python so the script runs correctly: ```bash ~/ComfyUI/venv/bin/python skills/comfyui/scripts/download_weights.py --base ~/ComfyUI ``` Pass URLs as arguments, or pipe a file/list on stdin: ```bash echo "https://example.com/model.safetensors" | ~/ComfyUI/venv/bin/python skills/comfyui/scripts/download_weights.py --base ~/ComfyUI ``` Or save the user’s list to a temp file and run: ```bash ~/ComfyUI/venv/bin/python skills/comfyui/scripts/download_weights.py --base ~/ComfyUI < /tmp/weight_urls.txt ``` To force the built-in download (no pget): add `--no-pget`. 3. **Subfolder:** The script infers the ComfyUI models subfolder from the URL/filename (e.g. `vae`, `clip`, `loras`, `checkpoints`, `text_encoders`, `controlnet`, `upscale_models`). The user can optionally specify a subfolder per line as `url subfolder` (e.g. `https://.../model.safetensors vae`). You can also pass a default with `--subfolder loras` so all URLs in that run go to `models/loras/`. 4. **Existing files:** By default the script skips URLs that already exist on disk; use `--overwrite` to replace. 5. **Paths:** Files are written under `~/ComfyUI/models/<subfolder>/`. Tell the user where each file was saved and that they can run the workflow once the ComfyUI server is (re)started if needed. Supported subfolders (under `ComfyUI/models/`): `checkpoints`, `clip`, `clip_vision`, `controlnet`, `diffusion_models`, `embeddings`, `loras`, `text_encoders`, `unet`, `vae`, `vae_approx`, `upscale_models`, and others. Use `--subfolder <name>` when the auto-inference is wrong. ## After run Outputs are saved under `ComfyUI/output/`. Use the `images` list from the script output to locate the files (filename + subfolder). ### ⚠️ Always send the output to the user After a successful ComfyUI run, **you must deliver the generated image(s) to the user**. Do not reply with only the filename in text or with NO_REPLY. 1. Parse the script output JSON for `images` (each has `filename`, `subfolder`, `type`). 2. Build the full path: `ComfyUI/output/` + subfolder + filename (e.g. `ComfyUI/output/z-image_00007_.png`). 3. **Send the image to the user** via the channel they're on (e.g. use the message/send tool with the image `path` so the user receives the file). Include a short caption if helpful (e.g. "Here you go." or "Tokyo street scene."). Every successful run must result in the user receiving the image. Never leave them with only a filename or no delivery. ## Resources ### scripts/ - `comfyui_run.py`: Queue a workflow, poll until completion, print `prompt_id` and `images`. No args — you edit the JSON before running. - `download_weights.py`: Download model weight URLs into `~/ComfyUI/models/<subfolder>/`. Uses [pget](https://github.com/replicate/pget) when available (installs to `~/.local/bin` if missing); fallback to built-in download. Input: URLs as args or one per line on stdin. Options: `--base`, `--subfolder`, `--overwrite`, `--no-pget`. Infers subfolder from URL/filename when not given. ### assets/ - `default-workflow.json`: Default workflow. Copy and edit (prompt, style, seed) then run with the edited path; or run as-is for a generic run.