--- name: cocoon description: >- Drive a Cocoon dataflow as a peer alongside the human: edit `cocoon.yml` and node modules, inspect graph state, run/steer nodes, peek at port data, and collaborate via presence (suggestions, callouts). Read when you're in a repo with a `cocoon.yml` and/or a `cocoon serve` core is reachable. --- # Cocoon **Agent-first, flow-based data processing.** A collaborative data-mining environment where the agent builds the graph and the human steers and monitors — and the same flow carries you from raw data to insights to running workflow automation, in one tool. You are working inside a Cocoon project: a directory containing a `cocoon.yml` (the **flow**), possibly a `nodes/` dir with the node modules it references, and (typically) a running `cocoon serve` core that the human's browser editor is connected to. You are a **peer client** of that same core, alongside the editor — never a privileged observer. Connect, ask, act, disconnect; the core stays the source of truth. > **Finding the Cocoon repo.** This skill is installed in the user's home directory, so file paths these guides reference (`examples/bgg/...`, `core/contract.ts`, `src/lib/...`) are relative to the Cocoon repo root, not your cwd. Locate it once with `dirname $(dirname $(readlink $(which cocoon)))` — the `cocoon` CLI symlinks into the repo's `core/cli.ts`. Read the referenced files there when a guide points at one; the node source is the contract. ## What the human sees A browser canvas shows the flow as a graph of nodes coloured by status, with each edge labelled by the item count pulled through it. The canvas is read-only — edges are not drawn by hand. To change the flow, either: - **the human** opens `cocoon.yml` in their own text editor and edits YAML alongside the canvas; the core watches the file and reloads with minimal disturbance, or - **you** (the agent) write the same file via raw `Edit`/`Write`, and/or announce **suggestions** — change-sets the human applies with one click. Each node carries a hover toolbar (run-to-here, persist, trash, …) and may carry a **control** — a code-declared affordance attached to the node (a steering knob, a chart, a form, an annotation UI). A control renders on two **surfaces**: inline on the node itself (`surface: 'node'` — compact) and in a detached **window** (`surface: 'window'` — roomy). You may also drop **callouts** — chat-friendly speech bubbles pointing at a node, stepped through by ◀ N ▶ in the header. ## Vocabulary - **Flow** — a dataflow graph persisted as a single `cocoon.yml`. - **Node** — one data-processing operation. One co-located source file: `process` (Node-side transform) + optional `control.{data,render,event}` (Node-side) + optional `hook` (browser-side renderer). Plain JS/TS, no build step. - **Port** — a node's input/output channel. An `in:` key whose value is a `cocoon://` URI is an **edge** (port-to-port wiring); a purely literal `in:` value is **config** (no handle, shown as a title slice). - **Edge** — `cocoon:///out/` reference, the only edge form. - **Control** — first-class node concept, peer to ports. Two tiers: - **steering** — typed, code-declared knobs (toggle/select/text/number) rendered inline; pure pull (set → `stale` → re-pull, zero side-effects); state is an ephemeral core-held overlay, never YAML. **Use for inputs that change what `process()` computes** (a fetch limit, a threshold, an algorithm mode). A choice that only re-shapes already-computed data for display — grouping, metric, sort within a visualisation — is *presentation*, not a graph input: put it inside the free-form control as a `control.event` (re-derive, no rerun), not a steering knob that forces a full re-pull. - **free-form** — server-built HTML, optionally with an author-written browser hook. Split: `control.data` (core-side, async, bounded) → `control.render` (HTML/+hook, per `ctx.surface` = `'node'` inline or `'window'` detached) → `control.event` (durable write + `markStale`; a selection is just an event). **No schema — the node *is* the control.** - **Visualisation** — a control with a render hook and no `event`; a selectable one adds `event`. Controls *are* the view layer. - **Hook** — the browser half of a node: an imperative `mount/update/destroy` renderer exported from the same source file. ## Architecture: one core, many clients A standalone, transport-agnostic Node **core** owns the runtime, the resolver, processing and all port data. The browser **editor** is a pure viewer (no save path, no edge-connect, no YAML pane) that loads the file itself and receives only a stream of per-node *state* over one WebSocket — never bulk data. **You connect to the same core** via the CLI, alongside the editor; reads, runs, and presence updates are simultaneous and visible in both. A separate headless mode (`cocoon run --target …`) owns its own throwaway Runtime and streams one port to stdout. Use it only when specifically requested. ## The cocoon.yml format **There is no schema — the loader honours every key it doesn't understand** (no in-app writer means nothing gets dropped on disk). Shape: ```yaml description?: 'free text' env?: { … } # merged into process.env (under .env / .env.defaults) nodeDirs?: ['~/my-project/nodes'] # extra node roots nodes: # required : type: # required; resolved by convention (see below) '?': 'inline docs' # always write one; also accepted as `description:` group?: 'a/slash/path' # semantic visual cluster persist?: true|false # serve cached output from disk in?: : : [, , …] # multi-edge: concat out?: : ~ # statically-seeded output port ``` - **Always write a `'?'` help text.** Every node you add should carry one — a self-contained statement of what *this* node does. Don't refer to other nodes in it (no "feeds X", "after Y", "like Z"); wiring is the graph's job, and such references rot when the flow changes. - **Node ids** are the keys under `nodes:`; they are the only identity references use. Renaming is `Edit` across the file. - **`type:` resolves by convention** — no registry. The core looks for `/nodes/.{ts,js,…}` and in any `nodeDirs:` root (leading `~/` expands to `$HOME/`). A duplicate type name across roots is a hard error (never shadowing). - **Edge vs config — the grammar's sole discriminator.** An `in:` value is an edge iff it matches `cocoon:///out/` exactly; anything else is a literal config value (code string, number, nested object/array), preserved verbatim and shown as a title slice on the node. There are no empty input stubs; converting config↔port is a one-line YAML edit. - **Multi-edge concat.** `in: { data: [cocoon://A/out/x, cocoon://B/out/y] }` feeds the node `A.x ⧺ B.y` (`Array.flat()` depth 1). The node receives a flat list and must never re-flatten. - **Comments and unknown keys are preserved on disk** because nothing writes the file. Edit freely; formatting is yours. - **What is NOT in the file:** persist toggle state, control state, control drafts, suggestions — all runtime overlays, ephemeral by design. The authoritative source for *what `type` means* is the node module file, not the YAML. ## Editing the flow Edit `cocoon.yml` and node modules as **text**, via raw `Edit`/`Write` — there is no structural API and no save path in the editor. The core watches the flow file: a save triggers a **selective reload** (see below). For an explicit reload after a programmatic edit, run `cocoon reload`. Node *module code* does not need a reload at all — it is hot-swapped at execution time by the resolver when its mtime changes. This covers the node's **sibling libs** too: the resolver keys re-import on the newest mtime across the entry and everything it imports via relative `./` paths, so editing a shared helper hot-reloads the nodes that import it (no `serve` restart). The only thing that needs a `serve` restart is core-runtime code (the runtime itself, the resolver, the protocol). ### Reload semantics `cocoon reload` (and the watcher) re-parse the YAML and apply a selective diff: per node, comparing its **compute signature** (`type`, `in:`, static `out:`) plus its entire transitive upstream: - self + upstream unchanged → **preserved** (output kept) - self unchanged, upstream moved → **`stale`** (last output still visible) - self changed / brand-new → **reset `idle`** - removed → **purged** Persisted nodes that were *reset* re-hydrate from disk. Editing a comment, `group`, `?`, or any unknown pass-through key costs zero state. A `nodeDirs:` / `env:` change is a full reset — as is an edit to the flow-local `.env` / `.env.defaults`, which the core watches alongside the flow file, so a credential fix reloads on its own (no `serve` restart). ## Execution model *Pull, not push.* Nothing recomputes behind your back: you **run to** a node and the core processes it plus its transitive upstream in topological order, memoising completed upstream nodes. The explicitly-pulled target always re-runs (the persist-cache fast path still applies; persist *is* "serve cached"). Six streamed statuses — `idle · queued · running · done · stale · error` — the only thing the editor colours by. - **`stale`** = inputs changed, result deliberately kept (the in-memory output stays visible; `process` to refresh). Re-running a node ages everything reachable downstream. - **Stale upstream is reused by default.** "Run to here" memoises a `stale` upstream like a `done` one — its kept output feeds downstream, `process()` is not re-entered, and the consumer finishes `stale` too (a derivative-of-stale result is never shown as fresh). The cheap-iteration default: hammer a downstream node without paying for an expensive upstream chain. To force a recompute, pull the upstream directly (the target always recomputes) or pass `--rerun-stale` (editor shift-click). So a new value from `set-control`/`control-event` takes effect only when you pull *that* node — pulling its downstream reuses the pre-change output. - **Errors block downstream.** A failed node surfaces as `error`; its dependents become `error "Blocked — upstream X failed"`. Independent branches still run. - **Three result-clearing semantics:** *persist toggle off* deletes the on-disk cache only (live result + `done` stay); *trash* drops output + cache → `idle`; *stale* is the automatic one above. - **Persist is a runtime override, never YAML.** Resets on `serve` restart. ## Talking to the core: the CLI Requires a running `cocoon serve [--port N]`. Default target is `ws://localhost:22242`; override with `--core ` or `COCOON_CORE`. Exit codes: `0` ok · `1` query failed · `2` no core reachable. Invoke as `cocoon …` from anywhere — the global CLI is the supported entry point. (`pnpm core …` is only available when the cwd is the cocoon repo itself.) ``` # Read (does not change state) cocoon query overview # status, counts, loadErrors, type histogram cocoon query node # status, error/errorStack/errorAt, inputDigest, # modulePath, controls/controlState, controlData, # logCount + logTail (newest 3 ctx.debug lines) cocoon query logs [--limit N] # buffered ctx.debug() + control-hook errors ([hook]) cocoon query upstream [--depth N] cocoon query downstream [--depth N] cocoon query peek [--descend FIELD] [--where 'x => …'] [--select a,b,c] [--limit N] [--expand F[,F2,…]] # iterate these fields in `sample` rows # (one level deep, capped at 50 elements) cocoon presence # other clients' open controls / drafts / selection # Act cocoon process [--rerun-stale] # run on the LIVE session; blocks until settled. # Default: stale upstream is reused (target may # finish `stale`). --rerun-stale forces every # stale upstream to recompute first. cocoon cancel # stop a running node; lands `error: Cancelled`, # output dropped, downstream blocks. No-op if idle. cocoon set-control # steer a declared knob; pure pull (node → stale) cocoon control-event # fire a declared control.event headlessly (write half); [--json ''] # staleness is the handler's call (see Free-form controls) cocoon refresh-control # re-derive a free-form control, no pull (read half; # = control-event $mount). After a direct file write. cocoon reload # re-read the flow file after a YAML edit cocoon switch # re-point the running core at another flow # (file or dir). Fresh Runtime; old session state dropped. cocoon suggest # propose a control edit; BLOCKS for Apply/Discard [--json ''] [--label NAME] [--note TEXT] [--timeout MS] cocoon callout "" # drop a chat-friendly POINTER (labels C1, C2, …) [--id ID] [--tone info|warn|error] [--from NAME] cocoon callout-clear # dismiss your own callout cocoon errors # subscribe to the error stream over WS; one batch # per fresh failure — node errors + control-hook errors. # Long-lived — designed for a Monitor. ``` **All output is bounded.** Even `peek` returns a per-key schema + a small sample, not the rows; size tracks the schema, never the row count. A 153k-row port never crosses the wire. Arrays inside `sample` cells are shape-collapsed by default (`‹array [{title,year,…}] ×4›`); name the field in `--expand` to iterate it instead — single-level descent, 50-element cap, schema `example` stays bounded. Use it when a candidate row carries short structured arrays (`exemplars`, `top`, …) and you want the actual values, not the shape. **`ctx.debug()` is captured per node, not lost to stdout.** Each node buffers its most recent run's `ctx.debug()` lines (control `data`/`event` debug too, plus browser control-hook errors — a `hook`'s `mount`/`update`/`destroy` throw — tagged `[hook]`). `query node` shows the newest 3 inline (`logTail`) plus the total `logCount`; `query logs ` returns the full bounded buffer (newest 500, `--limit N` for fewer). `overview` shows only the aggregate `logLines` count. The buffer is ephemeral — it resets when the node re-runs and is gone on restart. This is where a node's own progress/diagnostic prints surface; for a failure, `error`/`errorStack` on `query node` is usually enough, then reach for `query logs` when the node logged its way to the bug. **`modulePath` is your way into a node.** Returned by `query node`, it's the absolute path of the file backing the node's `type`. **Read it** — the source IS the documentation (the YAML is wiring only). Reading the code is the primary way to learn how to work with an interactive node, so its docs live in comments: the code already shows *what* and *how*, while the *why* and suggested usage belong in a compact top-of-file comment — read that first. It is also the **only** way to learn a free-form control's field names: they are HTML `name` attributes inside `control.render`, which you never see rendered. **`cocoon switch ` re-points the *running* core at a different flow** — what the human does by clicking the header path and picking a recent. Fresh Runtime, every client repaints, the watcher follows; all old-flow session state (persist/control overlays, results) is dropped. Missing/unparseable file → hard no-op (exit 1), current flow untouched. Recents live at `~/.cocoon/recents.json`. **`set-control` and `reload` mark `stale` but run nothing — `process` to apply.** Pick up the new value by `process`ing the changed node *itself* (the target always recomputes); `process`ing a downstream reuses the stale output instead. `set-control` JIT-resolves the module, so a just-edited schema is honoured without a prior pull; a write the schema rejects (unknown key, wrong kind/range, or unknown control) is a silent no-op surfaced as `IGNORED` (exit 0; an unknown *node* is exit 1). **Free-form controls over the CLI — two halves.** A free-form control's `control.data` is a live *projection* (re-read from its durable file every derive); re-deriving it re-streams `controlData`/HTML to every client with **no `process()`, no `stale`, no status change** — pure presentation, still pull-only. - **`control-event --json ''` — write half.** Fires one of the node's declared `control.event` handlers exactly as a UI click does (`control.event(ctx, {event, payload})`, then re-derive). Prefer it over a hand-rolled `Write`/`Edit` whenever the node declares the operation: you reuse its validated handler (with ports/context) instead of rebuilding the on-disk shape, and a single event can't clobber a human edit landed since your last read. No new capability — only events the node handles (Read `modulePath` for names + payload shapes). Staleness is the *handler's* call: one that runs `ctx.markStale()` (e.g. `merge_done`) ages the node + downstream → then `process` to fold; one that doesn't (`cell_edit`, `seed_rows`) stays pure presentation. - **`refresh-control ` — read half.** Sugar for the reserved `$mount` event (skips the handler, just re-derives). Fire it after writing the node's OWN durable file directly, so the human watches the table fill in real time. No-op on a node with no free-form control. Use `process` instead when a write must flow *downstream* — that's a graph change, not a view refresh. **`process` and `suggest` resolve on a value, not a message count.** `process` waits for the streamed status to settle terminal; `suggest` waits for the peer presence echo of your `ChangeSet.id`. Both can block indefinitely — use `--timeout` on `suggest` if the human may be away. For a long-running `process`, fire it with `Bash(cocoon process X, run_in_background: true)` — the harness notifies you on completion, no monitor verb needed. To abort one mid-flight (a crawl you no longer want, a runaway fetch), `cocoon cancel `: it's cooperative — the run's `ctx.signal` aborts and the core stops driving the generator at its next `yield`/`breathe`, so it lands within a tick or two as `error: "Cancelled"` with its output dropped (downstream blocks like any failure; re-`process` to clear). A node that isn't running is a no-op. Any blocked `process` puller of that node unblocks too. ## Collaborating with the human Presence is an **optional, orthogonal side-channel**. Each connected client (editor tab, agent) may announce an opaque blob; the core relays it and interprets nothing. **Nothing in processing depends on it.** Empty presence is normal — it doesn't mean broken. Three primitives, each with its own semantics: - **Suggestion** (`cocoon suggest`) — the human↔AI **write path**. You read the human's *unsaved* control text from presence (`controlDrafts[node][field]`, never scraped from HTML), do the work, and announce a change-set as your own presence. The editor surfaces it as one toast; **Apply only injects the value into the still-unsaved field** — durability is the human's own Save afterwards. The verdict rides back in the editor's presence; `suggest` blocks until you get `applied` / `discarded` / `stale` (the surface moved on; self-invalidated). - **Callout** (`cocoon callout`) — a chat-friendly **pointer at a node**, not a CTA. Use it to give your chat conversation a handle: "at C2 — should we drop its `view:`?". Fire-and-forget: the editor snapshots callouts on first observation, so the marker survives your disconnect. The human's reply belongs in chat, not the editor. Close the loop with `callout-clear` when the flagged work is done. - **Reading presence** (`cocoon presence`) — see every other client's blob: open controls (`openControls`), unsaved drafts (`controlDrafts`), node selection (`selectedNodes` — single click or shift-drag rectangle), viewport, label. The mirror of your callouts: agent → human is `callout`, human → agent is `selectedNodes`. **Rules:** - **Presence is connection-keyed and evaporates on disconnect.** (One-shot `suggest` holds its socket open by design until the verdict arrives.) - **Presence is never a data path.** `controlDrafts` is the human's UI text; don't gate processing on it; don't treat it as a port. - **Free-form controls have no schema.** The node *is* the contract. To know which fields exist, **Read** `modulePath`. Inventing a field name Applies into nothing. - **An empty `controlDrafts` is not a blocker.** "Help me fill out this form" with an empty draft is the same loop as "translate what I pasted" with a full one — just no input text to transform. `controlData` (in `query node`) holds the bounded slice the human is currently looking at; the "which row is shown" answer almost always lives there. ## How the human refers to things The human might not use the terms above. Map their words; but use the correct terminology in your reply. - "the flow" / "the graph" — the `cocoon.yml` + its live core session - "a node" / "this node" / "the X node" — a node id (look at `query overview` if unsure) - "the form" / "the dialog" / "this control" — the free-form control on the focused node - "what I have open" / "the thing I'm working on" — `presence` → first peer's `openControls`; `controlDrafts` for its content - "these nodes" / "the selection" — `presence` → first peer's `selectedNodes[]` - "this field" / "the X field" — one form-field `name` inside `control.render` — read `modulePath` to learn the names - "what I typed" / "my draft" / "what I pasted" — `presence[…].controlDrafts[node][field]` verbatim - "a knob" / "a setting" / "the toggle" — a code-declared steering control — read via `query node`, write via `set-control` - "run it" / "recompute" / "refresh the data" — `cocoon process ` on the live session (a graph change; flows downstream) - "refresh the table/view" / "update what I'm looking at" — `cocoon refresh-control ` after you wrote the node's file (a view re-derive; no pull) - "add these rows" / "mark this done" / "edit that cell" / "commit the merge" — a node-declared `control.event` → `cocoon control-event --json '…'` (Read `modulePath` for the event names; prefer over a raw file write when the node already handles it) - "suggest" / "draft this" / "help me fill out" — `cocoon suggest` → one Apply/Discard toast - "flag this" / "point at X" / "highlight X" — `cocoon callout ""` — labels `C1`, `C2`, … ## Interaction rules - **Bare invocation: assume flow work, then clarify.** Default the intent to "the human wants to work on a `cocoon.yml`", but ask which file (and whether to create or resume) before acting. - **Bootstrap eagerly.** New flow: write a minimal `cocoon.yml` (one node) and start `cocoon serve` as soon as the first node exists, so the human can follow along on the canvas. - **Resume eagerly.** Existing flow: run `cocoon serve &` first thing — no pre-check. If a core is already serving the same file, the new invocation auto-attaches and exits 0 (prints the URL). - **Open the canvas, once.** On the first `serve` of a session, `open ` so the human gets a tab. Don't reopen on subsequent restarts — the existing tab reconnects on its own. - **Watch errors proactively.** Arm a `Monitor` on `cocoon errors` immediately after `serve`. The verb subscribes to the live core's failure stream over WS and prints one batch per transition into error state: `node "" failed` + the real stack. Works whether you launched the core or attached to a human-started one. Each batch is usually enough to diagnose without `query node`; fall back to `query node` only when the stack can't name the bug. **Browser control-hook errors ride the same stream** — tagged `node "" control error (hook)`. They're a crashed *visualisation*, not a failed node: status is untouched and downstream isn't blocked, so the data may be fine — reach for `query logs ` (the `[hook]` lines carry the full stack) rather than `query node`. ## Rules to know before acting - **The flow file is the wiring; the modules are the flow.** YAML edits go on `cocoon.yml`. Behaviour edits go on the node module file (`Read` it first; `modulePath` from `query node` is the path). Both are picked up live. - **All graph state-changes are pull-driven.** Edits, `set-control`, and `reload` only mark `stale`; nothing runs without `process`. - **The connect handshake replays everything** (`hello` with your `clientId` + `graph` + per-node state + presence) before anything you ask. The CLI handles this; a custom client must attach its listener before opening the socket. - **A loadError on a node module is a common silent blocker.** Check `query overview` → `loadErrors` first when a node won't run. - **`inputDigest` is the high-value debug field.** A `node` query at error time shows the bounded shape of what `process()` was actually fed — almost always names the bug. `errorAt` (nodes using `trackedMap`/`trackedFilter`) pinpoints the exact offending row. - **Don't HTML-scrape what the human sees.** `controlDrafts` is the only reliable source for current control values, `modulePath` for the schema. ## Writing new nodes When the task is **authoring** a node module (rather than driving an existing flow), read [`writing-nodes.md`](./writing-nodes.md) — a companion guide covering: the `process` contract, steering controls, free-form controls (`data`/`render`/`event`), the browser `hook`, the default dark-theme styles `CocoonNode` already ships, the project palette, design considerations for the two surfaces, and a minimal scaffold to start from. When the node carries a **chart**, also read [`writing-charts.md`](./writing-charts.md) — a survey of the eight embeddable libraries (ECharts, Observable Plot, D3, Vega-Lite, Plotly, uPlot, Chart.js, Cytoscape), a decision matrix for picking one by the constraint that hurts most, per-library notes on chart types / perf / brushing / CDN pin / gotchas, and a verified WebFetch-friendly URL per library for getting the full chart-type catalogue. Each library has a runnable standout demo in `examples/charts/nodes/`.