---
name: forge-llm-streaming
description: Streaming LLM output correctly. SSE setup, partial-token UTF-8 decoding, AbortController cancellation end-to-end (browser + server + upstream), mid-stream errors as events not HTTP status, append-only UI rendering, backpressure, time-to-first-token. Contains ready-to-paste SSE handler + browser fetch reader + cancellation chain. Use when building any LLM-facing UI that should feel responsive.
license: MIT
---

# forge-llm-streaming

You are streaming LLM output to a UI. Default agent-written streaming code reads tokens, naively appends to a string, breaks on the first emoji that spans two chunks, ignores cancellation, and re-renders the entire UI on every token. The user sees flickering, occasional broken Unicode, and a stop button that does nothing. This skill exists to fix all of that.

The mental model: **streaming is a producer/consumer system over the network. The producer sends events. The consumer reassembles them, handles partial state, manages cancellation, and renders incrementally.**

## Quick reference (the things you must never ship)

1. Reading the stream into a full string, then rendering once at the end.
2. Re-rendering the entire markdown tree on every token.
3. Stop button that hides text but keeps the LLM running.
4. Mid-stream errors returned via HTTP 500 (cannot do this once 200 was sent).
5. Server endpoint missing `Cache-Control: no-cache, no-transform`.
6. SSE behind a proxy that buffers (no `X-Accel-Buffering: no` for nginx).
7. `JSON.parse(streamBuffer)` mid-stream (partial JSON throws).
8. UTF-8 decoded byte-by-byte instead of via a streaming decoder.
9. `Promise.all` over individual chunks (loses streaming).
10. UI auto-scroll yanking the user back when they scrolled up.

## Hard rules

### Transport

**1. Server-Sent Events (SSE) for most cases. WebSockets only for bidirectional.** SSE is simpler, works over plain HTTP, reconnects automatically, plays well with proxies.

**2. HTTP/2 or HTTP/3 for SSE in production.** HTTP/1.1 limits parallel SSE streams per browser to 6; HTTP/2 multiplexes.

**3. Server headers:**

```ts
// server-side, before writing the body
res.setHeader("Content-Type", "text/event-stream");
res.setHeader("Cache-Control", "no-cache, no-transform");
res.setHeader("Connection", "keep-alive");
res.setHeader("X-Accel-Buffering", "no");  // nginx will buffer otherwise
```

### Event shape

**4. Stream structured events, not raw tokens.** Each event has type + payload.

```ts
type StreamEvent =
  | { type: "delta"; text: string }
  | { type: "tool_call"; name: string; args: unknown }
  | { type: "done"; usage: { input_tokens: number; output_tokens: number } }
  | { type: "error"; code: string; message: string; retryable: boolean };
```

**5. One event per server-sent message.** Multiple events per message harder to parse and easier to corrupt.

**6. Include a sequence number on each event.** Clients can detect drops and resync.

**7. A final `done` event marks completion.** Without it, the client cannot distinguish "stream completed" from "connection dropped."

### Unicode and partial tokens

**8. Tokens are bytes, not characters. Buffer until valid UTF-8.** A Chinese character or emoji often spans 3-4 bytes; a token boundary may split them.

```ts
// browser-side: TextDecoder with streaming mode
const decoder = new TextDecoder("utf-8");
const reader = response.body!.getReader();
let buffered = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  // streaming: true keeps trailing bytes for the next call
  buffered += decoder.decode(value, { stream: true });

  // Parse complete events from `buffered`...
}
buffered += decoder.decode();  // flush any remaining
```

**9. Most SDKs do this for you. Verify yours does.** Anthropic SDK, OpenAI SDK, Vercel AI SDK all handle it. Rolling your own raw fetch and reading byte-by-byte is where corruption appears.

**10. Code blocks and JSON in streamed output are not parseable until complete.** Do not try to parse mid-stream JSON unless using a partial-JSON parser (`json-streaming`, `partial-json`).

### Tool calls in stream

**11. Tool calls arrive as multiple events: name, arguments delta, finish.** Reassemble before executing.

**12. Stream tool arguments to the UI as they arrive.** A user watching a five-second tool call appreciates seeing the arguments form in real time.

### Cancellation (the critical chain)

**13. Cancellation is first-class. Wire it through every layer.**

```ts
// 1. Browser - AbortController passed to fetch
const ctrl = new AbortController();
stopButton.onclick = () => ctrl.abort();

const response = await fetch("/api/chat", {
  method: "POST",
  body: JSON.stringify({ message }),
  signal: ctrl.signal,   // <-- propagates abort
});

// 2. Server - detect client disconnect, cancel upstream
import { AbortController } from "node:abort_controller";
export async function POST(req: Request) {
  const upstreamCtrl = new AbortController();

  // when the client disconnects, cancel the upstream LLM call
  req.signal.addEventListener("abort", () => upstreamCtrl.abort());

  const upstream = await anthropic.messages.stream(
    { model, max_tokens, messages },
    { signal: upstreamCtrl.signal },
  );

  // ... pipe upstream to the response ...
}
```

**14. The "stop" button on the UI sends abort. It does not just clear the visible text.** Visual stop without backend stop is a lie - the LLM keeps generating tokens nobody reads, costing money.

### Error handling mid-stream

**15. An error AFTER the first token is part of the stream, not an HTTP error.** Once you have sent `200 OK` and started streaming, the rest MUST be in the stream. Send an `error` event; do not change the HTTP status.

```ts
try {
  for await (const delta of upstream) {
    res.write(`data: ${JSON.stringify({ type: "delta", text: delta })}\n\n`);
  }
  res.write(`data: ${JSON.stringify({ type: "done" })}\n\n`);
} catch (err) {
  // cannot send 500 - we already sent 200 OK
  res.write(`data: ${JSON.stringify({
    type: "error",
    code: "upstream_failure",
    message: "Upstream LLM call failed mid-stream.",
    retryable: true,
  })}\n\n`);
} finally {
  res.end();
}
```

**16. The client renders mid-stream errors gracefully.** A red inline message at the end of partial output, not an alert dialog.

**17. Retry with the partial output as context if useful.** For long generations where the model failed partway, "continue from here" is sometimes the right UX.

### UI rendering

**18. Append, do not re-render.** Streaming text uses an append-only buffer. Re-rendering markdown on every token is what causes UI lag.

**19. Markdown rendering is incremental.** Use a streaming-aware markdown library, or buffer until paragraph boundaries before re-rendering.

```ts
// reference: buffer until a paragraph boundary, then re-render
let textBuffer = "";
let lastRenderedAt = 0;

function onDelta(text: string) {
  textBuffer += text;
  // re-render at most every 100ms OR on a paragraph break
  const now = performance.now();
  if (textBuffer.endsWith("\n\n") || now - lastRenderedAt > 100) {
    requestAnimationFrame(() => {
      outputEl.innerHTML = renderMarkdown(textBuffer);
      lastRenderedAt = now;
    });
  }
}
```

**20. Auto-scroll only when the user is at the bottom.** If the user scrolled up to read earlier output, do not yank them back.

```ts
function isAtBottom(): boolean {
  return outputEl.scrollHeight - outputEl.scrollTop - outputEl.clientHeight < 50;
}

function onDelta(text: string) {
  const wasAtBottom = isAtBottom();
  // ... render ...
  if (wasAtBottom) outputEl.scrollTo({ top: outputEl.scrollHeight });
}
```

**21. Show a cursor or pulse during generation.** A blinking cursor at the end of text is enough.

### Backpressure

**22. The client cannot fall behind the stream.** A slow rendering loop on the client queues events. Process synchronously per event; render via `requestAnimationFrame` batching.

**23. The server cannot overwhelm the client.** Monitor outbound bandwidth on concurrent streams.

### Reconnection

**24. SSE reconnects automatically with `Last-Event-ID`.** Assign sequence IDs if you want resumability.

**25. The client signals reconnect to the user.** A "reconnecting..." message beats silent partial output.

### Latency budget

**26. Time-to-first-token is the metric.** Users tolerate slow completion if first token arrives in <500ms. Two seconds before first token feels broken even if total completes in three.

**27. First-token latency depends on prompt length.** Long system prompts increase TTFT; prompt caching (Anthropic, OpenAI) reduces it dramatically.

## Common AI-output patterns to reject

| Pattern | Why wrong | Fix |
| --- | --- | --- |
| `const text = await response.text()` for stream | Defeats streaming | Use the SDK stream or `body.getReader()` |
| Re-render markdown on every token | UI lag | Throttle via requestAnimationFrame |
| Stop button that just hides text | LLM keeps generating | AbortController end-to-end |
| `res.status(500).json(...)` mid-stream | Cannot after 200 sent | `error` event in the stream |
| SSE response no `Cache-Control: no-cache` | nginx buffers indefinitely | Set header explicitly |
| `JSON.parse(streamBuffer)` mid-stream | Throws on partial | Wait for `done`, or use partial-JSON parser |
| `for await` then `Promise.all` | Loses streaming | Pipe event-by-event |
| `decoder.decode(value)` no `stream: true` | Breaks multi-byte chars | `stream: true` mode |
| Auto-scroll to bottom always | Yanks the reader | Check `isAtBottom` first |
| Synchronous DOM updates per token | Frame drops | requestAnimationFrame |

## Worked example: full SSE chat endpoint (Hono + Anthropic SDK)

```ts
// server: app/api/chat/route.ts
import { Hono } from "hono";
import { streamSSE } from "hono/streaming";
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();
const app = new Hono();

app.post("/api/chat", async (c) => {
  const body = await c.req.json();

  return streamSSE(c, async (stream) => {
    const upstreamCtrl = new AbortController();

    // forward client abort to the upstream LLM call
    c.req.raw.signal.addEventListener("abort", () => upstreamCtrl.abort());

    try {
      const upstream = client.messages.stream(
        {
          model: "claude-sonnet-4-6",
          max_tokens: 2000,
          messages: body.messages,
          temperature: 0,
        },
        { signal: upstreamCtrl.signal },
      );

      for await (const event of upstream) {
        if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
          await stream.writeSSE({
            data: JSON.stringify({ type: "delta", text: event.delta.text }),
          });
        }
      }

      const final = await upstream.finalMessage();
      await stream.writeSSE({
        data: JSON.stringify({
          type: "done",
          usage: { input_tokens: final.usage.input_tokens, output_tokens: final.usage.output_tokens },
        }),
      });
    } catch (err) {
      if (upstreamCtrl.signal.aborted) {
        // client cancelled; do not send error event (they're gone)
        return;
      }
      await stream.writeSSE({
        data: JSON.stringify({
          type: "error",
          code: "upstream_failure",
          message: "Upstream LLM call failed.",
          retryable: true,
        }),
      });
    }
  });
});

export default app;
```

```ts
// browser: chat.ts
const outputEl = document.getElementById("output") as HTMLElement;
const stopBtn = document.getElementById("stop") as HTMLButtonElement;
const cursor = document.getElementById("cursor") as HTMLElement;

let textBuffer = "";
let lastRenderedAt = 0;

function isAtBottom() {
  return outputEl.scrollHeight - outputEl.scrollTop - outputEl.clientHeight < 50;
}

function scheduleRender() {
  const now = performance.now();
  if (textBuffer.endsWith("\n\n") || now - lastRenderedAt > 100) {
    requestAnimationFrame(() => {
      const atBottom = isAtBottom();
      outputEl.innerHTML = renderMarkdown(textBuffer);
      cursor.style.display = "inline";
      if (atBottom) outputEl.scrollTo({ top: outputEl.scrollHeight });
      lastRenderedAt = now;
    });
  }
}

export async function chat(messages: Message[]) {
  const ctrl = new AbortController();
  stopBtn.onclick = () => ctrl.abort();

  textBuffer = "";
  outputEl.innerHTML = "";

  let response: Response;
  try {
    response = await fetch("/api/chat", {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify({ messages }),
      signal: ctrl.signal,
    });
  } catch (err) {
    if ((err as Error).name === "AbortError") return;
    throw err;
  }

  const reader = response.body!.getReader();
  const decoder = new TextDecoder("utf-8");
  let lineBuffer = "";

  try {
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      lineBuffer += decoder.decode(value, { stream: true });

      // parse complete SSE events
      const events = lineBuffer.split("\n\n");
      lineBuffer = events.pop() ?? "";
      for (const raw of events) {
        if (!raw.startsWith("data:")) continue;
        const payload = raw.slice(5).trim();
        if (!payload) continue;
        const event = JSON.parse(payload);
        if (event.type === "delta") {
          textBuffer += event.text;
          scheduleRender();
        } else if (event.type === "done") {
          // final render, hide cursor
          requestAnimationFrame(() => {
            outputEl.innerHTML = renderMarkdown(textBuffer);
            cursor.style.display = "none";
          });
          return;
        } else if (event.type === "error") {
          textBuffer += `\n\n*[error: ${event.message}]*`;
          scheduleRender();
          return;
        }
      }
    }
  } finally {
    lineBuffer += decoder.decode();  // flush
    cursor.style.display = "none";
  }
}
```

What this demonstrates: AbortController on the browser fetch (rule 13); server forwards abort to upstream (rule 13); structured events with `type` discriminator (rule 4); streaming UTF-8 decode (rule 8); paragraph-boundary throttling on UI render (rule 19); auto-scroll only when at bottom (rule 20); mid-stream error as event, not HTTP status (rule 15); proper headers set by `streamSSE` helper (rule 3).

## Workflow

When building a streaming LLM UI:

1. **Pick the transport (SSE) and define the event types.**
2. **Set the right server headers and confirm a curl test streams without buffering.**
3. **Implement the client with proper streaming decode (use the SDK).**
4. **Render in an append-only way, batch renders with requestAnimationFrame.**
5. **Wire cancellation end-to-end. Verify with browser DevTools that aborting kills the upstream call.**
6. **Handle mid-stream errors as events; render inline.**
7. **Test on a slow network.** Throttle to 3G and look at the experience.

## Verification

```bash
bash skills/llm/forge-llm-streaming/verify/check_streaming.sh path/to/stream-handler.ts
```

Flags: missing AbortController in fetch, SSE responses without no-cache, JSON.parse on streamed buffers.

## When to skip this skill

- Non-streaming LLM calls (one-shot completions where latency is not user-facing).
- Batch processing pipelines.
- Voice / audio streaming - different protocols (WebRTC, WebSocket binary).

## Related skills

- [`forge-prompt-engineering`](../forge-prompt-engineering/SKILL.md) - the prompt the stream returns from.
- [`forge-tool-use`](../forge-tool-use/SKILL.md) - tool calls within a stream.
- [`forge-frontend-nextjs`](../../design/forge-frontend-nextjs/SKILL.md) - if Next.js, see streaming RSC + this pattern.