Docs · Streaming

Server-Sent Events for token-by-token output.

Stream model output as it's generated. Sub-second time-to-first-token, full guardrail metadata in the final event, and graceful mid-stream error handling.

Quickstart

Enable streaming

Set stream: true on the request body. The response uses Server-Sent Events with Content-Type: text/event-stream. The OpenAI and Anthropic SDKs handle the SSE parsing automatically.

from openai import OpenAI

client = OpenAI(base_url="https://api.anoman.io/v1", api_key="anm-sk-...")

stream = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Tell me a short story"}],
    stream=True,
)

for chunk in stream:
    # OpenAI SDK parses SSE for you — chunk is a typed object
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)

Cloudflare proxy note. Our edge proxies forward SSE frames as they arrive — each chunk resets the proxy timeout, so streaming requests survive arbitrarily long completions. Non-streaming requests are subject to a 100-second proxy timeout; use streaming for long-thinking models (e.g., R1, Opus).

Wire format

Chunk anatomy

Each frame is a single line: data: <json> followed by a blank line. The stream is terminated with data: [DONE]. Chunks match OpenAI’s chat.completion.chunk shape, except the second-to-last frame is the Anoman metadata block.

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1709876543,"model":"claude-sonnet-4-6","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Once "},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"upon a "},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"time..."},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":7,"completion_tokens":4,"total_tokens":11}}

data: {"_anoman":{"guardrails":{"injection":{"status":"pass","score":0.02}},"routing":{"mode":"realtime","region":"id"},"cache":{"hit":false},"cost_usd":"0.000041","weighted_tokens":44}}

data: [DONE]

First chunk: delta.role: "assistant" with empty content.
Intermediate chunks: delta.content token by token.
Last completion chunk: finish_reason populated + usage totals.
Penultimate frame: {"_anoman": {...}} — guardrail results, routing region, cache hit, cost, weighted tokens.
Terminator: data: [DONE].

Mid-stream errors

When upstream breaks halfway

Pre-call failures (auth, rate limit, guardrail block) return a normal HTTP error before the SSE response opens. Once streaming has started, the HTTP status is already 200 — so mid-stream upstream failures are signaled by an error frame in the stream body:

data: {"error":{"type":"provider_error","code":"upstream_error","message":"upstream stream interrupted","_anoman":{"failure_class":"upstream_error","accumulated_chars":127}}}

data: [DONE]

Your client should check every parsed chunk for an error key. When present, treat the response as failed (don’t commit the partial output) and apply your normal retry logic. The OpenAI SDK raises an exception automatically when it sees this frame.

Partial content note. The _anoman.accumulated_chars field tells you how many characters the model emitted before the failure. Useful for telemetry — and for deciding whether a partial result is salvageable in your UX.

Raw SSE handling

Without an SDK

When you can’t use the OpenAI/Anthropic SDK — edge runtimes, alternative HTTP clients, custom proxies — here’s a minimal SSE parser:

// Useful when you need fine-grained control — e.g. piping straight
// to a WebSocket, or running in an environment without the OpenAI SDK.

const response = await fetch(
  "https://api.anoman.io/v1/chat/completions",
  {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.ANOMAN_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [{ role: "user", content: "Hi" }],
      stream: true,
    }),
  },
);

if (!response.ok || !response.body) {
  throw new Error(`HTTP ${response.status}`);
}

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });

  // Split SSE messages on blank lines
  const lines = buffer.split("\n");
  buffer = lines.pop() ?? "";

  for (const line of lines) {
    if (!line.startsWith("data: ")) continue;
    const payload = line.slice(6).trim();
    if (payload === "[DONE]") return;
    const parsed = JSON.parse(payload);
    if (parsed.error) {
      throw new Error(parsed.error.message);
    }
    const delta = parsed.choices?.[0]?.delta?.content;
    if (delta) process.stdout.write(delta);
  }
}

Cancellation

Stopping a stream early

Closing the HTTP connection cancels the upstream call within ~50 ms. Tokens generated before cancellation are still billed; the dashboard logs the cancellation as routing_mode: streaming_cancelled.

import time
from openai import OpenAI

client = OpenAI(base_url="https://api.anoman.io/v1", api_key="anm-sk-...")

stream = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Write a 5-paragraph essay"}],
    stream=True,
)

deadline = time.time() + 5  # cancel after 5 seconds
collected = []
for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    collected.append(delta)
    if time.time() > deadline:
        stream.close()  # closes the HTTP connection
        break

print("Collected:", "".join(collected))

What’s not supported in streaming

Known limitations

Batch routing — streaming requests always use the real-time path. Batch is for non-interactive workloads where 5–30 minute SLA is acceptable in exchange for ~50% cost.
MCP per-tool RBAC — currently enforced on JSON responses only. If your workflow needs strict per-tool denial, call with stream: false until streaming RBAC ships.
Semantic cache writes happen at stream close, not per-chunk. Cache hits short-circuit the stream and return JSON (since the full response is already known).

Try streaming in the dashboard playground.

3-pane compare with streaming output side-by-side.

Open playground Back to docs