Docs · Streaming

Server-Sent Events for token-by-token output.

Stream model output as it's generated. Sub-second time-to-first-token, full guardrail metadata in the final event, and graceful mid-stream error handling.

Quickstart

Enable streaming

Set stream: true on the request body. The response uses Server-Sent Events with Content-Type: text/event-stream. The OpenAI and Anthropic SDKs handle the SSE parsing automatically.

from openai import OpenAI

client = OpenAI(base_url="https://api.anoman.io/v1", api_key="anm-sk-...")

stream = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Tell me a short story"}],
    stream=True,
)

for chunk in stream:
    # OpenAI SDK parses SSE for you — chunk is a typed object
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)

Edge proxy note. Our edge proxies forward SSE frames as they arrive — each chunk resets the proxy timeout, so streaming requests survive arbitrarily long completions. Non-streaming requests are subject to a 100-second proxy timeout; use streaming for long-thinking models (e.g., R1, Opus).

Where streaming works

Three endpoints support streaming

Pass stream:true on any of these endpoints and the gateway responds with Server-Sent Events. The full guardrail + metering pipeline runs identically on every path.

POST /v1/chat/completions — the OpenAI-compatible chat endpoint. Chunks follow OpenAI's chat.completion.chunk format (shown above).
POST /anthropic/v1/messages — the Anthropic Messages endpoint. Frames follow the Anthropic event format (message_start → content_block_delta → message_stop) instead of OpenAI chunks. See /docs/endpoints/anthropic-messages.
POST /anoman/v1/playground/run — the dashboard/studio playground endpoint that powers the 3-pane compare tool.

Wire format

Chunk anatomy

Each frame is a single line: data: <json> followed by a blank line. The stream is terminated with data: [DONE]. Chunks match OpenAI's chat.completion.chunk shape, except the second-to-last frame is the Anoman metadata block.

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1709876543,"model":"claude-sonnet-4-6","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Once "},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"upon a "},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"time..."},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":7,"completion_tokens":4,"total_tokens":11}}

data: {"_anoman":{"guardrails":{"injection":{"status":"pass","score":0.02}},"routing":{"mode":"realtime","region":"id"},"cache":{"hit":false},"cost_usd":"0.000041","weighted_tokens":44}}

data: [DONE]

First chunk: delta.role: "assistant" with empty content.
Intermediate chunks: delta.content token by token.
Last completion chunk: finish_reason populated + usage totals.
Penultimate frame: {"_anoman": {...}} — guardrail results, routing region, cache hit, cost, weighted tokens.
Terminator: data: [DONE].

Mid-stream errors

When upstream breaks halfway

Pre-call failures (auth, rate limit, guardrail block) return a normal HTTP error before the SSE response opens. Once streaming has started, the HTTP status is already 200 — so mid-stream upstream failures are signaled by an error frame in the stream body:

data: {"error":{"type":"provider_error","code":"stream_interrupted","message":"upstream stream interrupted","_anoman":{"failure_class":"upstream_error","completion_tokens":42}}}

data: [DONE]

Your client should check every parsed chunk for an error key. When present, treat the response as failed (don't commit the partial output) and apply your normal retry logic. The OpenAI SDK raises an exception automatically when it sees this frame.

Partial content note. The _anoman.completion_tokens field tells you how many completion tokens the model emitted before the failure. Useful for telemetry — and for deciding whether a partial result is salvageable in your UX.

Raw SSE handling

Without an SDK

When you can't use the OpenAI/Anthropic SDK — edge runtimes, alternative HTTP clients, custom proxies — here's a minimal SSE parser:

// Useful when you need fine-grained control — e.g. piping straight
// to a WebSocket, or running in an environment without the OpenAI SDK.

const response = await fetch(
  "https://api.anoman.io/v1/chat/completions",
  {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.ANOMAN_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [{ role: "user", content: "Hi" }],
      stream: true,
    }),
  },
);

if (!response.ok || !response.body) {
  throw new Error(`HTTP ${response.status}`);
}

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });

  // Split SSE messages on blank lines
  const lines = buffer.split("\n");
  buffer = lines.pop() ?? "";

  for (const line of lines) {
    if (!line.startsWith("data: ")) continue;
    const payload = line.slice(6).trim();
    if (payload === "[DONE]") return;
    const parsed = JSON.parse(payload);
    if (parsed.error) {
      throw new Error(parsed.error.message);
    }
    const delta = parsed.choices?.[0]?.delta?.content;
    if (delta) process.stdout.write(delta);
  }
}

Cancellation

Stopping a stream early

Closing the HTTP connection cancels the upstream call within ~50 ms. Tokens generated before cancellation are still billed; the dashboard logs the cancellation as routing_mode: streaming_cancelled.

import time
from openai import OpenAI

client = OpenAI(base_url="https://api.anoman.io/v1", api_key="anm-sk-...")

stream = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Write a 5-paragraph essay"}],
    stream=True,
)

deadline = time.time() + 5  # cancel after 5 seconds
collected = []
for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    collected.append(delta)
    if time.time() > deadline:
        stream.close()  # closes the HTTP connection
        break

print("Collected:", "".join(collected))

Guardrails & metering

Identical to non-streaming

Streaming changes only how bytes reach you — not what the gateway does around the call. Every request runs through the same pipeline as a non-streaming request.

Guardrails run before the first token. Pre-call checks (injection, PII, content moderation, policy) execute before the stream opens. A blocked request never starts streaming — it returns a normal HTTP error (e.g. 403), not an SSE body.
Metering is recorded at stream close. Token usage, weighted tokens, cost, and the trace come from the provider's final usage — the same numbers as a non-streaming call. A cancelled stream still meters the partial usage.
Errors leak nothing. A mid-stream provider failure emits a generic error frame (upstream_error) with no internal detail — the raw provider message stays in server logs only.

What's not supported in streaming

Known limitations

Batch routing — streaming requests always use the real-time path. Batch is for non-interactive workloads where 5–30 minute SLA is acceptable in exchange for ~50% cost.
MCP per-tool RBAC — currently enforced on JSON responses only. If your workflow needs strict per-tool denial, call with stream: false until streaming RBAC ships.
Semantic cache writes happen at stream close, not per-chunk. Cache hits short-circuit the stream and return JSON (since the full response is already known).

Try streaming in the dashboard playground.

3-pane compare with streaming output side-by-side.

Open playground Back to docs