Stream model output as it's generated. Sub-second time-to-first-token, full guardrail metadata in the final event, and graceful mid-stream error handling.

Enable streaming

Set stream: true on the request body. The response uses Server-Sent Events with Content-Type: text/event-stream. The OpenAI and Anthropic SDKs handle the SSE parsing automatically.

Python — OpenAI SDK

from openai import OpenAI
 
client = OpenAI(base_url="https://api.anoman.io/v1", api_key="anm-sk-...")
 
stream = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Tell me a short story"}],
    stream=True,
)
 
for chunk in stream:
    # OpenAI SDK parses SSE for you — chunk is a typed object
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)

TypeScript — OpenAI SDK

import OpenAI from "openai";
 
const client = new OpenAI({
  baseURL: "https://api.anoman.io/v1",
  apiKey: process.env.ANOMAN_API_KEY,
});
 
const stream = await client.chat.completions.create({
  model: "claude-sonnet-4-6",
  messages: [{ role: "user", content: "Tell me a short story" }],
  stream: true,
});
 
for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content ?? "";
  process.stdout.write(delta);
}

cURL

# Use -N to disable buffering — see chunks as they arrive
curl -N https://api.anoman.io/v1/chat/completions \
  -H "Authorization: Bearer anm-sk-..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [{"role": "user", "content": "Tell me a short story"}],
    "stream": true
  }'

Edge proxy note. Our edge proxies forward SSE frames as they arrive — each chunk resets the proxy timeout, so streaming requests survive arbitrarily long completions. Non-streaming requests are subject to a 100-second proxy timeout; use streaming for long-thinking models (e.g., R1, Opus).

Three endpoints support streaming

Pass stream:true on any of these endpoints and the gateway responds with Server-Sent Events. The full guardrail + metering pipeline runs identically on every path.

POST /v1/chat/completions — the OpenAI-compatible chat endpoint. Chunks follow OpenAI's chat.completion.chunk format (shown above).
POST /anthropic/v1/messages — the Anthropic Messages endpoint. Frames follow the Anthropic event format (message_start → content_block_delta → message_stop) instead of OpenAI chunks. See /docs/endpoints/anthropic-messages.
POST /anoman/v1/playground/run — the dashboard/studio playground endpoint that powers the 3-pane compare tool.

Each frame is a single line: data: <json> followed by a blank line. The stream is terminated with data: [DONE]. Chunks match OpenAI's chat.completion.chunk shape, except the second-to-last frame is the Anoman metadata block.

Wire format — example chunks

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1709876543,"model":"claude-sonnet-4-6","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Once "},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"upon a "},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"time..."},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":7,"completion_tokens":4,"total_tokens":11}}

data: {"_anoman":{"guardrails":{"injection":{"status":"pass","score":0.02}},"routing":{"mode":"realtime","region":"id"},"cache":{"hit":false},"cost_usd":"0.000041","weighted_tokens":44}}

data: [DONE]

First chunk: delta.role: "assistant" with empty content.
Intermediate chunks: delta.content token by token.
Last completion chunk: finish_reason populated + usage totals.
Penultimate frame: {"_anoman": {...}} — guardrail results, routing region, cache hit, cost, weighted tokens.
Terminator: data: [DONE].

When upstream breaks halfway

Pre-call failures (auth, rate limit, guardrail block) return a normal HTTP error before the SSE response opens. Once streaming has started, the HTTP status is already 200 — so mid-stream upstream failures are signaled by an error frame in the stream body:

Mid-stream error frame

data: {"error":{"type":"provider_error","code":"stream_interrupted","message":"upstream stream interrupted","_anoman":{"failure_class":"upstream_error","completion_tokens":42}}}
 
data: [DONE]

Your client should check every parsed chunk for an error key. When present, treat the response as failed (don't commit the partial output) and apply your normal retry logic. The OpenAI SDK raises an exception automatically when it sees this frame.

Partial content note. The _anoman.completion_tokens field tells you how many completion tokens the model emitted before the failure. Useful for telemetry — and for deciding whether a partial result is salvageable in your UX.

Without an SDK

When you can't use the OpenAI/Anthropic SDK — edge runtimes, alternative HTTP clients, custom proxies — here's a minimal SSE parser:

TypeScript — raw fetch (no SDK)

// Useful when you need fine-grained control — e.g. piping straight
// to a WebSocket, or running in an environment without the OpenAI SDK.
 
const response = await fetch(
  "https://api.anoman.io/v1/chat/completions",
  {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.ANOMAN_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [{ role: "user", content: "Hi" }],
      stream: true,
    }),
  },
);
 
if (!response.ok || !response.body) {
  throw new Error(`HTTP ${response.status}`);
}
 
const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
 
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
 
  // Split SSE messages on blank lines
  const lines = buffer.split("\n");
  buffer = lines.pop() ?? "";
 
  for (const line of lines) {
    if (!line.startsWith("data: ")) continue;
    const payload = line.slice(6).trim();
    if (payload === "[DONE]") return;
    const parsed = JSON.parse(payload);
    if (parsed.error) {
      throw new Error(parsed.error.message);
    }
    const delta = parsed.choices?.[0]?.delta?.content;
    if (delta) process.stdout.write(delta);
  }
}

Stopping a stream early

Closing the HTTP connection cancels the upstream call within ~50 ms. Tokens generated before cancellation are still billed; the dashboard logs the cancellation as routing_mode: streaming_cancelled.

Python — cancel mid-stream

import time
from openai import OpenAI
 
client = OpenAI(base_url="https://api.anoman.io/v1", api_key="anm-sk-...")
 
stream = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Write a 5-paragraph essay"}],
    stream=True,
)
 
deadline = time.time() + 5  # cancel after 5 seconds
collected = []
for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    collected.append(delta)
    if time.time() > deadline:
        stream.close()  # closes the HTTP connection
        break
 
print("Collected:", "".join(collected))

TypeScript — cancel via AbortController

const controller = new AbortController();
 
setTimeout(() => controller.abort(), 5000);  // cancel after 5s
 
try {
  const stream = await client.chat.completions.create(
    {
      model: "claude-sonnet-4-6",
      messages: [{ role: "user", content: "Write a 5-paragraph essay" }],
      stream: true,
    },
    { signal: controller.signal },
  );
  for await (const chunk of stream) {
    process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  }
} catch (e) {
  if ((e as Error).name === "AbortError") {
    console.log("\n[cancelled]");
  } else {
    throw e;
  }
}

Identical to non-streaming

Streaming changes only how bytes reach you — not what the gateway does around the call. Every request runs through the same pipeline as a non-streaming request.

Guardrails run before the first token. Pre-call checks (injection, PII, content moderation, policy) execute before the stream opens. A blocked request never starts streaming — it returns a normal HTTP error (e.g. 403), not an SSE body.
Metering is recorded at stream close. Token usage, weighted tokens, cost, and the trace come from the provider's final usage — the same numbers as a non-streaming call. A cancelled stream still meters the partial usage.
Errors leak nothing. A mid-stream provider failure emits a generic error frame (upstream_error) with no internal detail — the raw provider message stays in server logs only.

Known limitations

Batch routing — streaming requests always use the real-time path. Batch is for non-interactive workloads where 5–30 minute SLA is acceptable in exchange for ~50% cost.
MCP per-tool RBAC — currently enforced on JSON responses only. If your workflow needs strict per-tool denial, call with stream: false until streaming RBAC ships.
Response cache writes happen at stream close, not per-chunk. Response-cache hits replay as a normal SSE stream.

Try streaming in the dashboard playground

3-pane compare with streaming output side-by-side.

Streaming