Concept

Batch routing

Send work that isn't needed instantly through native provider batch APIs — about 50% cheaper, with an SLA the gateway guarantees.

When to batch

Non-interactive work, ~50% cheaper

Batch routing runs your requests through the provider's native batch API, which is about 50% cheaper than real-time. The trade-off is latency — results arrive within an SLA window (minutes), not instantly. Use it for anything a human isn't waiting on.

Good for batch

Document analysis & extraction
Background agent tasks
Data enrichment pipelines
Overnight reports & eval runs

Never batch

Interactive chat (a human is waiting)
Streaming requests (stream: true)
Enterprise & PAYG traffic — both are real-time only (SLA)

Enqueue

Ask for batch, get a 202

Send a normal /v1/chat/completions request with metadata.prefer_batch=true (or set the key's default_routing=batch so all traffic batches). On rate-limit overflow, traffic auto-batches too. The gateway returns a 202 Accepted with a job_id and a poll_url instead of a completion.

# Send a normal completion, but ask for batch routing via metadata.
curl https://api.anoman.io/v1/chat/completions \
  -H "Authorization: Bearer anm-sk-..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Extract line items from this invoice ..."}],
    "metadata": {"prefer_batch": true}
  }'

# → 202 Accepted
# {
#   "id": "job_a8f3c2b1d9",
#   "object": "batch_job",
#   "status": "queued",
#   "sla_minutes": 5,
#   "poll_url": "/anoman/v1/batch/job_a8f3c2b1d9",
#   "reason": "customer_preference"
# }

Poll & retrieve

202 while processing, 200 when done

Poll GET /anoman/v1/batch/{job_id} — it returns 202 with sla_remaining_minutes while the job runs, then 200 with the chat completion plus a savings_usd field showing what you saved versus real-time. You can also list all jobs or cancel one that's still queued.

# Poll until the job completes. 202 while processing, 200 when done.
curl https://api.anoman.io/anoman/v1/batch/job_a8f3c2b1d9 \
  -H "Authorization: Bearer anm-sk-..."

# → 202 (still processing)
# { "id": "job_a8f3c2b1d9", "status": "processing",
#   "sla_remaining_minutes": 8, "poll_again_in_seconds": 30 }

# → 200 (complete)
# {
#   "id": "chatcmpl-job_a8f3c2b1d9",
#   "object": "chat.completion",
#   "choices": [{"message": {"role": "assistant", "content": "..."}}],
#   "usage": {"prompt_tokens": 1024, "completion_tokens": 256},
#   "batch_job_id": "job_a8f3c2b1d9",
#   "routing_mode": "batch",
#   "savings_usd": 0.0042
# }

Full field-by-field reference: the batch jobs endpoint.

SLA

SLA by tier, with escalation

Each tier gets a maximum wait for batch results. If a job nears its SLA window without finishing, the worker auto-promotes it to the real-time API at full cost — so the customer never sees a breach.

Tier	Batch SLA
Starter	30 min
Pro	5 min
PAYG	N/A — real-time only
Enterprise	N/A — real-time only

SLA escalation: A job that consumes most of its SLA window without completing is automatically resubmitted via the real-time API at full price. You get your result on time; you never see a missed deadline.

Safety

Guardrails run before batching

Every request passes the full guardrail pipeline before the routing decision. A request that fails a guardrail returns 403 immediately and is never enqueued — batched or not, nothing bypasses your policies.

Related: Caching (stacks with batch for further savings) and Usage & costs (batch economics live on the usage API).

Watch the batch queue live

Batch queue status and completions stream into the Live Feed in real time.

Open Live Feed Back to docs