Send work that isn't needed instantly through native provider batch APIs — about 50% cheaper, with an SLA the gateway guarantees.

Non-interactive work, ~50% cheaper

Batch routing runs your requests through the provider's native batch API, which is about 50% cheaper than real-time. The trade-off is latency — results arrive within an SLA window (minutes), not instantly. Use it for anything a human isn't waiting on.

Good for batch

Document analysis & extraction
Background agent tasks
Data enrichment pipelines
Overnight reports & eval runs

Never batch

Interactive chat (a human is waiting)
Streaming requests (stream: true)
Enterprise & PAYG traffic — both are real-time only (SLA)

Ask for batch, get a 202

Send a normal /v1/chat/completions request with metadata.prefer_batch=true (or set the key's default_routing=batch so all traffic batches). On rate-limit overflow, traffic auto-batches too. The gateway returns a 202 Accepted with a job_id and a poll_url instead of a completion.

cURL

# Send a normal completion, but ask for batch routing via metadata.
curl https://api.anoman.io/v1/chat/completions \
  -H "Authorization: Bearer anm-sk-..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Extract line items from this invoice ..."}],
    "metadata": {"prefer_batch": true}
  }'
 
# → 202 Accepted
# {
#   "id": "job_a8f3c2b1d9",
#   "object": "batch_job",
#   "status": "queued",
#   "sla_minutes": 5,
#   "poll_url": "/anoman/v1/batch/job_a8f3c2b1d9",
#   "reason": "customer_preference"
# }

Python

from openai import OpenAI
 
client = OpenAI(base_url="https://api.anoman.io/v1", api_key="anm-sk-...")
 
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Extract line items ..."}],
    extra_body={"metadata": {"prefer_batch": True}},
)
# The raw HTTP response is 202 with a job_id + poll_url.
# Or set the key's default_routing = "batch" so every request batches.

202 while processing, 200 when done

Poll GET /anoman/v1/batch/{job_id} — it returns 202 with sla_remaining_minutes while the job runs, then 200 with the chat completion plus a savings_usd field showing what you saved versus real-time. You can also list all jobs or cancel one that's still queued.

Poll / retrieve

# Poll until the job completes. 202 while processing, 200 when done.
curl https://api.anoman.io/anoman/v1/batch/job_a8f3c2b1d9 \
  -H "Authorization: Bearer anm-sk-..."
 
# → 202 (still processing)
# { "id": "job_a8f3c2b1d9", "status": "processing",
#   "sla_remaining_minutes": 8, "poll_again_in_seconds": 30 }
 
# → 200 (complete)
# {
#   "id": "chatcmpl-job_a8f3c2b1d9",
#   "object": "chat.completion",
#   "choices": [{"message": {"role": "assistant", "content": "..."}}],
#   "usage": {"prompt_tokens": 1024, "completion_tokens": 256},
#   "batch_job_id": "job_a8f3c2b1d9",
#   "routing_mode": "batch",
#   "savings_usd": 0.0042
# }

List / cancel

# List all batch jobs for the key
curl https://api.anoman.io/anoman/v1/batch \
  -H "Authorization: Bearer anm-sk-..."
 
# Cancel a job that is still queued (not yet submitted)
curl -X DELETE https://api.anoman.io/anoman/v1/batch/job_a8f3c2b1d9 \
  -H "Authorization: Bearer anm-sk-..."

Full field-by-field reference: the batch jobs endpoint.

SLA by tier, with escalation

Each tier gets a maximum wait for batch results. If a job nears its SLA window without finishing, the worker auto-promotes it to the real-time API at full cost — so the customer never sees a breach.

Tier	Batch SLA
Starter	30 min
Pro	5 min
PAYG	N/A — real-time only
Enterprise	N/A — real-time only

SLA escalation: A job that consumes most of its SLA window without completing is automatically resubmitted via the real-time API at full price. You get your result on time; you never see a missed deadline.

Guardrails run before batching

Every request passes the full guardrail pipeline before the routing decision. A request that fails a guardrail returns 403 immediately and is never enqueued — batched or not, nothing bypasses your policies.

Related: Caching (stacks with batch for further savings) and Usage & costs (batch economics live on the usage API).

Watch the batch queue live

Batch queue status and completions stream into the Live Feed in real time.

Batch routing