How to Cut LLM Costs by 50% with Batch Routing

LLM costs compound quickly in production. A workload that costs $50/month in development becomes $5,000/month when you scale it by 100x. The good news is that not all traffic needs a real-time response — and for the traffic that can wait, batch routing is the most reliable way to cut costs in half.

What is batch routing?

Batch APIs from OpenAI, Anthropic, and Google accept jobs and return results asynchronously, typically within 15 minutes. In exchange for the delay, they charge 50% of real-time prices. The same model, the same output quality — at half the cost.

The catch: you have to send requests in the right format, poll for results, handle failures, implement SLA escalation if jobs take too long, and stitch results back to the original requests. Most teams never implement this because the operational overhead is significant. Anoman handles all of it transparently.

Which workloads are candidates for batching?

Good candidates for batch routing include:

• Document analysis and extraction
• Background agent tasks (not customer-facing)
• Data enrichment pipelines
• Evaluation and testing runs
• Overnight report generation
• Any workload where the user is not waiting for an immediate response

Not suitable for batch routing:

• Streaming responses
• Interactive chat
• Real-time agents with synchronous tool calls
• Enterprise tier (SLA requirements)

How batch routing works in Anoman

When you send a request with prefer_batch: true in metadata, or when your rate limit is temporarily exhausted (overflow mode), Anoman enqueues the job, sends it to the provider's batch API at the next submission window, polls for completion, and stores the result. The caller polls a job status endpoint.

If the job approaches its SLA deadline without completing, Anoman automatically escalates it to real-time at full cost — the caller never sees a missed SLA.

POST /v1/chat/completions
→ 202 Accepted (batch job enqueued)

{
  "id": "job_a8f3c2b1",
  "status": "queued",
  "sla_minutes": 15,
  "poll_url": "/anoman/v1/batch/job_a8f3c2b1"
}

GET /anoman/v1/batch/job_a8f3c2b1
→ 200 OK (complete)

{
  "choices": [...],
  "savings_usd": 0.0042,
  "routing_mode": "batch"
}

Savings in practice

Enabling batch routing

Three ways to activate batch routing:

1. Per API key (dashboard): Set default_routing: batch on an API key — all eligible traffic goes batch automatically.
2. Per request: Pass metadata: { prefer_batch: true } on individual requests.
3. Overflow mode: Let batch activate automatically when the real-time rate limit is hit. Zero configuration required.

from openai import OpenAI

client = OpenAI(
    api_key="anm-sk-...",
    base_url="https://api.anoman.io/v1",
)

# Opt this request into batch routing
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize this document: ..."}],
    extra_body={"metadata": {"prefer_batch": True}},
)

# If 202, poll for result
if hasattr(response, "id") and response.id.startswith("job_"):
    print(f"Batch job queued: {response.id}")
    print(f"Poll at: {response.poll_url}")

For high-volume non-interactive workloads, batch routing is the single highest-leverage cost optimization available. At scale, a 50% reduction in token cost compounds significantly — a $10,000/month workload becomes $5,000/month with zero change to output quality.