Concepts · Caching

Two caching layers that stack.

Provider-side prompt cache discounts repeated prefixes by 90%. Anoman's semantic cache serves identical requests for free. Together they cut spend on stable workloads.

Two layers

Different problems, different solutions

Layer A — Semantic cache

Anoman-side, Redis-backed

Caches the complete response for identical requests. On hit: return immediately, zero upstream call, zero weighted tokens deducted.

Best for: FAQ bots, dev/test loops, repeated benchmark queries.

Layer B — Provider prompt cache

Upstream-side, transparent

Caches the large system prompt prefix server-side at Anthropic / OpenAI / Google. On hit: still calls upstream but charges 10% on the cached tokens.

Best for: agents with multi-thousand-token system prompts.

Layer A — Semantic cache

How it works

Key — hash(customer_id + model + messages). Entries are partitioned per customer; no cross-customer leakage is possible.
TTL — 5 minutes default on Pro / PAYG / Enterprise. Configurable per key (1-300s) in the dashboard.
What counts as “identical” — exact match on the full messages array plus model. Any whitespace difference makes it a different request.
Opt-in only — semantic cache is off by default. Enable per-request via header or per-key in dashboard settings.
Auto-bypass — requests with stream: true or temperature > 0.3 skip cache (the caller is asking for variation).

{
  "choices": [{
    "message": {"role": "assistant", "content": "The capital of Indonesia is Jakarta."}
  }],
  "_anoman": {
    "cache": { "hit": true, "type": "semantic", "age_seconds": 14 },
    "weighted_tokens": 0,        // ← no quota deduction
    "cost_usd": "0.000000"       // ← free
  }
}

On a hit, the response carries the original _anoman.cache.age_seconds field so you know how old the cached content is.

Layer B — Provider prompt cache

Transparently applied to long system prompts

When your request has a system prompt > the provider’s minimum, Anoman automatically injects the cache control markers so the upstream caches it server-side. Subsequent requests with the same prefix get the cache discount.

Provider	Minimum prompt	Discount	Cache lifetime
Anthropic	2,048 tokens (~8K chars)	90% off cached input	5 min
OpenAI	Any length (automatic)	90% off cached prefix	~5–10 min
Google Gemini	32,768 tokens (~130K chars)	97.5% off cached input	Explicit CachedContent API

// You sent a 4000-token system prompt that matched a recent cached prefix.
// Provider charges 10% on the cached tokens; output is full price.
{
  "choices": [{
    "message": {"role": "assistant", "content": "..."}
  }],
  "usage": {
    "prompt_tokens": 4127,
    "completion_tokens": 380,
    "prompt_tokens_details": { "cached_tokens": 4000 }  // ← 4000 were cached
  },
  "_anoman": {
    "cache": { "hit": true, "type": "provider" },
    "weighted_tokens": 1600,   // ← discounted from ~14,300 without cache
    "cost_usd": "0.0048"
  }
}

Provider cache is on automatically — no header opt-in. The discount is applied to your billing without any client-side change. To verify it’s firing, check usage.prompt_tokens_details.cached_tokens on the response.

Controls

Headers that change cache behavior

Header	Values	Effect
x-anoman-cache	semantic / off	Force semantic cache on or off for this request.
x-anoman-no-cache	1 / true	Skip semantic cache check + write (provider cache still applies).
x-anoman-no-provider-cache	1 / true	Skip injecting provider cache markers (rare; useful for A/B testing cost).

# Semantic cache is opt-in. Two ways to enable:

# 1. Per-request header
curl https://api.anoman.io/v1/chat/completions \
  -H "Authorization: Bearer anm-sk-..." \
  -H "x-anoman-cache: semantic" \
  -d '...'

# 2. Set as default on the API key (dashboard → Keys → Settings)
#    Then it applies to every request without the header.

When each fires

Workload	Semantic	Provider	Why
FAQ chatbot	✓	—	Same questions repeat. Cache full answers.
Agentic coding	—	✓	Multi-turn with big system prompt. Cache prefix.
RAG over docs	—	✓	Same retrieved chunks across turns. Provider cache.
Eval / benchmark	✓	—	Re-running the same prompts repeatedly.
Dev/test loop	✓	—	Iterating on prompts. Don’t pay for repeats.
Creative writing	No	—	User wants variation — semantic cache would fight you.

See your savings

The dashboard Usage page shows Cache savings and Batch savings as separate columns. The semantic-cache savings number is “what you would have spent without cache hits”. Provider-cache savings are baked into the per-request cost_usd already.

Inspect cache hits in real time.

Live Feed marks every cache hit with a CACHED pill.

Open Live Feed All concepts