Recipe · RAG with Caching

Retrieval-augmented chat with 90% cache savings.

Embed your docs once. Put the stable system prompt in the right place so provider cache fires on every query. Pay 10% on the cached prefix, full price only on the small user message.

Why this pattern

Cache the prefix, not the question

A naive RAG chain puts everything in the user message — instructions, retrieved chunks, and the question. The system prompt is empty. This works but the entire prompt is uncached on every call.

A smarter chain puts the stable instructions in the system prompt (rare changes) and the variable retrieval + question in the user message. Anoman auto-injects cache markers when the system prompt is long enough; subsequent calls hit the provider prompt cache and pay 10% on the cached prefix.

See /docs/concepts/caching for the cache mechanics in detail.

1. Index

Embed once, batch-style

Embeddings are cheap (~$0.02 per million tokens for text-embedding-3-small). Batch 100 chunks per call to amortize HTTP overhead.

# index.py — embed every doc once, store in any vector DB
import os
import json
from pathlib import Path
from openai import OpenAI

client = OpenAI(
    base_url="https://api.anoman.io/v1",
    api_key=os.environ["ANOMAN_API_KEY"],
)

EMBED_MODEL = "text-embedding-3-small"   # 1536 dims, $0.02/Mtok
DOCS = Path("./knowledge_base")

# Cheap chunk function — use a real chunker (e.g. langchain's
# RecursiveCharacterTextSplitter) for production.
def chunk(text: str, size: int = 1200, overlap: int = 200):
    chunks = []
    i = 0
    while i < len(text):
        chunks.append(text[i:i+size])
        i += size - overlap
    return chunks

# Batch embed for throughput
records = []
texts: list[str] = []
metadata: list[dict] = []
for doc_path in DOCS.glob("*.md"):
    text = doc_path.read_text()
    for j, ch in enumerate(chunk(text)):
        texts.append(ch)
        metadata.append({"source": doc_path.name, "chunk": j})
        if len(texts) >= 100:    # batch of 100 per embedding call
            response = client.embeddings.create(model=EMBED_MODEL, input=texts)
            for emb, meta in zip(response.data, metadata):
                records.append({**meta, "embedding": emb.embedding})
            texts, metadata = [], []

# Flush remainder
if texts:
    response = client.embeddings.create(model=EMBED_MODEL, input=texts)
    for emb, meta in zip(response.data, metadata):
        records.append({**meta, "embedding": emb.embedding})

# Persist — replace with your vector DB (Qdrant, pgvector, etc.)
Path("./index.jsonl").write_text(
    "\n".join(json.dumps(r) for r in records)
)
print(f"Indexed {len(records)} chunks")

2. Query with cacheable system prompt

The pattern

# query.py — retrieval-augmented chat call
import json
from pathlib import Path
import numpy as np
from openai import OpenAI

client = OpenAI(
    base_url="https://api.anoman.io/v1",
    api_key=os.environ["ANOMAN_API_KEY"],
)

# Load the index. Replace with your vector DB query.
INDEX = [json.loads(line) for line in Path("./index.jsonl").read_text().splitlines()]
EMBEDDINGS = np.array([r["embedding"] for r in INDEX])

def top_k(query: str, k: int = 5) -> list[dict]:
    q = client.embeddings.create(
        model="text-embedding-3-small",
        input=query,
    ).data[0].embedding
    sims = EMBEDDINGS @ np.array(q)        # cosine if normalized
    idx = sims.argsort()[-k:][::-1]
    return [INDEX[i] for i in idx]

Cache anti-pattern: if you put retrieved chunks into the system prompt, every query has a different system prompt and the cache never fires. Keep system stable; put context in the user message.

3. Verify the cache fired

Read cached_tokens

# After the FIRST call, the system prompt is cached.
# Subsequent calls should show cached_tokens > 0 in usage.

response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[
        {"role": "system", "content": STABLE_SYSTEM},
        {"role": "user", "content": "Different question..."},
    ],
)

cached = response.usage.prompt_tokens_details.cached_tokens
print(f"Cached prefix tokens: {cached}")     # > 0 = cache fired

# Cost comparison via _anoman extension:
print(f"Cost this call: ${response._anoman['cost_usd']}")

For Anthropic models the system prompt must be ≥ 2,048 tokens to be cacheable. For OpenAI it’s automatic for any length. For Google it’s 32,768+ tokens (via explicit CachedContent API). See the per-provider table at /docs/concepts/caching.

4. Stream for UX

Streaming + cache stack

Set stream: true. Provider cache still applies — you save 90% on the prefix and stream the small variable part token-by-token.

# For UX, run retrieval + streaming in parallel. The first chunk
# from the model arrives in ~1s; retrieval takes ~200ms.

stream = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[
        {"role": "system", "content": STABLE_SYSTEM},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
    ],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    yield delta   # if running in a FastAPI streaming response

Production tips

Cache lifetime is ~5 minutes. Idle traffic doesn’t benefit. For sparse-traffic apps, send a synthetic warming request every 4 minutes to keep the cache hot.
System prompt versioning — any change invalidates the cache. Use a content-hash version comment at the top of the system prompt so you can detect accidental drift in your code.
Don’t cache user-PII in the system prompt — Anoman’s PII guardrails apply, but you shouldn’t persist sensitive customer data in a multi-tenant cache layer.
Semantic cache on top — for FAQ-style apps where the same question repeats, layer Anoman’s semantic cache (x-anoman-cache: semantic) on top. Identical requests then return without an upstream call at all.
Reranking after retrieval — top-k by cosine is rough. For higher quality, pass the candidates through a cheap rerank model (qwen-2.5-7b) before stuffing into the user message.

Inspect cache savings live.

Usage page shows cache_savings_usd as a separate column.

Open Usage More recipes