Two patterns: classifier-first picks a model up front, confidence-based escalates only when needed. Both cut spend by 80%+ vs naive routing to a single premium model.

Why bother — Most requests are easy

Production traffic distributions are wildly skewed. Across most chatbot + support traffic, ~60% of requests are factual lookup or simple formatting — a budget-tier model handles them perfectly. Routing those to claude-sonnet-4-6 is paying ~15× more than necessary.

The trick: a tiny model to decide which model to use. Or: try the cheap one first and escalate on low confidence.

Pattern 1 — Classifier first — Pick the model up front

A 7B model classifies the question difficulty. The classifier adds ~$0.000_05 per request — negligible compared to a sonnet call at $0.012.

Classifier-first routing

import os
import json
from openai import OpenAI
 
client = OpenAI(base_url="https://api.anoman.io/v1", api_key=os.environ["ANOMAN_API_KEY"])
 
# Tiny, cheap classifier. ~$0.000_05 per call.
CLASSIFIER_MODEL = "qwen-2.5-7b"
 
# Heavy model for hard requests.
HEAVY_MODEL = "claude-sonnet-4-6"
# Cheap model for easy requests.
LIGHT_MODEL = "gpt-4o-mini"
 
CLASSIFIER_SYSTEM = """
Classify the difficulty of the user's question as one of:
  easy      — Factual lookup, simple instruction, short formatting task
  medium    — Multi-step reasoning, document summarization, code generation
  hard      — Algorithm design, math proof, multi-turn planning, long context
 
Respond with ONLY the single word. No explanation.
""".strip()
 
def classify(question: str) -> str:
    response = client.chat.completions.create(
        model=CLASSIFIER_MODEL,
        messages=[
            {"role": "system", "content": CLASSIFIER_SYSTEM},
            {"role": "user",   "content": question},
        ],
        max_tokens=10,
        temperature=0,
    )
    return response.choices[0].message.content.strip().lower()
 
def ask(question: str) -> str:
    difficulty = classify(question)
    model = {
        "easy":   LIGHT_MODEL,
        "medium": LIGHT_MODEL,
        "hard":   HEAVY_MODEL,
    }.get(difficulty, HEAVY_MODEL)  # safe default: escalate
 
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": question}],
        # Track which path took for monitoring
        metadata={"routed_via": "classifier", "difficulty": difficulty},
    )
    return response.choices[0].message.content
 
print(ask("Capital of Indonesia?"))                            # easy → cheap
print(ask("Prove that the halting problem is undecidable."))   # hard → premium

Failure mode: classifier mis-routes a hard question to the cheap model. Tune the system prompt to err on the side of escalation, or use the hybrid pattern below.

Pattern 2 — Confidence escalation — Try cheap, escalate on bail

Instructs the cheap model to respond with ESCALATE when uncertain. The cheap model gets a chance to handle every request; we only pay premium when it explicitly bails. ~5% escalation rate is typical for well-prompted setups.

Confidence-based escalation

# Try the cheap model first. Only escalate when the cheap model says
# "I'm not sure" or doesn't follow the output format.
 
CHEAP = "gpt-4o-mini"
HEAVY = "claude-sonnet-4-6"
 
ASK_SYSTEM = """
Answer the user's question. If you're not confident, respond with
exactly: ESCALATE
 
Otherwise, give the answer.
""".strip()
 
def ask_with_escalation(question: str) -> tuple[str, str]:
    """Returns (answer, model_used)."""
    response = client.chat.completions.create(
        model=CHEAP,
        messages=[
            {"role": "system", "content": ASK_SYSTEM},
            {"role": "user",   "content": question},
        ],
        max_tokens=300,
        temperature=0,
    )
    answer = response.choices[0].message.content.strip()
 
    if answer.startswith("ESCALATE") or answer == "":
        # Cheap model bailed. Escalate.
        response = client.chat.completions.create(
            model=HEAVY,
            messages=[{"role": "user", "content": question}],
        )
        return response.choices[0].message.content, HEAVY
 
    return answer, CHEAP

Pattern 3 — Hybrid — Classifier + escalation

Use the classifier as a fast gate to route obvious hard requests directly to premium (no double-call overhead). For everything else, the cheap-with-escalation path applies. Best of both.

Hybrid — classify then escalate

# Combine both: the classifier picks a candidate model, the candidate
# can still escalate on low confidence. Catches mis-classifications.
 
def smart_ask(question: str) -> dict:
    difficulty = classify(question)
 
    if difficulty == "hard":
        # Skip cheap layer entirely.
        return {"answer": ask_heavy(question), "model": HEAVY, "path": "direct"}
 
    answer, model_used = ask_with_escalation(question)
    return {
        "answer": answer,
        "model": model_used,
        "path": "escalated" if model_used == HEAVY else "cheap",
    }

Economics — Back-of-envelope

Economics — back-of-envelope per 10K requests

Assume 10,000 requests/day:
  - 60% easy (factual)
  - 30% medium (summarization, simple code)
  - 10% hard (multi-step reasoning)

Naive routing — everything to claude-sonnet-4-6:
  - Avg cost: ~$0.012 per request (~500 in + ~300 out tokens)
  - Daily:    $120
  - Monthly:  ~$3,600

Classifier-first routing:
  - Classifier:  10K × $0.00005 = $0.50
  - Easy → cheap (6K × $0.0005)  = $3
  - Med  → cheap (3K × $0.0008)  = $2.40
  - Hard → heavy (1K × $0.012)   = $12
  - Daily:    $17.90
  - Monthly:  ~$540
  - Savings:  85% vs naive

Confidence escalation — assume 5% escalation rate from cheap:
  - Cheap call:   10K × $0.0008 = $8
  - Heavy retry:  500 × $0.012  = $6
  - Daily:    $14
  - Monthly:  ~$420
  - Savings:  88% vs naive

Production tips

Track metadata — pass metadata: {difficulty, path} on every call so the dashboard Usage page shows the routing distribution. Helps you tune the classifier prompt.
Monitor escalation rate — if it's >15%, your classifier is mis-routing too often. Tune the prompt or move to the hybrid pattern.
Latency-conscious paths — the classifier adds ~200 ms. For latency-sensitive UX, prefer confidence escalation (single call cheap path, escalate only when needed).
Cache the classifier — same question gets same classification. Run the classifier at temperature: 0 with the response cache on for its API key (opt-in, exact match, rolling out) so exact repeats can be served from Anoman's cache; hits are billed at the normal rate.
A/B test the threshold — split 5% of traffic to "always use premium" as a quality control. Compare end-user thumbs-up rate vs the routed cohort.

Track your routing distribution in the dashboard.

By-model breakdown with cost per route.

Route easy → cheap, hard → premium.