anoman
Sign InGet API Key
Documentation home

Faithfulness

Does the model's answer stay grounded in the context you supplied? An opt-in, warn-only observability signal you review after the fact.

A response-groundedness signal

Opt-in preview: Faithfulness is currently a preview feature you opt into. It does not score every response automatically — only requests you've configured for it and that carry grounding context are evaluated.

Faithfulness answers one question: does the model's answer stay grounded in the context you supplied (retrieved docs, tool results)? It's a CPU-only signal computed asynchronously.

It is async and warn-only — it never blocks or alters a response. It's an observability signal you review after the fact, not a guardrail that gates traffic.

Turn it on

  1. Enable content capture on the policy group (the per-group opt-in that lets Anoman store the request/response content needed to score groundedness) — see Policy groups.
  2. Send requests with an explicit anoman_context — the grounding context to check against. No context means the response is marked N/A.
  3. Scored responses appear at /dashboard/faithfulness (until then it shows a "no evals yet" state).

Privacy: Captured content is KEK-encrypted at rest, kept in-region, and private to your tenant.

cURL

# After enabling content capture on the policy group, send requests with an
# explicit anoman_context — the grounding to check the answer against.
curl https://api.anoman.io/v1/chat/completions \
  -H "Authorization: Bearer anm-sk-..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      {"role": "user", "content": "What is our refund window?"}
    ],
    "metadata": {
      "anoman_context": "Refunds are accepted within 14 days of purchase..."
    }
  }'
 
# No anoman_context → the response is marked N/A (nothing to check against).

Python

from openai import OpenAI
 
client = OpenAI(base_url="https://api.anoman.io/v1", api_key="anm-sk-...")
 
client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "What is our refund window?"}],
    extra_body={"metadata": {"anoman_context": retrieved_docs}},
)
# Scoring is async & warn-only — the completion returns immediately, unchanged.

A coarse band, never a raw score

The dashboard shows a coarse band — not a raw numeric score (the signal is calibration-gated; the raw float is never surfaced). Unsupported spans are shown redacted. A request that carries no anoman_context is scored na (not applicable).

  • ok — well-supported
  • uncertain — partially supported
  • low — unsupported

cURL

# Your tenant's faithfulness evals (customer-scoped; tenant is server-derived).
curl https://api.anoman.io/anoman/v1/faithfulness/activity \
  -H "Authorization: Bearer anm-sk-..."
 
# {
#   "data": {
#     "window_days": 30,
#     "degraded": false,
#     "totals": { },
#     "by_band": [ ],
#     "by_model": [ ],
#     "daily": [ ],
#     "recent": [
#       {
#         "timestamp": "2026-08-13T10:00:00Z",
#         "model": "claude-sonnet-4-20250514",
#         "band": "ok",                  # ok | uncertain | low | na
#         "has_context": true,
#         "unsupported_spans": []
#       }
#     ]
#   },
#   "error": null
# }
# A coarse band only — never a raw numeric score. Unsupported spans are redacted.

Related: Traces & requests (where each scored request lives) and Policy groups (the content-capture toggle).

Try faithfulness

Enable content capture on a policy group, then review groundedness on the Faithfulness page.

On this page