Catch unusual AI behaviour before it becomes a problem.
Two-layer detection: Z-score/Welford sliding windows for fast signals, Isolation Forest per-customer models for subtle drift.
Detection layers
Statistical + ML anomaly detection
Layer 1 — Z-score / Welford
Online algorithm, no history required. 4 rules run on every request:
- →token_spike — tokens >> rolling average
- →request_burst — RPM spike
- →guardrail_storm — block rate spike
- →cost_outlier — cost >> session average
Layer 2 — Isolation Forest
Per-customer models, trained nightly from a 14-day analytics lookback (200-sample minimum). 8-dimensional feature vector:
Fail-soft design
Exceptions never block the completion pipeline. Anomaly scoring is fire-and-forget — a model load failure returns a degraded score, not a 500.
Inline scoring
ML score computed with a 5-minute process-local model cache. Models stored in the in-memory cache with 48h TTL, refreshed nightly by training job.
Ops alerts
Email alerts via Resend for anomalies, batch failures, and SLA breaches. Configurable threshold and recipient.
Dashboard integration
Acknowledge anomalies, link to trace for investigation. Severity cards in dashboard with open/high counts.