Assurance

Assurance answers three questions about your decisions: what was decided, is the model still well calibrated on your data, and has anyone changed the record.

The audit logCopy Anchor Link

For every question answered, Assurance writes one metadata-only record: request, organisation and key IDs, the model asked and the model that served it, the adapter, region, question type, number of options, which model answered (for Auto), status, whether it escalated, the calibrated probabilities, confidence, the chosen option, latency, and a UTC timestamp.

It never contains your state, the question text, option text or answer text. A failed request gets one request-level row.

Records form a hash chain per organisation: each record's hash covers the previous hash, so editing, deleting, reordering or inserting a record breaks the chain. You can verify the chain at any time. To detect the removal of the most recent records, store the head hash the report prints outside the system.

The log is written as append-only files when assurance is turned on for your organisation or Vault installation; without it no records are written and the /v1/assurance/* routes are not available. In Vault those files stay in your cluster.

Outcomes and calibrationCopy Anchor Link

Send the correct answers back once you know them:

POST /v1/assurance/outcomes
Content-Type: application/json

{ "outcomes": [ { "request_id": "...", "question_name": "tone", "true_index": 1 } ] }

true_index is the index of the correct option. Send 1 to 1,000 outcomes per call. Outcomes are accepted only for requests in your own organisation's log (an unknown request_id and question_name pair gets 404), and the latest label wins. These routes use your normal API key.

From outcomes Assurance computes, per model, adapter and question type:

• accuracy,
• expected calibration error (ECE, 10 equal-width bins on the top label),
• Brier score,
• reliability bins.

Drift compares the current window to a baseline. It raises an alert when ECE rises by more than 0.05 or accuracy falls by more than 0.05 (configurable, with at least 30 samples per window).

ReportsCopy Anchor Link

GET /v1/assurance/report?from=2026-11-01&to=2026-12-01&format=json
GET /v1/assurance/verify

format is json, md or html. Reports include latency percentiles, error rate and escalation counts. In Vault, the same reports come from the assurance command line tool, and metrics are exposed in Prometheus format at /metrics (request latency, requests by status, escalations and questions by type).

In the consoleCopy Anchor Link

The console's Assurance page shows, for the last 7, 30 or 90 days and for your whole organisation:

• Decisions logged (including failed requests) and how many questions have a known outcome (one you sent back).
• Confidence gap: the expected calibration error in percentage points. 0 is perfect; 5 pts means the model's confidence was off by about 5 points on average. It needs outcomes, and below 30 outcomes the page marks it as rough.
• Log integrity: whether your whole hash chain verifies.
• Confidence vs. reality: average confidence against the share actually right, per confidence range.
• Speed and errors per model: median and 99th-percentile model processing time, error rate and escalations (Auto).
• Drift: an alert when the confidence gap rises or accuracy falls by more than 5 points against the previous period of the same length. The page says when no model has enough outcomes (30 in both periods) to be checked, so a missing alert is never read as "all clear".

If the audit log is not turned on for your deployment, the page says so and shows no numbers.

Not built yet: alert delivery by webhook or email (drift alerts appear in the report and on the console page only), downloading the log or a report from the console, and per-model alert thresholds. Monthly calibration checks are planned as part of the Vault service.

Next stepsCopy Anchor Link