Confidence

Confidence says how sure the model is about its answer. It runs from 0 to 1. Higher means more certain. Use a low value to send an answer for human review.

Which answers have confidenceCopy Anchor Link

How it is computedCopy Anchor Link

These are TypeSafe's published formulas. Sieve computes them on the server from the unrounded probabilities, then rounds.

Choice. With p_max the largest probability and n the number of options:

confidence = (p_max - 1/n) / (1 - 1/n)

A uniform spread gives 0. One option with probability 1 gives 1. With a single option the formula divides by zero, and the answer is 1.0.

Score. With m the most likely level, p_i the probability of level i and n levels:

MAD_uniform = (1/n) * sum over i of |i - (n-1)/2|
confidence  = max(0, 1 - sum over i of (p_i * |i - m|) / MAD_uniform)

MAD_uniform is the mean absolute distance from the middle level when probability is spread evenly. A score answer is confident when the probability sits close to one level, not only when one level is high.

Worked examplesCopy Anchor Link

• Choice, 4 options, top probability 0.95: (0.95 - 0.25) / (1 - 0.25) = 0.93.
• Score, 3 levels, probabilities 0.1 / 0.7 / 0.2: m = 1, the spread is 0.1*1 + 0.2*1 = 0.3, MAD_uniform = (1 + 0 + 1) / 3 = 0.667, so confidence = 1 - 0.3 / 0.667 = 0.55.

RoundingCopy Anchor Link

Every probability, confidence and score in a System One answer is rounded to 2 decimal places. Rounded probabilities sum to exactly 1.00.

Using confidenceCopy Anchor Link

• Set a threshold for each question. Send answers below it to review.
• Do not treat confidence as accuracy. It measures how spread out the probabilities are. Check calibration on your own data before you rely on a threshold. Assurance reports calibration over time.
• Probabilities are calibrated with a temperature per model (see the model cards). The temperature changes the probabilities, not the chosen option.
• sieve-auto uses confidence to decide when to ask the larger model. See Auto cascade.

Next stepsCopy Anchor Link