Route uncertain answers to review
Act on confident answers automatically and send the doubtful ones to a person, using confidence for choice and score questions and the probability for yes/no.
Every answer comes with numbers that say how sure the model is. A common pattern is to act automatically when the model is sure and send the rest to a person. This guide routes support tickets to a team that way.
You need: a working setup from the quickstart, and a few hundred labelled examples of your own to pick a threshold.
How the gate works
| Question type | Gate on | Sure when |
|---|---|---|
choice | confidence | Close to 1 |
score | confidence | Close to 1 |
noul (yes/no) | noul, the probability of yes | Close to 0 or 1. Near 0.5 means unsure. |
confidence measures how concentrated the probabilities are. It is not accuracy. See Confidence for the formulas.
Ask the question
from typesafe_sdk import Choice, TypeSafeClient
ticket = "My card was charged twice for the same invoice."
with TypeSafeClient(model="sieve-latest") as client:
response = client.system_one(
state=ticket,
questions={
"team": Choice(
instructions="Which team should handle this support ticket?",
criteria={
"billing": "Charges, refunds, invoices and payment methods",
"technical": "Errors, outages and integration problems",
"account": "Sign-in, members and organisation settings",
},
),
},
)
team = response.choices["team"]Gate on confidence
THRESHOLD = 0.6 # pick yours from labelled data (next step)
if team.confidence >= THRESHOLD:
assign(ticket, team.choice)
else:
send_to_review(ticket, suggestion=team.choice, probabilities=team.probabilities)assign and send_to_review stand for your own code. Passing the model's suggestion and probabilities to the reviewer saves them time.
For a yes/no question, gate on distance from 0.5 instead, for example act when noul >= 0.9 or noul <= 0.1 and review the rest.
Pick the threshold from your own data
- Run the question over labelled examples and keep the answer, the confidence and the right label.
- For a few candidate thresholds, work out the share of answers above it (automated) and the accuracy of those answers.
- Choose the lowest threshold whose automated accuracy you can accept.
- Re-check after you change the model, the instructions or the options. Confidence values are not comparable across models.
Answers are rounded to 2 decimal places, so compare with >= on a 2-decimal threshold.
Let Sieve escalate for you
If the goal is accuracy rather than human review, use sieve-auto. It answers every question with Sieve-4B and asks Sieve-31B again only for the questions where 4B's confidence is below a threshold. The whole request is billed at the Auto price, whether or not a question escalates. You can still gate the final answer as above. See Auto cascade.
Next steps
Last updated