Sieve-9B
Model card for Sieve-9B (v1), a mid-size decision model. It returns probabilities and does not generate text.
Hugging Face: sthanika-ai/Sieve-9B (Text Classification, Apache-2.0)
API model name: sieve-9b-v1. The alias sieve-9b points to Sieve-9B-v2 in the current model catalog, not to this build.
Sieve-9B answers typed questions about a text or JSON state. It returns a probability for each option. It never generates text.
Key facts
| Item | Value |
|---|---|
| Base model | Qwen/Qwen3.5-9B, revision c2022362 (post-trained model) |
| Backbone kept | Text part only: 32 layers plus embeddings, 7.94B parameters |
| Removed | LM head, multi-token-prediction layer and vision encoder (1.72B parameters) |
| Trained weights | LoRA plus a 2.1M-parameter pointer head trained from scratch |
| Question types | choice (up to 255 options), noul, score |
| Memory | About 20 GB in bf16 (17.6 GiB measured with a 4,000-token state) |
Each question runs as its own row from the state's cache, so questions do not affect each other.
Limits
- State: 8,192 tokens. Each question with its options: 8,192 tokens.
- Served API: up to 255 options per question.
- Accuracy drops on long states. In the held-out benchmark, states over 384 tokens were truncated to 384 (training limit). Accuracy was 0.697 on those, versus 0.889 on states of 384 tokens or fewer. Test long states before relying on this model for them.
- Accuracy also falls as options grow: 0.976 (2 options), 0.823 (3-10), 0.431 (51-255), 0.377 (256-512). The 77 questions with 256-512 options exceed the served 255-option limit.
Calibration
- One global temperature, T = 1.631, fitted by NLL on the decision-v7 calibration split (1,148 questions).
- T does not change answers.
temperature=1.0gives raw probabilities. - Note: this v1 card has one temperature. The per-category tables belong to Sieve-9B-v2.
Accuracy on decision suites (T = 1.631, development partitions only)
| Suite | Accuracy | ECE |
|---|---|---|
| decision-v7 development (in-domain, 1,264 questions) | 0.872 | 0.028 |
| transfer-v4 development (656) | 0.829 | 0.045 |
| transfer-v9 development, strict (936) | 0.753 | - |
| Held-out decision benchmark (6,224) | 0.857 | 0.055 |
| MMLU, 4-way (14,042) | 0.758 | - |
| SciQ (1,000) | 0.984 | - |
| Banking77 (3,079, in-distribution) | 0.857 | - |
| MASSIVE intent, 60 labels (2,974) | 0.765 | - |
| Typed-decisions test (2,000) | 0.704 | - |
The card says the suites are development partitions only. No test partition has been scored.
Tool selection (BFCL v3, decision form)
Not BFCL leaderboard numbers. Function selection 0.963 (1,909 questions). Irrelevance detection 0.720 (1,118 questions).
Benchmarks (self-reported)
| Measure | Score |
|---|---|
| Decision Index Full score (leaderboard 0.3.2) | 44.59 |
| Public index (0.3.2 suite) | 41.88 |
| Public index (own run, 0.2.1 suite) | 41.71 (raw 56.47, breadth 40.06), 150,317 requests, 44 benchmarks |
Latency (self-reported)
One A100 80GB, bf16, merged, CUDA graphs, one choice question on a ~75-token state, p50:
| Options | 2 | 5 | 10 | 25 | 50 | 100 | 255 |
|---|---|---|---|---|---|---|---|
| ms | 18.6 | 19.3 | 22.8 | 42.8 | 74.5 | 128.4 | 330.5 |
Excludes tokenisation and HTTP.
Training
- LoRA r=16, alpha=32, on attention, MLP and DeltaNet projections. Two adapters from one start on different data mixes, averaged 50/50.
- Learning rate 5e-5, OneCycle. 2 epochs. 8 records per step. bf16.
- Choice options permuted, with "none of the above" augmentation. Cross-entropy loss.
- Four training data files: decision-v7 training split (public classification, NLI and sentiment datasets, policy minimal pairs, rule-structure records), two date-policy files, and 1,920 rule-structure records from 240 structures not in the decision-v7 split.
- Training data is not published. Recipe and file hashes are in
training_config.json. - The card says LoRA rank 32 in one place and r=16 in another; we are checking this with the authors.
Intended use
Calibrated probabilities for typed questions about text or JSON, without generated text.
Licence
Apache-2.0 for the adapter, head and code. Backbone: Qwen3.5 (Apache-2.0 per the card). Public training datasets keep their own licences. The Qwen3.5 terms are pending legal review.
Data retention
The model keeps nothing between calls. Through the hosted API, Sieve does not store prompts, state or answers (see data retention).
Install and call
pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"Load with load_sieve() and call decide(). Server: sieve-serve --model sthanika-ai/Sieve-9B --graphs.
Through System One: "model": "sieve-9b-v1". See the quickstart.
Next steps
Last updated