Sieve-2B-v3
Model card for Sieve-2B-v3, a small decision model with per-category calibration. It returns probabilities and does not generate text.
Hugging Face: sthanika-ai/Sieve-2B-v3 (Text Classification, Apache-2.0)
API model name: sieve-2b-v3. The alias sieve-2b points to this build in the current model catalog.
Sieve-2B-v3 answers typed questions about a text or JSON state. It returns a probability for each option. It never generates text.
Key facts
| Item | Value |
|---|---|
| Base model | Qwen/Qwen3.5-2B, revision 15852e8c (text part only) |
| Method | Pointer head that scores the end of each option against the decision point. Calibrated softmax. |
| Trained weights | Full fine-tune of the 24 decoder layers: 1,373,263,680 parameters. Token embeddings and final norm frozen. Pointer head: 256-dimensional projections in fp32, trained from scratch. |
| Packaging | PEFT adapter using modules_to_save for the trained layers, plus a zero rank-1 LoRA on the embeddings. No rank-r LoRA on the decoder layers. |
| Question types | choice, noul, score |
| Memory (bf16) | 4.3 GB on the CUDA-graph server, 10.7 GB on the eager server |
Limits
- 16,384 tokens for the state and 16,384 for each question with its options. Longer requests are refused, not truncated.
- The decision engine limits are 65,536 tokens (state) and 32,768 tokens (question).
- Up to 255 options for
choicein the API (see limits and errors). scorequestions were not in training. They use the same head and the global temperature. Treatscoreanswers from this model as untested.
Calibration
- Global T = 0.968.
- 25 per-category temperatures from 0.869 to 1.257.
- Category = question type x option-count bin (2, 3-4, 5-9, 10-25, 26+) x state kind (empty, text, JSON). Lookup falls back from the most specific match to the type, then to the global T.
- Temperatures do not change answers.
temperature=1.0gives raw probabilities. - Results on 143,822 calibration questions (per-category): accuracy 0.8781, NLL 0.3009, Brier 0.1606, ECE 0.0028. Raw (T = 1): ECE 0.0053.
Benchmarks (self-reported, not on the leaderboard)
- Decision Index 0.3 public suite: 53.51 (raw index 65.39, breadth 51.09).
- Area scores: Knowledge and Reasoning 30.9; Language Understanding 49.3; Retrieval and Classification 66.5; Tools and Automation 80.5; Arts and Human Taste 47.6.
- Sample scores: HellaSwag 0.873; GSM8K 0.390; MMLU-Pro 0.298; GPQA Diamond 0.270.
Latency (self-reported, A100 80GB PCIe, eager, one at a time)
Median 59 ms, mean 143 ms, p80 115 ms, p95 329 ms. These are upper bounds because the runs shared GPUs.
Training
- 428,018 training records from public datasets and generated decision records. Training files are not published. File hashes are in
training_config.json. - 1 epoch, 3,344 steps, 128 records per step, 10.6 hours on two A100 80GB PCIe GPUs.
- AdamW. Learning rate 1e-5 (backbone), 5e-5 (head). Head trained alone for the first 100 steps.
- Selected checkpoint: step 3,300 (highest accuracy on a 2,004-question calibration sample, NLL as tie-breaker).
- Augmentation: permuted options, "none of the above" variants, irrelevant distractors.
- The card says the split is 60/10/30 with 71,018 calibration records, which does not match 428,018 total. Treat these figures as unverified.
Intended use
Calibrated probability decisions over typed questions about text or JSON, without text generation.
Licence
Apache-2.0 for the model and code. Training data licences: see the dataset cards (pending legal review).
Data retention
The model keeps nothing between calls. Through the hosted API, Sieve does not store prompts, state or answers (see data retention).
Install and call
pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"
pip install --no-build-isolation causal-conv1d==1.7.0 # recommended on A100 and H100Needs sieve 0.3.0 or later. The backbone downloads on first use.
Through System One: "model": "sieve-2b-v3" (or "sieve-2b"). See the quickstart.
Next steps
Last updated