Sieve-2B
Model card for Sieve-2B (v1), a small decision model. It returns probabilities for typed questions and does not generate text.
Hugging Face: sthanika-ai/Sieve-2B (Text Classification, Apache-2.0)
API model name: none. This v1 build is not in the current model catalog; the alias sieve-2b points to Sieve-2B-v3.
Sieve-2B answers typed questions about a text or JSON state. It returns a probability for each option. It never generates text.
Key facts
| Item | Value |
|---|---|
| Base model | Qwen/Qwen3.5-2B-Base, blocks 1-16 of 24 (read_layer=16). Base revision: not stated on the card. |
| Method | Option fork. The state is encoded once. Each option is scored without generating text. |
| LoRA | Rank 8, alpha 16, on attention and MLP projections of blocks 1-16. 3,637,248 parameters. |
| Scorer head | LayerNorm, then 512, then 1. 1,053,697 parameters. |
| Trained parameters | 4.69M |
| Parameters at inference | 1.43B (full base text model 1.88B). Blocks 17-24 and the generation head are not loaded. |
| Question types | choice, noul (scored as a 2-option choice) |
| Memory | About 3 GB in bf16. Initial download about 4.3 GB. |
| Temperature | One shipped value, T = 1.7144, applied by choice(). Optional fit_schema(name, examples) refits a 2-parameter logistic with about 100 labelled examples. |
Limits
- Up to 32,768 tokens for the state, the question and the longest option. Longer requests are refused by the decision engine and the API.
- Training prompts were up to 2,048 tokens.
- The card sets no option-count cap. The held-out pool used up to 150 options. The API limit for
choiceis 255 (see limits and errors). - Answers do not change under option reordering (0.0% changes on the card's test).
Calibration
- Temperature changes probabilities, not the winning option.
- MMLU ECE falls from 0.135 (raw) to 0.040 with the built-in temperature. transfer-v4 (clean) falls from 0.114 to 0.074. SciQ, banking77 and MASSIVE get slightly worse.
Benchmarks (self-reported, single seed)
| Measure | Score |
|---|---|
| Decision Index Full score (leaderboard 0.3.2) | 20.53 |
| Public index (own run, 0.3 suite) | 22.16 |
| Public index (own run, 0.2.1 suite) | Not shown: the card's table has garbled headers, so the value cannot be labelled |
Selected results: MASSIVE (K=59) 71.5; MMLU 60.8; SciQ 97.5; transfer-v4 (clean, 328 questions) 73.2; held-out pool (K up to 150) 77.7; banking77 76.5.
Latency (self-reported)
Single process, bf16, idle A100, 400 sampled requests: median 85 ms, p95 223 ms.
Training
- From scratch, 2 epochs, learning rate 5e-5.
- 26,241 rows per epoch. About 18,200 from ten public datasets and 7,999 synthetic option-comparison questions.
- Overlap check: 33 of 150,317 scored suite rows (0.02%) share an input with training data.
- An earlier run at learning rate 2e-4 scored 13.8 points lower on MMLU (one paired comparison).
Intended use
Calibrated probabilities for typed decision questions about text or JSON, without generated text.
Limitations
- Single seed. Earlier runs of the same recipe family varied by about plus or minus 0.2 to 1.5 points.
- The learning-rate result rests on one paired comparison.
- MMLU and transfer-v4 are the weakest results.
Licence
- Adapter, head and code: Apache-2.0. Base Qwen3.5: Apache-2.0 per the card.
- Several public training datasets (ag_news, yelp_review_full, imdb, sst5, trec, amazon_reviews_multi) state no licence on their Hub cards. Pending legal review.
Data retention
The model keeps nothing between calls. Through the hosted API, Sieve does not store prompts, state or answers (see data retention).
Install and call
pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"
pip install --no-build-isolation causal-conv1d==1.7.0 # recommendedNeeds transformers>=5.17 and peft>=0.21.
This build is not served through the API. "model": "sieve-2b" calls Sieve-2B-v3. See the quickstart.
Next steps
Last updated