Sieve-27B
Model card for Sieve-27B (v1), a large decision model. It returns probabilities and does not generate text.
Hugging Face: sthanika-ai/Sieve-27B (Text Classification, Apache-2.0)
API model name: sieve-27b-v1. The alias sieve-27b points to Sieve-27B-v2 in the current model catalog, not to this build.
Sieve-27B answers typed questions about a text or JSON state. It returns a probability for each option. It never generates text.
Key facts
| Item | Value |
|---|---|
| Base model | Qwen/Qwen3.8-27B as written on the card, revision 1d4bf0f2. We are confirming this base name with the authors. Only the text part is used. |
| Method | LoRA adapter (rank 32, alpha 64, dropout 0.05) on 12 projection modules (attention, MLP, linear attention) plus a pointer head trained from scratch |
| Question types | choice (up to 255 options), noul, score (maximum level count not stated) |
| Memory | About 55 GB in bf16 (52,683 MiB after loading). One 80 GB GPU is enough. |
Limits
- State: 16,384 tokens. Each question with its options: 16,384 tokens.
- Longer requests are refused, not truncated.
sieve0.2.1 or later uses these limits by default. Earlier versions stop at 8,192.- The decision engine declares 65,536 (state) and 32,768 (question).
Calibration
- One global temperature, T = 1.050, fitted by NLL on the calibration split (43,309 questions).
- T does not change answers.
temperature=1.0gives raw probabilities. - Results: accuracy 0.9013, NLL 0.2499, Brier 0.1369, ECE 0.0054 (raw T = 1: ECE 0.0061).
Benchmarks (self-reported)
| Measure | Score |
|---|---|
| Decision Index Full score (leaderboard 0.3.2) | 60.17 |
| Public score (own run, 0.3 suite) | 60.04 (raw index 69.32, breadth 58.67) |
- Area scores: Knowledge and Reasoning 43.5; Language Understanding 65.7; Retrieval and Classification 67.8; Tools and Automation 77.5; Arts and Human Taste 40.8.
- Selected: HellaSwag 0.976; WinoGrande 0.926; BFCL 0.978; GSM8K 0.599; GPQA Diamond 0.485; HLE 0.102.
- Coverage: 140,178 scoreable requests. The card says both "43 benchmarks" and that the index averages 37; we are checking this with the authors.
Latency (self-reported)
Per Decision Index request, A100 80GB PCIe, eager, one at a time: median 151 ms, mean 416 ms, p80 425 ms, p95 966 ms. The card notes this is not the maintainers' RTX PRO 6000 measurement.
Training
- 153,988 records. 1 epoch, 32 records per step, 4,813 steps, about 27 hours on two A100 80GB PCIe GPUs.
- AdamW, learning rate 5e-5 (adapter and head). OneCycle with 10% warm-up. bf16.
- Checkpoint step 4,500, chosen by accuracy on a fixed 2,001-question calibration sample (NLL as tie-breaker). Benchmarks were not used for selection.
- Augmentation: permuted choice options and "none of the above". Cross-entropy loss.
- Training files are not published.
- The card's split figures (30,776 calibration and 123,177 test) do not match its total or its 50/10/40 ratio. Treat them as unverified; we are checking them with the authors.
Limitations
- The adapter alone is not a text-generation model. Load it through
sieve. - Without
causal-conv1d, transformers falls back to a slower convolution whose probabilities can differ slightly. - Requests over the token limits are refused.
- No Inference Provider currently hosts the model.
Intended use
Calibrated probabilities over options for typed questions about a text or JSON state, without generation.
Licence
Apache-2.0 for the model and code. Backbone terms: being confirmed for the base model named above.
Data retention
The model keeps nothing between calls. Through the hosted API, Sieve does not store prompts, state or answers (see data retention).
Install and call
pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"
pip install --no-build-isolation causal-conv1d==1.7.0 # recommended
sieve-serve --model sthanika-ai/Sieve-27B --graphsThrough System One: "model": "sieve-27b-v1". See the quickstart.
Next steps
Last updated
Sieve-9B-v2
Model card for Sieve-9B-v2, a mid-size decision model with per-category calibration. It returns probabilities and does not generate text.
Sieve-27B-v2
Model card for Sieve-27B-v2, a large fully fine-tuned decision model with per-category calibration. It returns probabilities and does not generate text.