Sieve-27B-v2
Model card for Sieve-27B-v2, a large fully fine-tuned decision model with per-category calibration. It returns probabilities and does not generate text.
Hugging Face: sthanika-ai/Sieve-27B-v2 (Text Classification, Apache-2.0)
API model name: sieve-27b-v2. The alias sieve-27b points to this build in the current model catalog, and so does sieve-preview.
Sieve-27B-v2 answers typed questions about a text or JSON state. It returns a probability for each option. It never generates text.
Key facts
| Item | Value |
|---|---|
| Base model | Qwen/Qwen3.8-27B as written on the card, revision 1d4bf0f2, text part only. We are confirming this base name with the authors. |
| Method | Pointer head that scores each option's end against the decision point, with a calibrated softmax |
| Trained weights | Full fine-tune of the 64 decoder layers: 24,353,196,544 parameters. Token embeddings and final norm frozen. Pointer head: 256-dimensional projections in fp32, trained from scratch. |
| Packaging | PEFT modules_to_save for the 64 layers (bf16), plus a zero rank-1 LoRA on the embeddings. No LoRA rank applies to the decoder layers. |
| Question types | choice (up to 255 options), noul, score (not trained; uses the global temperature) |
| Memory | About 52 GB in bf16 (50.3 to 53.1 GB on the CUDA-graph server). One 80 GB GPU is enough. |
Limits
- Training and default: 16,384 tokens for the state, and 16,384 for each question with its options.
- Longer requests are refused, not truncated.
- The decision engine declares 65,536 (state) and 32,768 (question).
Calibration
- Global T = 0.999.
- 25 per-category temperatures from 0.928 to 1.127.
- Categories: type x option count (2, 3-4, 5-9, 10-25, 26+) x state kind (empty, text, JSON). The most specific match is used, otherwise the global T.
- Temperatures do not change answers.
temperature=1.0gives raw probabilities. - Results on 132,395 calibration questions (per-category): accuracy 0.9263, NLL 0.1874, ECE 0.0020.
Benchmarks (self-reported)
- Decision Index 0.3 public suite: 68.02 (raw index 75.61, breadth 67.38). Not on the leaderboard. The leaderboard Full score for this model is not available yet.
- Area scores: Knowledge and Reasoning 56.8; Language Understanding 66.0; Retrieval and Classification 73.4; Tools and Automation 85.8; Arts and Human Taste 58.9.
- Coverage: 140,178 scoreable requests over 43 benchmarks, with no truncation or errors.
- The card notes that the Decision Index board ranks by a different Full score.
Latency (self-reported, NVIDIA B300, in-process, eager, one at a time; upper bounds)
Median 136 ms, mean 347 ms, p80 322 ms, p95 918 ms. Cached 1,433-token state, server p50 with CUDA graphs: about 96 to 98 ms for 2 to 50 options.
Training
- 400,985 records. Split 60/10/30 (train, calibration, test) as stated, with 65,959 calibration and 201,027 test records. These figures do not match the stated ratio (10% of 400,985 is not 65,959); treat them as unverified.
- 3,133 steps, 1 epoch, 128 records per step. 27.55 hours (55.1 GPU-hours) on two B300 GPUs.
- AdamW. Learning rate 4e-6 (backbone), 5e-5 (head). Weight decay 0.01. Gradient clip 1.0. 5% warm-up, then cosine decay to 0.1x.
- bf16 compute with fp32 master weights. FSDP2. Gradient checkpointing.
- Training files are not published (hashes in
training_config.json).
Requirements
sieve0.3.2 or later. It loads the full fine-tune in place. The adapter alone is not a text-generation model.- Optional on A100 and H100:
causal-conv1d==1.7.0. Results can differ slightly from the B300 runs, which used PyTorch convolution.
Limitations
scorequestions were not in training.- Benchmark scores are self-reported.
Intended use
Routing and classification-style decisions over text or JSON state, returning calibrated option probabilities without generated text.
Licence
Apache-2.0 for the model and code. Backbone terms: being confirmed for the base named above.
Data retention
The model keeps nothing between calls. Through the hosted API, Sieve does not store prompts, state or answers (see data retention).
Install and call
pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"Through System One: "model": "sieve-27b-v2" (or "sieve-27b"). See the quickstart.
Next steps
Last updated