PrismML Releases Bonsai 27B: 1-bit and Ternary Builds of Qwen3.6-27B Running on Laptops and Phones
” style=”width:100%;border:0;height:600px;display:block;overflow:hidden;” scrolling=”no” loading=”lazy” title=”Bonsai 27B interactive explainer”>
That structure shapes the pressure mechanism below.
How Compression Works
Each weight is coded, with one shared FP16 scale per group of 128. The effective weight is w_i = s_g · t_i.
The ternary value carries log2(3) ≈ 1.585 pieces. One FP16 scale at 128 weights adds 16/128giving ≈1.71 bits per weight. That’s a ~9.4× reduction versus FP16. Binary costs 1 + 16/128 = 1.125 fragments, ~14.2× reduction.
The presentation works end-to-end on all matrix heavy components. That’s embedding, projection, MLP projection, and LM head. Only the negligible tail of normality and scale parameters remain high precision.
Measured as a real measure, the Qwen3.6-27B “4-bit” build (Q4_K_XL) is 5.2 bits in weight. The “2-bit” architecture (IQ2_XXS) is 2.8. Bonsai is also a departure from BitNet, which avoids collapse only by pre-training from scratch.
The obvious question is how much compression costs in accuracy.
Working
PrismML tested 15 benchmarks in inference mode, using EvalScope with vLLM on H100 GPUs. Ternary Bonsai 27B retains 94.6% of the FP16 base, while 1-bit Bonsai 27B retains 89.5%.
| What’s different | A true bpw | You step on it | Thinking is a measure | Density (1/GB) |
|---|---|---|---|---|
| Qwen3.6-27B FP16 | 16.0 | 54GB | 85.07 | 0.051 |
| Qwen3.6-27B Q4_K_XL (“4-bit”) | 5.2 | 17.6GB | 84.99 | 0.155 |
| Qwen3.6-27B IQ2_XXS (“2-bit”) | 2.8 | 9.4GB | 72.73 | 0.199 |
| Ternary Bonsai 27B | 1.71 | 5.9GB | 80.49 | 0.400 |
| 1-bit Bonsai 27B | 1.125 | 3.9GB | 76.11 | 0.530 |
| Section | FP16 | Ternary | 1-bit |
|---|---|---|---|
| Mathematics | 95.33 | 93.40 | 91.66 |
| Coding | 88.74 | 85.96 | 81.88 |
| Knowledge and thinking | 83.15 | 76.96 | 73.39 |
| Agent call and tool | 80.00 | 74.01 | 66.03 |
| The following instruction | 78.47 | 71.77 | 65.74 |
| An idea | 72.61 | 65.19 | 59.57 |
A typical sub-4-bit architecture fails in a different way. IQ2_XXS falls to 57.5 in AIME26 and 56.4 in LiveCodeBench. It still scores 88.93 on MMLU-Redux, so the short-form benchmarks close the fold. Gemma-4-31B Q2_K_XL repeats that pattern in the second base model.
Scores alone, however, do not define exemption. Memory does.
Memory is a Binding Limit
Installing the phone is tighter than the storage numbers suggest. iOS limits a single application to about half of the physical memory. So a 12GB iPhone yields about 6GB.
KV reserve is the second budget. Only 16 of the 64 layers carry the growing full attention cache, so FP16 costs ≈64 KB/token. A 262K window costs ≈17.2GB, and a 4-bit KV cache reduces that to ≈4.3GB.
Tolerance is measured. Against its base FP16-KV, Ternary Bonsai shows 0.0011 nats of forward-KL output in MATH-500. Q4_K_XL shows 0.0146.
The peaks follow. For 100K tokens with FP16 cache, 1-bit maxes out at 11.6GB and ternary at 14.7GB. The resulting Q4_K_XL line requires ≈25.6GB.
Once the model is fit, the result is the next question.
Throughput and DSpark Speculative Decoding
| The platform | What’s different | tg128 | page 512 |
|---|---|---|---|
| M5 Max | Binary | 66.4 | 874 |
| The M5 Pro | Ternary | 26.2 | 393 |
| iPhone 17 Pro Max | Binary | 11.0 | 111 |
| H100 (CUDA) | Binary | 104.8 | 2755 |
Generation is bound to memory-bandwidth, so fewer bytes per step means more tokens per second. The first fill is computerized and the gain is low.
PrismML also ships with a DSpark editor trained against the Bonsai 27B target. For H100 at draft depth k=4, the binary target reaches an acceptable length τ=3.6. That’s 143.8 tok/s, 1.37× speed. Validation is not lost, so the output is always the same as the distribution. In Apple Silicon the drafter is disabled by default at batch size 1.
Running it
Ternary 27B is the default demo repo. Start the server, or generate directly:
./scripts/start_llama_server.sh # OpenAI-compatible API + chat/vision UI on :8080
./llama-cli -m ./Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-Q2_0.gguf
--mmproj ./Ternary-Bonsai-27B-gguf/mmproj.gguf -c 0
-p "Explain KV cache growth."
mlx_lm.generate --model prism-ml/Ternary-Bonsai-27B-mlx-2bit
--prompt "Explain KV cache growth."The tooltip uses the standard OpenAI style tools list:
curl
-H "Content-Type: application/json"
-d '{
"messages": [{"role": "user", "content": "What is the weather in Lisbon?"}],
"tools": [{
"type": "function",
"function": {
"name": "get_weather",
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
}
}]
}'The phone rang again choices[0].message.tool_calls. Thinking mode is turned on by default; thinking_budget_tokens changing it according to each request.
That maps to four usage patterns.
Use Cases
Local laptop-based agents use a ternary architecture to run cache code that is more than 262K tokens long. The phone’s spatial reasoning uses a 1-bit architecture; the white paper measures 672 tokens per 1% iPhone battery. Privacy-sensitive and offline workflows keep commands on-device by design. Combined with a 4-bit KV cache, a single GPU serving matches the quality of the 27B standard on a 24GB card.
Key Takeaways
- Bonsai 27B moves Qwen3.6-27B into binary or ternary weights, not a new pretrain.
- Ternary keeps 94.6% of FP16 in 5.9GB; 1-bit saves 89.5% in 3.9GB.
- PrismML claims that the 1-bit architecture is the first 27B-class model to fit a phone.
- Standard sub-4-bit builds are optionally dropped in AIME, LiveCodeBench, and agent functions.
- Everything runs under Apache 2.0, in llama.cpp (CUDA, Metal) and MLX.
Check it out Technical details, GitHub Repo again Model weight. Also, feel free to follow us Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to Our newspaper. Wait! are you on telegram? now you can join us on telegram too.
Need to work with us on developing your GitHub Repo OR Hug Face Page OR Product Release OR Webinar etc.? contact us
