If you want to try out out the GGUFs from https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th... be aware that you need Prism's llama.cpp fork to get them to work, from https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-...

This should work:

  cd /tmp

  # Get the Prism macOS runtime
  curl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz
  tar -xzf bonsai-runtime.tar.gz

  # Get the ~5.95 GB GGUF model:
  curl -fL https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf

  # Run the server, I used port 8331
  ./llama-prism-b10685-7dffb15/llama-server \
    -m Ternary-Bonsai-2-27B-PTQ1_0.gguf \
    --port 8331 -ngl 99 -fa on -c 32768

Then open http://localhost:8331 for the (very good) baked in llama-server web UI... or run a prompt via the API like this:

  uvx llm openai endpoint http://127.0.0.1:8331/v1 \
    --model bonsai-2-27b --responses hi

That's running at ~20 token/second for me on an M5 Pro (after a server restart I got 44 token/second, not sure why), but I'm pretty sure something isn't working right, on startup the server said "ggml_metal_device_init: - the tensor API is not supported in this environment - disabling".

I really wish people would stop saying N times smaller than something when making a comparison; that makes no sense - it's 1/9th (11.11%) the size. You don't get a smaller quantity by multiplying by a number greater than 1.0. You could instead reverse the subjects being compared - "the original model is 9x bigger than this new smaller, efficient model" or some such. That makes sense.

I keep seeing this being used when people talk about efficiency or performance gains and it's just very unintuitive language.

There's a chap called Bijian Bowen who does very quick agentic coding challenges for new models (very soon after release!) mainly for toy games or websites. He just did one for this model and included a comparison with the base model Qwen 3.8 which shows the "near-lossless" claim should be taken with a grain of salt. It is an interesting model if you are GPU starved and want local, but you might have trouble finding things it is good at.

Has the hype fizzled out yet? Can anyone post a link to something they've done with success?

These are small enough that you can run them entirely in the browser https://huggingface.co/spaces/webml-community/ternary-bonsai...

Remember to clear the downloaded weights afterward.

Like the last model, it's amazing they work as well as they do. Use it for any longer task and they fall apart spectacularly and in interesting ways.

> Ternary Bonsai 2 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, for 1.76 effective bits per weight

If I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what's the special sauce that makes them better?

https://news.ycombinator.com/item?id=49611128

Let’s see, so if you get the same 1/9th the size compression ratio with GLM-5.3-Flash, then you’d end up with a ~72GB model that’s about as good as GPT-5.6 Sol (high), according to artificialanalysis.ai

Which is within reach of some higher end consumer hardware, especially with layer offloading.

You have to wonder what kind of trouble the “labs” are in when this is becoming possible. Lots of money, where’s the moat?

Nice! Does anyone know how this compares to the Unsloth quantizations of this model? https://unsloth.ai/docs/models/qwen3.8#run-qwen3.8-guide

What is never totally clear with a lot of these releases is the scope of what it's good at. Models that can run with good speed on affordable consumer hardware for coding only is the dream. I am never going to use this for writing, images, or "general knowledge". Coding only

Running at about 7-8 tok/s (~60 tok/s prefill) on a Mac Mini M2 16GB.

So far feels smarter than Bonsai 1 27B, it’s slightly larger than the Q1_0 quant. Super exciting stuff :)

Bonsai 2 27B · Radeon RX 7900 XTX

  - 89 tokens/sec generation with speculative decoding
  - 81 tokens/sec at 20k context
  - 474 tokens/sec ingestion at 20k — about 42 seconds
  - 10.1 GiB peak VRAM with a 24k context window

ROCm 7.2.3 · PQ2_0 · Qwen Q4 MTP, draft length 2

---- versus ----

Qwen3.8-27B IQ3_S · Radeon RX 7900 XTX

  - 79 tokens/sec generation on a short coding prompt
  - 61 tokens/sec at 60k context
  - 53 tokens/sec at 95k context
  - 558 tokens/sec ingestion at 60k — about 108 seconds
  - 19.9 GiB peak VRAM during coding tests with a 100k context window

Vulkan · GSQ-RCO IQ3_S · MTP, draft length 2 · vision projector loaded

Fyi, if you're trying to run this under AMD/HIP:

PTQ1_0 has no optimized MMQ-Path in their llama-cpp fork, try running PTQ2_0 (needs a bit more vram, but is about 2x faster on my 6700 XT)

https://gist.github.com/nilsherzig/b8266d001c5c01bdb3d81d209...

Love this for the folks with 16gb graphics cards - 3.8 27b has been incredible but not quite runnable on anything less than 32gb - will try loading this up on my 16gb intel b50 and see how it goes - not sure these quants can be accelerated by the XPU cores yet but maybe in time!

They need to make a Big Bonsai, something at the enterprise levels that can compete with DSV4 Flash etc.

I'm hoping they release an 8B v2 based on the Qwen 3.8 series in the near future - that would give us a really powerful model that could be run directly on users phones.

Testing on a MBP m4 pro 24gb

~100t/s prefill, ~15t/s, dropping to ~10t/s later with 64k context.

The issue is I have yet to find a useful agentic local llm that I can run on this machine.

Just given a relatively simple task on a swift app, took 25 minutes, brainstorming like crazy but can not decide on what to do. Eventually I killed it. GPT 5.6 sol-medium took 3 minutes to complete the same task for reference.

For anyone who wants to try Bonsai 2 without setting up runtimes, downloading the right quant etc, try here - https://triangllabs.ai/otis

What did they do with their benchmarks? I've never seen Qwen3.6 27B this close to Qwen3.8 27B in any aggregated summary... Makes one questions the entire accuracy section.

Stupid question: Does "total model footprint of 5.9GB" mean it will run in 8GB of RAM ? Or is that the size on disk ?

I tried their WebGPU version and it immediately started looping. Yeah "near lossless" my ass. Plus the reasoning that it looped on was clearly wrong and unlike the non quantized 27B

Never heard of Bonsai before, but that looks great and promising for local on-device inference.

Yet, seems like there is still another year for improvements.

I like local models (but not mainly using them) for offline needs.

I wonder how their talks with Apple went. Having this run on the TPU opposed to just the GPU, which drains a significant amount of battery life by comparison, is what I'm really interested in.

I think if they made this for Qwen3.8-Next it could fit in a single 5090?

Cautiously optimistic. The V1 was noticeably weak on world knowledge but here the 3.8 base model is geared more towards reasoning than world knowledge anyway so might not matter as much

GPT Astra did some benchmarking on the DGX Spark. Speed: 34.38 tokens/sec for generation.

Seems like we don't have a drafter model yet so it could not test with speculative decoding on. ngram speculative decoding did not help too much either - not enough accepted tokens.

Smaller size I suppose does not mean better performance in this case - we maybe limited by Spark's low memory bandwidth.

I wonder if they could apply the same to Qwen3.8-Flash-Next.

If they did 60gb -> 6gb to Qwen-27B, could they possibly do the same to the MoE model. 36gb lossless 126B model seems like an impossible task.

What I'd love to see is this done for DS4.1 Flash.

That would bring it down to the point where it can fit in 128GB on things like the Spark or Strix Halo.

On openrouter I had to filter out glm 5.3 flash instances running fp4 - making sure it only ran fp8 - as the quantized models kept going crazy / off track.

Assume bonsai has the same ticks

I'd love to see a Bonsai model start with a 100B+ parameter model and get that down to <30 GB. But maybe at that point we call it Topiary?

LLM quants seem to eerily converge to modern/not so modern graphics techniques. You wouldn't think it would apply but it's obvious in hindsight. In fact mining graphics ideas is probably a good inspiration for efficient LLM architecture.

For example, the Hadamard activation transform used here feels a lot like multiplying Fourier basis ala DFT; strong parallels to how image codecs work to make the residuals more compressible (especially discrete block codecs like are used in GPU compressed textures).

I thought I was being clever suggesting that you could even abuse texture decode units to efficiently sample compressed LLMs with hardware; turns out Apple foundation models are already doing this [1].

[1] https://arxiv.org/abs/2507.13575

RAM requirements? My current rule of thumb is “a byte per parameter”, but I doubt this runs in 1/9th that (~ 3GiB).

Also, perf speedup?

I'm not following the local mdoel scene too closely but this seems quite amazing. Is this able to be run on Apple silicon too?

I tried it on my 6GB GPU and got 0.67 tokens per second. Need more than 6GB to run it well.

undefined

Hello! May I ask, is this model compatible with my RX 9070 on Linux?

(In case anyone remembers the compression post from yesterday[1], I checked and this one doesn't qualify for further compression - it's not zero-biased at all.)

[1] https://news.ycombinator.com/item?id=49732931

Awesome results. Opens up doors for a lot of people.

On actual agentic task , it just fail.

Well this is quite impressive!

Is it just me or are local models getting better (catching up) a lot faster than the frontier models are getting better (creating distance)?

If true, that would be a very welcome development.

undefined

[flagged]

[flagged]

[dead]

[flagged]

[flagged]

[dead]

[flagged]