Where the Bits Go: Non-Uniform Quantization of Qwen3.5-9B

TL;DR

We quantize Qwen3.5-9B using our new post-training quantization stack that optimizes bit-budget allocation across the model. This can reduce the model's memory footprint by 82.5%, from ~18 GB to 3.2 GB.

We share three versions of models of various average precisions:

• OraQuant-Q4_K_M: Qwen3.5-9B-OQ-Q4_K_M (5.08 bits per weight, or bpw);
• OraQuant-Q3_K_M: Qwen3.5-9B-OQ-Q3_K_M (4.19 bpw);
• OraQuant-IQ2_XXS: Qwen3.5-9B-OQ-IQ2_XXS (2.84 bpw).

The output models are fully compatible with GGML Unified Format (GGUF) and can be served using the standard llama.cpp.

We compare our models with the equivalent state-of-the-art from Unsloth. The result: the lower the precision, the better Ora models perform compared to Unsloth ones! In particular, for 2.84 bpw, Ora's models score 40 on AIME-25, while the equivalent Unsloth counterpart fails completely with a zero score.

Below the precision of 3 bits, what plays an even bigger role is how the quantized weights are represented and encoded. To check this, we produced a fourth model in another encoding called EXL3:

• OraQuant-EXL3: Qwen3.5-9B-OQ-EXL3 (2.81 bpw, requires a different exllamav3 runtime).

Comparing our two models, equivalent in terms of memory but different in terms of the encoding format (GGUF vs. EXL3), we observe that the EXL3 is a better performer: it scored 64.6 on AIME-25 vs. the 40.0 score of the GGUF model.

Introduction

Nowadays, deploying capable large language models (LLMs) on resource-constrained devices like phones and laptops is limited mainly by the models' big sizes. For example, Qwen3.5-9B, a highly capable model open-sourced by Alibaba, requires roughly 18GB just for its weights, far more than what is typically available in such devices. To run such a model locally, one needs to compress it.

There are two levers for shrinking a language model: pruning, which removes redundant parameters from the model, and quantization, which reduces the precision of each weight's representation. Pruning is a topic for a future post, here we focus on quantization. When it comes to reducing the memory footprint of the model, the consensus is that quantization is "the most bang for the buck". In contrast to pruning that genuinely removes weights from the model, quantization is a more mild and graceful approach to model compression.

There are several degrees of quantization. BF16 is the format where the model's weights are represented with 16 bits, and this format is broadly used for deploying and accessing the models. Quantization from 16 bits to 8 bits is typically lossless; quantization from 8 bits to 4 bits only incurs minimal accuracy loss. Quantization below 4 bits, however, is trickier. This is the regime where benchmark accuracies start to severely degrade and, without careful treatment, low-bit quantized models become entirely useless. Producing viable models in this precision can be viewed as the holy grail of LLM compression.

In this blog post we show how we took Qwen3.5-9B from 16 down to 2.8 bits per weight (bpw), which is an 82.5% reduction in memory footprint (~18GB -> 3.2GB).

We deliberately test this on a difficult model. Qwen3.5-9B is a dense reasoning model, meaning that all 9B parameters take part in every generated token. That makes low-bit quantization especially hard: during reasoning the quantization errors can accumulate over thousands of tokens. If low-bit quantization is going to break somewhere, this is a good place to look.

The next section is an optional background. Expand it if you want to learn more about:

• Deciding on where the bits go is hard: Why heuristics struggle below 4 bits, and where state-of-the-art Qwen3.5-9B quantizations sit;
• What we do: Optimization: How we explore the bit-allocation space;
• Format defines performance: Why GGUF vs. EXL3 starts to matter below 3 bits.

Otherwise skip directly to the results!

Background

Results

Experimental setup

Bits per weight (bpw) throughout this post are honest file-size averages over all stored weights, including embeddings, format metadata, and the LM head. The vision tower and MTP heads are excluded.

To assess each model's performance, we consider the following four benchmarks:

• AIME-25: competition math, hours-long thinking traces;
• MATH-500: shorter-form math;
• GPQA-diamond: graduate-level science, multiple choice;
• MMLU-Pro: broad knowledge under a strict answer-extraction protocol.

We follow the evaluation protocol of RedHat AI: temperature 1.0, top-p 0.95, presence penalty 1.5, and a 64k-token generation budget. Multi-seed sampling is used for AIME-25 (8 seeds) and for MATH-500 and GPQA-diamond (3 seeds each). MMLU-Pro is run once per model with a stricter protocol (no thinking, greedy decoding, 2k generation budget). That MMLU-Pro setup is consistent within this post, but it is not comparable to Red Hat's published MMLU-Pro figures, which use a different harness and generation config.

Table of scores

In Figure 1, we show the scores of the models considered in this blog post. Average over the four benchmarks (AIME-25, MATH-500, GPQA-diamond, MMLU-Pro):

† Mixed protocol: reasoning scores as published by Red Hat; MMLU-Pro value derived under our in-house protocol.

Results analysis

General trend: Optimization helps more at low precision

At around 5 bpw, optimization buys almost nothing: simple GGUF recipes are already good enough.

At ~4 bits, however, optimized allocation improves AIME-25 by 14 points at the same file size.

Below 3 bits, the gap becomes much larger: an optimized allocation produces a model that can still reason, while a heuristic allocation at essentially the same size falls apart.

AIME-25: the stress test

The per-model scores appear in the AIME-25 view of Figure 2, with seed counts and standard deviations in the tooltips. Three results stand out:

Cross-format comparison:

• OraQuant-Q4_K_M (83.3 at 5.08 bpw) is statistically tied with BF16's 82.9.
• Both OraQuant-Q4_K_M and OraQuant-Q3_K_M (79.2 at 4.19 bpw) beat RedHatAI-w4a16 (68.8 at 8.97 bpw).

This means that uniform 4-bit weight quantization damages long reasoning even when embeddings and linear attention are not quantized.

In turn, an optimized allocation does not damage reasoning, even at less than half of the storage.

At ~4 bits: Ora and Unsloth builds differ by 13.8 points (79.2 vs. 65.4), which is driven entirely by where the bits were placed and how the tensors were fitted.

Below 3 bits:

• Unsloth's UD-IQ2_XXS scores a genuine 0.0. Across all 8 seeds (total of 240 generations) it produced not a single correct answer. 82% of its generations never reach a final answer at all.
• Even the optimized allocation loses most of the long-thinking regime (40.0 vs. 82.9). We explain how it loses it in the next section.

Below 3 bits: What fails?

Pass/fail scoring of AIME-25 hides the exact failure mechanism. As shown in Figure 3, classifying every generation as correct, answered but wrong, or no final answer separates two very different ways of losing points:

• Above 3 bits: The degraded math ability. At 4.17 bpw, Unsloth's Q3_K_M still terminates fine, with an 8% no-answer rate, but its accuracy collapses to 71% of BF16. At the same size, OraQuant-Q3_K_M retains accuracy at roughly 96%, which is almost indistinguishable from BF16. This score gap between Unsloth and Ora is caused by heuristic quantization damaging reasoning quality, with better bit allocation preventing it.
• Below 3 bits: Degeneration of conciseness. OraQuant-IQ2_XXS fails to commit on 49% of generations, exceeding the 64k generation budget, yet it still solves 78% of what it does answer. The math ability is intact, what breaks is the ability to stop reasoning. Unsloth's UD-IQ2_XXS, however, is not only overly chatty, but also just wrong: 82% of answers never conclude and none that conclude are correct.

The second failure mode is illustrated in Figure 3: one can observe that below 3 bpw, the dominant failure is generations that never conclude, not wrong answers.

MATH-500, GPQA-diamond, MMLU-Pro

The remaining three benchmarks are the other views of Figure 2. We observe two phenomena:

Above 3 bits:

• MATH-500 and GPQA are less discriminative because they require shorter generations. These do not allow the quantization error to compound, so both Unsloth and Ora quantizations land close together.
• However, what clearly separates Unsloth and Ora is long thinking traces (AIME-25) and broad knowledge (MMLU-Pro). The latter exhibits a gap of +4.4 at 4.19 bpw.

Below 3 bits:

• All benchmarks become discriminative, because the Unsloth checkpoint shoots off the cliff: its outputs are barely above the 25% random floor for MATH-500 and GPQA. In turn, at the same size, OraQuant GGUF checkpoint gives 82.2 and 57.9 on MATH-500 and GPQA, respectively.

Thus the ultra-low-bit regime draws the sharpest line: heuristic bit assignments break the model, whereas Ora's optimal allocation preserves it.

Discussion

Where the bits go? GGUF type per tensor

Figure 4 shows how the GGUF bits are distributed across the model. Specifically, each cell shows which GGUF quantization type a weight matrix received: violet-to-pink for lattice i-quants, green-to-mint for k-quants, darker color means fewer bits within a family. The x-axis corresponds to the model's layer; rows are grouped into full attention, linear attention, and MLP. Ora's optimized allocation appears on the top, while Unsloth's fixed recipe-based allocation is shown on the bottom.

Unsloth's allocation panels appear as flat, repeating bands, because every layer of a module gets the same allocation type. Ora's optimized panels have texture, modulating precision for each tensor.

At ~5 bpw the two checkpoints look broadly similar: with enough bits, most tensors sit comfortably at Q4/Q5. The benchmarks of these models are almost identical, too.

At ~4 bpw, the models start to diverge:

• Ora's optimizer pushes late-layer modules down to Q2_K, then spends the savings on raising small decay projections and token embeddings to Q6_K and varying MLP precision layer by layer.
• In turn, using a hand rule, Unsloth uniformly pays Q8_0 for decay projections, drops token embeddings to Q3_K, and applies the same precision to all MLP modules.

The fine-grained optimization wins additional 14 points of AIME-25 at the same file size.

At ~3 bpw the same pattern sharpens into starker trade-offs:

• Embedding and output head are pinned to Unsloth's types here, so the following comparison is purely about the model's body.
• Again, Unsloth applies flat rates: Q4_K on every linear-attention out-projection, Q8_0 on every decay/gate projection, IQ2_XXS on MLP modules.
• Ora's optimizer selectively changes this: it drops twenty of the twenty-four out-projections to Q2_K, trims the decay/gate projections from Q8_0 to Q6_K. It spends the freed bytes on attention modules and output projections (up to IQ4_NL/IQ3_S), again applying per-layer mix of five different types across the MLP.

At this budget these reallocations are not cosmetic: they give +53 MATH-500 points reported above.

Changing the encoding, not just the allocation: OraQuant-EXL3

Bit allocation in EXL3

Everything so far allocated bits across GGUF formats only. The failure-mode analysis above raised a harder question: at the ultra-low-bit regime, the best allocation still fails to conclude half its long generations. Does the reason for this lie purely in the small bit-width budget, or does this indicate a more fundamental issue with the model file encoding?

GGUF's i-quants round each weight to a small fixed lattice, whereas EXL3's encoding is fundamentally denser at the same bpw (see above). If the verbosity comes from GGUF's lattice encodings rather than the byte count, swapping the encoding should cure this without increasing the bit-width budget.

To produce the EXL3 quantized model, we let our optimizer start from the original model and choose from trellis bitrates $K\in \{2,3,4\}$ instead of twelve GGUF quantization types. In Figure 5, we show that the result does not correspond to a flat per-module rule: full-attention K/V projections get assigned K4, most MLP layers get K2 with layer-by-layer K3 exceptions, and the linear-attention projections mix all three rates. This differs from the GGUF assignment on 81 of the 200 tensors.

Benchmark performance comparison

A mere format change combined with Ora's optimization leads to the largest fixed-byte quality jump in this blog post:

Same optimizer, same size, +12.3 average points. On MATH-500 the trellis build is statistically at the BF16 ceiling. Thus, at the ultra-low budget, the quality is far cheaper to buy by upgrading the encoding than by shuffling the allocation.

This matches where recent low-bit releases are converging. Escha Labs' Escha-W2 ships a large MoE model whose released tensors indicate trellis-family coding, served by the lab's own runtime. Syzygy's Mach-1 encodes a 35B MoE at 1.7 bpw with tiered trellis rates. Neither is directly comparable to Qwen3.5-9B, since both are larger-scale MoEs, which makes them friendlier to low-bit coding than a dense 9B. But both checkpoints feature a sub-3-bit trellis code and non-uniform rates chosen per tensor, thus making it clear that this encoding is preferable for ultra-low-bit quantization.

Figure 1 above reflects this: the OraQuant-EXL3 point dominates the sub-3.5 GB regime, while the GGUF ladders converge above 4.5 GB where allocation and format both stop mattering.

One caveat keeps this honest: EXL3 is not a llama.cpp format. The GGUF checkpoints serve anywhere llama.cpp runs, whilst the EXL3 model serves with exllamav3.

EXL3 pulls an ultra-low-bit model out of hesitation

With our sub-3 bpw OraQuant-EXL3 and OraQuant-IQ2_XXS in hand, and in the context of the AIME-25 failure mode analysis we did earlier, it is worth examining why their scores differ so drastically. For this, we additionally ran two probes over every AIME-25 generation (thinking trace plus final text, all seeds) from the sub-3 bpw models and the BF16 reference.

The first probe detects verbatim loops, using the character-level repetition detector from antidoom. A loop is detected when exactly the same string repeats at least four times.

The second probe scans the failures, meaning generations that never produce a final answer, and reveals if the model refused to commit to an answer it actually reached. For this, the probe scans the thinking traces for lines like "the final answer is…", "I will output…" or a boxed answer inside the trace.

Here are the results of the analysis:

The verbatim loop column is the signature of typical low-bit quantization damage. On a fifth of all generations, our OraQuant-IQ2_XXS falls into a doom loop of "Wait." repeated forever. Unsloth-UD_IQ2_XXS is even worse: 88% of its generations contain such a loop, and the loops start earlier than for our IQ2_XXS: median loop onset for Unsloth is at 18% of the generation vs. at 70% for our IQ2_XXS checkpoint. In turn, the BF16 reference does not exhibit such behavior, while our OraQuant-EXL3 demonstrates how infinite loops are not defined by the bit-width budget alone: with the same budget, it is nearly loop-free, with a failure profile looking like a "noisier" BF16.

The last column indicates how many answers are hidden in generations marked as "failed". Over half of BF16's (rare) failures contain a stated answer that never became final: the model reached a candidate answer, then kept second-guessing ("I will check for an error…", "Wait.") until the token budget ran out. This is the overthinking (or self-doubt) mode documented for R1-style reasoners. Therefore it is the model's own tendency, not something caused by the quantization. EXL3 preserves nearly the same failure composition (49%) while simply failing more often. In turn, in the heavily quantized GGUF checkpoints, loops crowd out those stated answers, so the share drops to 23%.

Both observations are supported by recent literature. For example, Lotfi et al. find that under aggressive quantization, up to 52% of failures contain the correct answer in the model's thinking traces. Moreover, quantized models over-sample hesitation markers ("wait", "but", "alternatively"), amplifying an overthinking tendency already present at full precision, which is exactly the composition shift we observed. Work on 2-bit inference shows that detecting loops and committing to an existing answer recovers 2-bit Qwen3-8B on MATH-500 from 17.2% to 74.2%.

These observations also point to the fix. The model's hesitation corresponds to accumulated quantization noise that cannot be fixed simply by bit allocation. Recovering it requires adapting the weights through, e.g., quantization-aware training (QAT): healing the already-quantized model against the BF16 model on general data. In our previous blog post, we already analyzed what QAT can do to an ultra-low-bit quantized model.

Conclusions

In this post we took Qwen3.5-9B from 16 down to 2.8 bpw, an 82.5% cut in memory footprint (~18 GB to 3.2 GB), using our post-training quantization stack. This model is a deliberately hard target: Qwen3.5-9B is a dense 9B reasoning model, so its quantization errors compound over thousands of reasoning tokens. Rather than assign bits by hand-crafted recipe, we let our optimization stack search the $\sim N^{250}$ space of per-tensor precision choices under a global size budget, using end-to-end model quality as the objective. We produced three GGUF checkpoints (5.08, 4.19, and 2.84 bpw) that run in stock llama.cpp with no custom kernels, plus a model in the EXL3 format (2.81 bpw) for exllamav3 runtime. We benchmarked all of them against size-matched Unsloth and RedHat baselines on AIME-25, MATH-500, GPQA-diamond, and MMLU-Pro.

The primary lesson of this study is that precision spent uniformly is precision wasted. RedHat's 8.97 bpw checkpoint quantizes everything to a flat INT4 and scores 10 AIME-25 points below a 4.19 bpw checkpoint that chooses, per tensor, among twelve formats.

What protects model quality is not the average number of bits, but putting the bits where the model needs them and packing them in the most suitable format. These two levers govern low-bit quantization quality and matter more the tighter the budget gets. This trend can be observed by the following three regimes spanning generous and hardest budgets:

• At ~5 bpw, quantization is a solved problem for this model. All the checkpoints have roughly the same quality. For example, our 5.08 bpw build is statistically at the BF16 ceiling on all four benchmarks.
• At ~4 bpw, the bit allocation algorithm plays a differentiating role. Specifically, Ora uses an explicit optimization algorithm, while Unsloth assigns bits based on the sensitivity heuristic. The respective checkpoints differ by 13.8 points, 79.2 vs. 65.4, on AIME-25.
• Below 3 bpw: The role of the bit allocation algorithm is even more profound: Ora's explicit optimization produces a viable checkpoint (82 on MATH-500, within 3 points of BF16), whilst Unsloth's heuristic approach generates a broken model (29 on MATH-500, chance-level GPQA). Moreover, the file format becomes the second, equally important lever: at the same bytes, recoding into the EXL3 recovers another 12 average points compared to GGUF.