A huge model file becomes considerably smaller, and suddenly local AI seems possible on a device within reach. The memory saving is visible. What remains invisible at first is which information is lost through the changed number representation and whether this loss matters for your task.
The llama.cpp quantisation documentation describes methods by which model weights are stored with lower precision. This can save space and change processing, but it can also cost quality. A bit count is therefore a technical property and not a general seal of quality.
A number experiment without a model
To understand the idea, a home-made toy method is enough to begin with. We limit numbers to the range from minus one to one and map them to evenly distributed levels. In theory, four levels fit into two bits, sixteen levels into four. This example does not reproduce any specific GGUF method.
def runden_auf_stufen(wert, stufen):
wert = max(-1.0, min(1.0, wert))
index = round((wert + 1) * (stufen - 1) / 2)
return 2 * index / (stufen - 1) - 1
werte = [-0.83, -0.12, 0.18, 0.76]
for stufen in (4, 16):
angenaehert = [runden_auf_stufen(x, stufen) for x in werte]
fehler = max(abs(a-b) for a, b in zip(werte, angenaehert))
print(stufen, angenaehert, "largest deviation:", fehler)Run the small Python example and then change a value close to a level boundary. You can observe directly how different inputs fall onto the same stored level. That is a loss of numerical detail, not a measured error in a language model’s answer.
Real quantisation methods are more complex. They can handle weights in blocks and use additional scaling information. For that reason alone, a model’s size cannot reliably be derived from a single bit figure as a simple storage calculation.
The actual task decides
Before a model comparison, set up five short tasks with known requirements. One note contains a number of participants. A second contains a negation. A third names an exception. A fourth contains contradictory details, and a fifth lacks a piece of information being looked for.
For each task, write down what must be preserved. For example: six participants, material only after approval, an unknown date stays unknown. This way, you judge the effects on your work and not just whether the answer sounds pleasant.
A smaller variant might suit simple summaries well and fail at a particular edge task. A single success is not a general statement about quality. Nor is every linguistic deviation important in substance; what matters are the requirements set in advance.
A fair comparison changes as little as possible
Choose variants of the same base revision and keep the prompt, runtime and tasks the same. If a different model generation comes into play, a different comparison begins. File name, source and checksum help you recognise the artefacts actually used later on.
Alongside deviations in substance, also note the memory requirements and time actually measured. The weights file alone does not describe the full runtime requirements; further processing and the context used also need resources. So do not transfer other people’s device figures to your computer.
Also avoid a seemingly precise ranking based on a single task. For your own decision, a rare lost restriction can be more important than a few seconds’ difference. The evaluation sheet should keep such errors visible.
What the smaller file really makes possible
Quantisation opens up additional local uses by changing the numerical representation of the weights. The right choice remains a trade-off between resources and what your tasks need in substance.
The small level experiment shows the basic principle without a download, GPU or model call. The later comparison of variants examines a different level: how does the chosen representation affect your actual application? If you keep these levels apart, the impressively small file becomes a traceable technical decision.
Sources and status
Sources as of 25 October 2026. llama.cpp quantize documentation.
