Serving Your Own Model with llama.cpp: GPU Offload, Caching, and the Trap of a Stuck Wrong Answer
The model is fine-tuned. The next question: how do you actually use it — not inside a training notebook, but served like a real API, with a cache so the same question doesn't get recomputed over and over? This post covers that process, and one caching risk that's easy to overlook: what happens when the thing sitting in the cache is a wrong answer?
💡 How to read this post: there are a few "Guess First" boxes scattered through the text. Take a guess before checking the answer.
Table of Contents
Why GGUF and llama.cpp?
The fine-tuned model (see the previous post on fine-tuning Llama 3.1 8B) is stored as a LoRA adapter sitting on top of a base model — a convenient format for training, but not a compact one for serving. GGUF is a file format built specifically for inference: the model gets merged into a single file, quantized, and read directly by llama.cpp — a C++ runtime that runs on CPU or GPU without needing a heavy Python stack.
This notebook loads the GGUF file exported from the fine-tuning step (4.92GB, Q4_K_M quantization) and serves it through llama-cpp-python.
What "Open Source LLM" Actually Means
Before getting into serving, one term worth clarifying: "open source LLM" gets used loosely for models whose weights can be freely downloaded — but that's more accurately called open-weight. Open code and weights don't necessarily mean the training data is open too, and licenses vary widely — some are genuinely permissive, others (including the Llama family) come with their own usage terms. Four traits are usually used to judge how "open" a model really is: public accessibility, community collaboration, development transparency, and freedom to customize — but "free to use" and "fully transparent" are two different things.
🎯 Guess First: How many times faster do you think text generation is on GPU compared to CPU-only, for the same 8B model?
See the answer
21x — from 5.51 tokens/second on CPU to 115.47 tokens/second with full GPU offload. This is a direct measurement, not a spec-sheet estimate.
How Much Does GPU Offload Actually Matter?
llama-cpp-python is compiled with CUDA support matching whatever GPU is detected. The comparison, measured on the same prompt and token count:
| Tokens/sec | |
|---|---|
CPU (n_gpu_layers=0) |
5.51 |
GPU (n_gpu_layers=-1) |
115.47 |
| Speedup | 21.0x |
One technical detail that's easy to miss: n_ctx (context length) needs to be set explicitly. The library's default is usually far smaller than what the model actually supports — here it's set to 4096, instead of a default that could silently truncate longer prompts.
Caching Answers: The Key Is More Than Just the Question Text
Generating tokens one at a time is expensive. If many users ask the same thing, recomputing the answer every single time is wasteful. The fix: cache answers in a database, keyed by a hash of the instruction + input + model tag + decoding parameters — not just the question text. Why include all of that? Because if the model changes or the decoding parameters change, an old answer shouldn't just get reused as-is.
The difference is obvious once it's measured:
| Prompt | Cache miss (model) | Cache hit (SQLite) |
|---|---|---|
| "who is the first president of indonesia?" | 250 ms | 0.06 ms |
| "what is a programming algorithm?" | 1,450 ms | 0.06 ms |
The gap is 3–4 orders of magnitude — which makes sense, since a cache hit is just a SQLite primary-key lookup, while a cache miss is a full generation pass.
Plot Twist: The Cache Can't Tell Right from Wrong
This is the most important part of the whole notebook. The cache only cares about one thing: did generation finish normally? If so, the answer gets stored — correct or not.
To demonstrate this, a small test is used: a list of five fictional presidents is given in the prompt, and the model is asked for positions 1 through 5. The answer key can be checked automatically in code, not read by hand.
The result: 60% (3/5). Both wrong answers followed the exact same pattern — the model repeated the correct name for the previous position instead of advancing to the next one (asked for position 3, it answered with position 2's name).
Using one of those wrong answers, here's the sequence:
The model is called for the first time → wrong answer.
The same question is asked again → the cache serves the identical wrong answer, in microseconds.
That cache entry is deleted.
The model is called again from scratch → the exact same wrong answer comes back.
Step 4 is the whole point. Because decoding is greedy (deterministic), deleting the cache entry fixes nothing — the same model, given the same prompt, will always produce the same output, right or wrong. What actually fixes it is one of three things: change the prompt, change the data, or change the model. The cache only speeds up delivery — it isn't a correction mechanism.
Cheat Sheet
GGUF = a file format for inference, not training — the model gets merged and quantized into a single file that llama.cpp can read.
"Open source LLM" often really means open-weight: the weights are open, but the training data and license terms may not be.
GPU offload can give a speedup of tens of times over CPU-only — but it needs to be measured directly, not assumed from a spec sheet.
An LLM cache key should ideally include the model and decoding parameters, not just the question text.
A cache speeds up delivering an answer that's already been computed — it doesn't and can't judge whether that answer is correct. Deleting a bad cache entry only removes the stored copy; if the model stays the same, the same mistake will come back.
Test Your Understanding
1. Why should an LLM cache key include the model tag and decoding parameters, not just the question text?
Because changing the model or decoding parameters (like temperature or max_tokens) can produce a different answer to the same question. If the cache key is based on question text alone, the system could serve an old answer that's no longer relevant to the configuration currently in use.
2. After a cache entry containing a wrong answer is deleted, why does the model give the exact same wrong answer again when called fresh?
Because decoding is greedy and deterministic — the model, prompt, and data are identical to the previous call, so the output is identical too. Deleting the cache only discards the stored copy of the answer; it doesn't fix whatever caused the model to get it wrong in the first place.
3. What's the difference between "open source" and "open-weight" for an LLM?
Open-weight means the model's weights can be downloaded and used, but the training data or full training code may not be open, and the license can come with its own conditions. "Open source" often gets used loosely to mean the same thing, even though a stricter definition of open source usually demands much broader openness.
The full notebook (including the CPU/GPU comparison and the wrong-answer cache demo) is at the GitHub repo — built to pair with the Llama 3.1 8B fine-tuning repo that produces the GGUF model it loads.
