A 2018 Mining Card Beat GPT-5
Old GPU + on-device optimisation + the right harness: the three things I had to get right
- Hardware: 2018 RTX 2080 Ti, 22GB modded, Turing sm75 — no FP8, no NVFP4
- Result: 98.2% HumanEval, 50–57 tok/s decode, 128K context, 21.9 of 22.5GB used
- Fix 1 — replace the chat template: prefix cache 0% → 97.4%
- Fix 2 — pick the harness: same weights, ±40 points
- Fix 3 — stop trusting defaults: one setting cost 6× load time and 38% throughput
What the card actually did
A GPU released in 2018: RTX 2080 Ti, 22GB modified, Turing architecture, sm75. It runs a dense 27.78B-parameter model at 128K context in 4-bit.
| Metric | Measured |
|---|---|
| VRAM after load | 21.9 / 22.5 GB (only 600 MB headroom) |
| Warm decode | 50 – 57 tokens/sec |
| Long-prompt prefill | 615 – 616 tokens/sec (9,624 tokens in 16.7s) |
| Multi-turn prefix-cache hit | 97.4% |
| Start to ready | 51 seconds |
| HumanEval (164 problems) | Gemma4-26B 98.2% · Qwen3.8-27B 95.7% |
| MBPP (100 problems) | Gemma4-26B 87.0% · Qwen3.8-27B 82.0% |
This 98.2% cannot be compared directly against the published table. The published numbers use the official raw completion protocol, and that protocol does not work with chat models: it stops on a newline followed by def, on the assumption that the model continues inside the function body. Chat models re-emit the whole signature at column 0, so the stop fires immediately. In my first run all five problems returned exactly nine characters. Switching to an instruction wrapper makes the score easier to earn.
So the ruler itself is saturated: it measures where the ceiling is, not who is higher. Which in turn says something real — the local model has reached the same tier. Its score sits in the same band as every entry in the published top ten. Not slightly behind: indistinguishable on this benchmark.
MBPP is where the difference still shows: the best local score is 87.0%, while the published frontier band starts at 87.8%. That is a real gap, and the local model is still behind it.
Two prerequisites that let 22GB hold a 27B model
The commonly repeated claim is that a 27B model needs at least 32GB of VRAM. That is not wrong, but it describes the official FP8 and NVFP4 checkpoints — which require a Blackwell-generation card. In practice it means buying a new GPU.
UD-Q4_K_XL compresses the weights to 16.35 GiB. Dynamic quantisation keeps more bits on layers that are sensitive to compression, which beats a flat quantisation at the same file size.
48 of the model's 64 layers use linear attention; only 16 use full attention. Very little KV cache has to be stored, and the linear-attention layers carry a constant-size state that does not grow with context.
A pure-attention model of the same size would exhaust 22GB on its KV cache alone. Fitting 128K of context onto this card is a property of the model design.
| Quantisation | Weights | Context | Verdict |
|---|---|---|---|
| UD-Q4_K_XL (used here) | 16.35 GiB | 128K, 4-bit KV | 21.9 of 22.5 GB measured |
| Q4_K_M | ~17.8 GB | 32 – 64K | workable |
| Q5_K_M | ~19.5 GB | 16 – 32K | tight |
| Q6_K | ~22.5 GB | — | does not fit fully |
| Q8_0 | ~29 GB | — | not viable |
Speculative decoding is free speed here: the model ships its own MTP head, and enabling it yields a draft acceptance rate of 0.86 — a mean of 4.45 tokens per forward pass.
Fix one: replace the official chat template
This is the least glamorous of the three and the most valuable. A GGUF file embeds a chat template that decides how the model reads a multi-turn conversation. The official template has three defects; the first two are parameter problems, the third is fatal.
- reasoning_effort defaults to xhigh, which burns an enormous number of thinking tokens
- effort values are whitelisted, so an OpenAI-style high is rejected outright
- history is re-rendered inconsistently across turns, changing the token prefix and invalidating the entire prompt cache
A cache miss means the model re-reads the whole conversation every time you say anything. This machine processes input at 615 tokens/sec, and community measurement on comparable hardware puts the cost at 163 seconds of waiting before the first token, per turn, at 64K context. After replacing the template, the multi-turn hit rate is 97.4% — by turn five only the newly added content is recomputed.
This single change is worth more than all the parameter tuning combined. It is also the step everyone skips.
Fix two: choose the shell that drives it
A model does not read files, write files, or run tests by itself. You need an agent tool to drive it — the software shell, or harness. This is the largest variable of the three.
I ran a controlled comparison: the same 40 problems, the same instruction, with the harness as the only variable.
| Stack | Pass rate | Median per problem |
|---|---|---|
| Gemma4-26B alone (control) | 97.5% | 7.5 s |
| Gemma4-26B × Crush | 57.5% | 33.0 s |
| Qwen3.8-27B alone (control) | 90.0% | 5.2 s |
| Qwen3.8-27B × Pi | 95.0% | 23.0 s |
The same model: plus 5 points through the right shell, minus 40 through the wrong one. Nothing about the model changed. This is not a capability problem.
So what happened in those 17 failures? I went through every raw record of all 40 runs. In 17 of them, no file was ever written — not written incorrectly, but never written at all. The model's entire reply was:
I need the description of the Python function you'd like me to implement. Please provide it.
The full description was right in front of it. I worked through the possibilities one by one:
| Hypothesis | Test | Verdict |
|---|---|---|
| The process crashed | Exit code 0, empty stderr, clean 14–18s responses | ruled out |
| The prompt was truncated | Context is 262144 — ample capacity | ruled out |
| Those problems were broken | Same failing input replayed 5 times: 1 success, 4 identical refusals | ruled out |
| Environmental flakiness | Median 41s for successful runs vs 17s for failures — failures answer immediately | supports: the model never entered a tool call |
Breaking the failure records apart reveals something more useful than the score. One shell's failures were 100% wrong code; the other's were 100% nothing written. Those are different kinds of failure — wrong code can be fixed, but a model that never acts gives you nothing to fix.
They fail different problems, which means these are shell-specific failure modes rather than difficulty. Choosing a harness is not choosing a nicer interface; it is choosing what fraction of the model's score you actually collect.
Fix three: stop trusting defaults
The simplest of the three, and the easiest to get wrong. Ollama sets the context default to the model's full 262144 tokens, which pushes the KV cache into system memory. Switching to 32768:
| num_ctx | Memory split | Load time | Decode |
|---|---|---|---|
| 262144 (default) | 22% / 78% CPU/GPU | 194.3 s | 59.4 tok/s |
| 32768 (corrected) | 100% GPU | 32.0 s | 95.5 tok/s |
One default value: six times the load time and 38% of the throughput. Note also that the weights were on the GPU the whole time — all 31 of 31 layers. What spilled was KV cache memory, not model layers running on the CPU. Those two claims get conflated constantly.
Three more settings will leave you with something that runs but is not usable:
- KV cache quantisation (-ctk q4_0 -ctv q4_0): without it, a 128K cache simply does not fit
- Flash Attention (-fa on): a prerequisite for quantised KV
- Speculative decoding (--spec-type draft-mtp --spec-draft-n-max 4): depth 4 is the sweet spot — 2 underfeeds the head, 6 and above runs out of VRAM at startup
Is an old card worth it?
An eight-year-old card reached this result on engineering, not hardware. Three things, none of them about the GPU:
Cache hit rate 0% → 97.4%; per-turn waiting time differs by an order of magnitude.
Same model, ±40 points; the right shell adds another 5.
One setting cost six times the load time and 38% of the throughput.
- A newer GPU
- More VRAM
- A cloud API budget
- A correct chat template
- A harness that preserves the prefix cache
- The patience to check every default
On-device models plus correct engineering plus the right shell put an eight-year-old machine in the top tier. Most of the GPU upgrades people have bought in the last few years were really the other three things left undone.
Full disclosure: the local numbers use an instruction-wrapper protocol while the published scores use the official completion protocol, so they are not directly comparable. HumanEval was released in 2021 and is almost certainly in every model's training data. The stack comparison covers 40 problems, giving a 95% confidence interval of roughly ±13 points — so gaps between the good harnesses are not resolvable, while the 40-point collapse is far outside the error band. The complete research report runs to 14 sections and 34 tables, including nine errors I found and corrected in my own work.
Sources & data
