A 27B model on two 4GB Maxwell GPUs: inference tuning

This blog post is drafted by Codex, and reviewed and edited by me - a lot of the discoveries are driven through Codex's incessant hard work, and I spent a lot of time learning the intricacies of the inference pipeline. Codex's side chat is an amazing feature, that let me ask all simple and complex questions. Let's go!

This is a silly machine for a 27B model, which is what made it useful.

The model is Prism's Bonsai-27B, a 3.9 GB binary-weight model. The machine has two Quadro K2200s with 4 GB of VRAM apiece. They are Maxwell cards from 2014, compute capability 5.0, each capped at roughly 39.5 W. There are no tensor cores, no DP4A, no GPU-to-GPU peer access, and CUDA graphs are unavailable. PCIe Gen2 x8. The other sits behind the chipset at Gen2 x4.

I spent 17 rounds profiling and changing the inference path anyway. The useful result was not one heroic kernel. It was a set of narrow improvements, each checked on natural prompts and rejected when it damaged the output.

The short version

• The final 1K natural-prose operating point was about 37 prompt tokens/s, 5.4 generated tokens/s, and 51.6 seconds end to end for the test run.
• Increasing the prompt microbatch to 1,024, tuning the Maxwell MMQ tile, and compressing only safe tensor-parallel FFN reductions produced the repeatable general gains.
• A new packed Q1 prefill kernel improved a 256-token engineering prompt by 10.46%, but was slower outside a narrow matrix-shape window. Production enables it only for 192 to 512 prompt rows.
• Full FP16 communication looked better at first: 6.1% faster prefill at 1K. It also generated slash-only garbage in five of eight extra runs. The failure was FP16 range overflow, not ordinary mantissa loss.
• The Q4 DSpark drafter fit only after moving its huge output projection onto the GPUs. It still lost to target-only generation on normal prose and code. N-gram speculation reached 9.70 tok/s on a deliberately repetitive prompt, but that number is a ceiling, not a general result.

Why this machine is awkward

Bonsai-27B is a 64-layer hybrid recurrent and attention model with binary Q1 weights and FP16 scales. Its low-bit storage is why a 27B model can be only 3.9 GB. Storage is not the same thing as working memory, though. Activations, recurrent state, the token embedding, the output head, CUDA allocations, and the draft model all need room too.

The two GPUs add up to 8 GB on paper, but they do not behave like one 8 GB card. Tensor-parallel inference splits each large projection across both devices. Their partial results must then meet again. On this machine that reduction is staged through host memory because the K2200s cannot use peer-to-peer copies. The second card's slower PCIe link makes an uneven split tempting, but it also makes balancing compute, transfers, and VRAM surprisingly fragile.

Maxwell adds another problem. During prompt processing, llama.cpp cannot use its DP4A-based low-bit matrix-multiply path. It expands Q1 weights and asks FP32 SGEMM to do the dense multiply. That is a reasonable fallback, but it throws away part of the reason the weights were packed in the first place.

For background, the relevant hardware behavior is documented in NVIDIA's Maxwell tuning guide. The upstream split modes are described in the llama.cpp multi-GPU notes.

How I measured it

I tracked prompt throughput and decode throughput separately. Prompt throughput is how quickly the server reads the existing context. Decode throughput is how quickly it emits new tokens after that. A change can help one and hurt the other, so a single tokens-per-second number is easy to misuse.

The main tests used natural prose, code, finance, book, and engineering prompts from the NVIDIA SPEED-Bench style set, including prompts around 1,000 tokens. Each candidate ran against a nearby control. I compared emitted token IDs, sampled log probabilities, non-finite values, collapse markers, latency, and temperature. The final transport validation used five paired runs across code, book, and finance prompts. The packed-Q1 validation used six prompts across 128 to 1,025 input tokens.

Runs began only after the GPUs cooled below the configured start threshold. The harness stopped at 90 C. In practice, the accepted final tests peaked at 77 C on GPU 0 and 72 C on GPU 1. Before each run the scripts set persistence mode, maximum supported application clocks, the high-performance power state, and aggressive fan control where the driver exposed it. Afterward they restored the idle policy and unloaded the model if nothing was using it.

I also kept a repeated-history prompt because it is good at exposing the upper bound of speculative decoding. It is not allowed to stand in for normal prose.

First, fit the model and find the bottleneck

The initial target-only path managed roughly 33.4 prompt tok/s and 5.42 decode tok/s. A straightforward layer split was a disaster: 19.40 prompt tok/s, 2.36 decode tok/s, and 107.6 seconds total. The cards were spending too much time handing state across the PCIe boundary.

Tensor splitting was better, but the obvious attempts to favor the faster card did not pay. A 55/45 split reduced prompt throughput by 8.1% and decode by 7.4%. A 48/52 split was closer but still slower. Keeping the output head local to one card also reduced decode throughput by about 3.8%.

The profile made the next move much clearer.

There were 992 large Q1 projection calls with n = 1024 in the profiled prefill. FP32 SGEMM consumed 37.92% of measured GPU time. Q1 matrix-vector kernels consumed another 35.15%. Host-to-device and device-to-host traffic together accounted for 8.79%. The fused recurrent GDN kernel was only 2.28%.

This ruled out a lot of attractive busywork. Faster exponentials, different CUDA wait modes, more CPU polling, thread affinity, launch queue experiments, LTO, and PGO all landed below a 1% acceptance threshold. Some were directionally positive, but none was large enough to distinguish from run noise.

What actually worked

Larger prompt batches and fewer checkpoints

Raising the prompt microbatch to 1,024 and reducing checkpoint overhead moved natural-prose prompt throughput from 32.92 to 34.53 tok/s. Decode rose from 5.26 to 5.36 tok/s and total latency fell from 55.79 to 53.88 seconds. That was a 4.9% prefill gain in the representative run.

This was the least exotic win. The model did more useful matrix work per scheduling boundary, which matters on an old GPU with weak launch and synchronization behavior.

A Maxwell-specific MMQ tile

The best existing low-bit matrix-vector configuration used a Y32 tile. It lifted prompt throughput from 34.53 to 35.25 tok/s and decode from 5.36 to 5.42 tok/s, cutting another 0.87 seconds from the run. Small, but repeatable.

Forcing the generic MMQ route everywhere was not the same thing. The scalar fallback ran at 17.60 prompt tok/s, about half the control. A sign-specific version that extracted bits inside a generic kernel fell to 10.89 tok/s. Packed weights do not guarantee a fast multiply. The storage layout, activation format, tile shape, and instructions all have to agree.

Scaled FP16 transport, only where it was safe

The tensor-parallel path communicates partial activations after projections. The straightforward format was FP32. Sending FP16 halves those bytes, which should help a pair of cards connected only through host-staged PCIe.

It did help. At 1,024 prompt tokens, raw FP16 transport increased prompt throughput by 6.10% and reduced latency by 4.09%. The improvement grew with context length, consistent with a communication-bandwidth effect.

The first quality checks passed, then the longer repeat set failed. A healthy reduction peaked at 6,508.1. A failing one reached 110,677.1, beyond FP16's maximum finite value of 65,504. We recorded 447 finite-to-infinity conversions and more than 21 million cumulative NaN inputs after the first bad reduction. The first overflow appeared at reduction 124.

That distinction matters. The problem was not primarily that FP16 had fewer mantissa bits. The model drove some communicated values outside FP16's numeric range. Once infinity entered the recurrent path, later state became NaN and the output collapsed.

I tried several alternatives:

• Q8 block transport reached a 3.52% prefill gain, but the all-reduction version produced slash-only output.
• A custom FP24 transport reached a median 3.37% gain at 512 and 1,024 tokens, but also collapsed. The standalone codec was accurate; repeated model dynamics amplified the small error.
• Q8 with smaller blocks and odd-layer-only routing sometimes stayed coherent, but the safe case was only 0.71% faster.

The deployable version dynamically scales reductions into FP16 range and applies the compressed path only to the FFN projections that stayed stable. FFN means feed-forward network: the large per-token transformation after attention or recurrent mixing. These projections move a lot of data, so compressing them still helps. Sensitive recurrent and attention reductions remain exact.

Across five final pairs, this selective path improved prompt throughput by a mean 3.23% and reduced latency by 2.24%. All five 64-token outputs matched the controls exactly. The maximum Jensen-Shannon divergence between checked token distributions was 2.94e-4, and there were no non-finite values or collapse markers.

This experiment follows the same broad idea as recent work on communication compression for tensor-parallel inference, Flash Communication, and quantized collective operations for multi-GPU state-space models. The Maxwell implementation is much narrower. It compresses a host-staged transport path and keeps an explicit quality gate.

A packed Q1 prefill kernel with a narrow dispatch window

The largest theoretical opportunity was the FP32 SGEMM fallback. If a kernel could consume Bonsai's Q1 weights directly, it would avoid expanding them to FP32 before every large prefill multiply.

I built a Maxwell-specific packed Q1 by Q8 kernel around the byte-SAD instruction, exposed in CUDA as __vsadu4. Activations are quantized in 32-value blocks and biased by 128. Eight prompt rows share an activation tile. The weight side stays in a bit-plane-friendly packed layout instead of becoming an FP32 matrix. NVIDIA documents the underlying packed SIMD intrinsics in the CUDA math API. LUT-GEMM is useful related work on performing matrix multiplication directly over low-bit weights.

The first result looked good in a small kernel test. Then the production graph disagreed. At the dominant standalone shape, the packed kernel took 83.25 ms while the incumbent expansion plus SGEMM path took 72.98 ms. My earlier microbenchmark had overestimated the cost of the existing expansion. Unbounded integration made 1K code prefill 3.86% slower and 1K prose 4.74% slower, even though the generated tokens matched exactly.

The kernel did win at a smaller shape. On a 256-token engineering prompt, prompt throughput improved by 10.46% and latency fell by 6.98%. The fix was to stop pretending one dispatch was universally better. Production now enables the packed kernel only from 192 to 512 prompt rows and falls back everywhere else.

Across the six-prompt production test, the maximum token-distribution divergence was 9.09e-5 and peak GPU temperature was 75 C. The 128, 513, 1,024, and 1,025-token cases stayed close to zero because they used the established path.

Why the quality gate changed the answer

Low-level inference work is full of benchmarks that complete successfully while the model is already broken. A kernel can return on time, produce finite numbers for a while, and still push a recurrent state into a bad basin several layers later.

Raw FP16, Q8 block transport, and FP24 would all look like wins in a throughput-only report. They are not usable wins. The selective scaled-FP16 path and the bounded packed-Q1 path were slower than their most aggressive variants, but they survived exact-token and distribution checks across different prompt types.

I used exact deterministic output where possible because it is easy to interpret. I also kept log-probability divergence because exact tokens can hide a narrowing safety margin. This is especially important for recurrent models, where a small transport error can persist in state instead of disappearing at the next attention block.

Why speculative decoding did not save the machine

DSpark is a semi-autoregressive drafter that schedules speculation using confidence. It is a more interesting fit for this model than adding a conventional full draft model, and its reported research result is strong on modern hardware. The problem here was memory and verification cost.

The Q4 DSpark drafter has a very large output projection. Leaving that projection on the CPU fit in memory but produced about 4.44 decode tok/s. Moving it to the GPUs with tiled low-bit matrix-vector work raised the result to 5.22 tok/s, but peak allocation reached 3,925 and 3,876 MiB. That was still about 3.6% slower than running Bonsai alone, and prompt processing regressed badly.

On a balanced natural prompt, the target generated at 5.63 tok/s without DSpark and 3.48 tok/s with it. Only 82 of 174 draft tokens were accepted, a 47.1% acceptance rate, and the draft pass itself took 15.67 seconds. The extra proposal and verification work cost more than the accepted tokens saved.

N-gram speculation told a more nuanced story. On natural prose, no speculation delivered 5.26 tok/s, adaptive n-gram 5.35, and always-on n-gram 5.06. On natural code, always-on n-gram fell to 4.42 tok/s. On deliberately repeated history, however, n-gram speculation reached 8.72 tok/s at match length 4 and 9.70 tok/s at match length 7.

This is consistent with the economics of speculative decoding and draft-and-verify methods: acceptance rate is only half the equation. Proposal cost, verification width, memory movement, and target-model shape matter too. Recent work such as DFlash attacks the draft-side memory problem directly. On the K2200s, there is not enough cheap compute or spare VRAM for a general drafter to disappear into the margins.

The failures were more informative than the microbenchmarks

A few ideas were dramatically bad:

• Forced scalar MMQ reduced prompt throughput by 50.08%.
• The generic sign-specific Q1 kernel reduced it by 69.11%.
• Storing dequantized values in FP16 reduced it by 17.98%.
• Explicit pinned staging and mapped host transport reduced full-model prefill by 28.54% and 28.14% respectively.
• A four-chunk FP16 communication pipeline reduced prefill by 5.51%.
• The 55/45 tensor split reduced prefill by 8.09%.

The pinned and mapped-host results are worth dwelling on. Mapped host memory improved an isolated transfer test by 26.22%. In the inference graph it was 28.14% slower. The microbenchmark removed the synchronization, allocation pressure, kernel sequencing, and competing transfers that made the real path expensive.

A byte-SAD Q1 dot product had the same shape of disappointment at a smaller scale. It was about 20% faster as a standalone operation on each GPU, but improved natural-prose prefill by only 0.88%. That fell below the 1% deployment gate.

Other ideas were simply flat: CUDA connection counts of 4 and 16, spin versus yield versus blocking waits, CPU poll intervals, launch queues, GDN fast exponentials, SiLU intrinsics, PGO, LTO, and alternative MMVQ warp counts. Q8 KV storage saved only 19 MiB per card and hurt decode. NCCL initialized but the installed library was a stub, so the server retained its generic butterfly reduction. Disabling flash attention crashed this model. CUDA graphs are an architectural non-starter on these cards.

What I would carry to newer hardware

The exact kernels are Maxwell-specific. The method transfers better than the code.

1. Profile the complete graph. A fast codec, dot product, or transfer is not evidence that inference will be faster. Two of the largest reversals in this work came from good microbenchmarks.
2. Treat communication precision as a per-tensor decision. Compressing every reduction was fast and unsafe. Compressing scaled FFN reductions kept most of the bandwidth win without poisoning recurrent state.
3. Dispatch by shape. The packed Q1 kernel was excellent at 256 rows and worse at 1,024. A bounded fast path is more useful than a universal kernel that wins only in its own benchmark.
4. Measure natural prompts. Repeated data made speculative decoding look almost twice as fast. Prose and code exposed the verification cost.
5. Make quality part of the benchmark. Throughput, exact tokens, distribution drift, non-finite counters, collapse checks, temperature, and end-to-end latency belong in the same result.

On Ampere or newer cards, DP4A-style integer paths, tensor cores, better P2P links, CUDA graphs, and more VRAM change the balance. I would still test selective communication compression and shape-aware dispatch first. The larger absolute bandwidth and compute budget should make a modern packed low-bit kernel easier to justify, while faster collectives may reduce the relative value of host-transport tricks.

Experiments still worth running

I have not tested the following ideas on this machine. They are the next research directions, not results from this post.

• QuaRot uses rotations to reduce activation outliers. That could make lower-precision transport safer, but it requires model-aware integration.
• TEAL removes low-magnitude activations at inference time. Maxwell's sparse execution support is limited, so the question is whether structured packing can turn sparsity into less memory traffic.
• CLaSp explores layer skipping with confidence. A recurrent hybrid model needs careful state validation before this is believable.
• VocabTrim reduces output-layer work by trimming the active vocabulary. Bonsai's enormous output matrix makes that especially relevant.
• SlimSpec targets cheaper speculative decoding. It may be a better direction than forcing a Q4 drafter into the last few megabytes of each K2200.

The current production configuration keeps Bonsai text-only, uses tensor parallelism across both K2200s, runs the larger prompt microbatch and Y32 tile, enables scaled FP16 only on validated FFN reductions, and dispatches packed Q1 prefill only in the 192 to 512-row window. Speculation is off by default.

When no request is active for five minutes, the server unloads the model. That last change adds zero tokens per second. On a machine with two 39.5 W cards, it may be the optimization that matters most.