A place for an idea · Taking shape together

Qwen3.8-27B at 64K on one RTX 3060

An experimental llama.cpp patch and measurements: Qwen3.8-27B generating at 29.65–29.69 tokens/s on a fixed C++ workload (33.21 peak) with a 65,536-token context allocated on one RTX 3060 12 GB. The optimization, Deferred-Commit Factor Replay, removes hundreds of MiB of speculative recurrent-state copies and spends the saved VRAM on model weights. It is a narrow, disclosed result with about 105 MiB of VRAM to spare, not a general default.

This moot is about one narrow result: Qwen3.8-27B generating at about 29.7 tokens per second with a 65,536-token context allocated on a single RTX 3060 12 GB, using an experimental llama.cpp patch.

The optimization is called Deferred-Commit Factor Replay. Speculative decoding normally keeps extra copies of the model’s recurrent state in case guesses are rejected. The patch removes hundreds of MiB of those copies and spends the saved VRAM on model weights.

It is a disclosed, narrow result, not a claim that every prompt runs this fast. The final profile leaves about 105 MiB of VRAM free. The patch, exact settings, caveats and raw measurements are in the repository.

What I’d like from you

If you have a 12 GB card, try to reproduce it and post your numbers, whatever they are: GPU, driver, prompt, accepted and drafted tokens, tokens per second. A report of where it fails is as useful as a confirmation.