Speculative Decoding Is Not a Speed Switch: Measure Acceptance Before You Call It Faster

Speculative decoding sounds like the rare inference optimization with a free lunch. Let a smaller component guess several future tokens, ask the target model to verify them in one pass, and commit the guesses that survive. The target model still controls the output, but generation can advance by several tokens instead of one. That is the clean explanation. The operational reality is less convenient: speculative decoding is not a speed switch. It is a workload specific bet whose return depends on acceptance, drafting cost, verification cost, and the behavior of the exact model you serve.

The vLLM team published a useful reminder on August 23. Its tests covered five drafting approaches across Gemma, Qwen, Kimi, and MiniMax models on AMD Instinct MI300X and MI355X GPUs. Several combinations exceeded twice the ordinary autoregressive throughput. The strongest cited examples included 2.87 times baseline for DFlash on Gemma 4 26B A4B, 2.83 times for Gemma 4 MTP on the same target, and 2.68 times for DFlash on Kimi K2.5. Other settings produced smaller gains, and some fell below baseline. The headline is not that speculative decoding is fast. The headline is that its speed is conditional.

The draft is overhead until the target accepts it

Every speculative round adds work. A draft component proposes candidate tokens. The target verifies those candidates from left to right. Once it rejects a candidate, later candidates in that proposal are discarded. If enough tokens survive, the saved target model steps outweigh drafting and verification. If too few survive, you paid for guesses that did not move generation forward. This is why a high acceptance rate is useful but not sufficient. A cheap drafter can deliver a throughput gain with modest acceptance. An expensive drafter can lose even when its guesses look impressive. The final judge is measured end to end throughput on the workload you actually run.

Proposal length is a tuning variable, not a trophy

It is tempting to increase the number of proposed tokens and assume more guesses create more speed. The vLLM results show why that instinct fails. Longer proposals create more chances to commit several tokens, but later positions are often accepted less frequently. Those weak later guesses still consume draft and verification work. Throughput can rise, flatten, then regress as proposal length grows. In the published tests, the best setting changed across model families and datasets. DFlash often performed well around seven proposed tokens, but that was not a universal optimum. Native MTP peaks varied even between related Qwen models. A copied configuration is a starting point, not evidence.

Agent workloads make the average more dangerous

I run local models through vLLM for agent work, and agent traffic is not one uniform stream of prose. A single session can move through planning, code, structured arguments, tool results, short acknowledgments, and long summaries. Each phase has a different level of predictability. Repeated code patterns may be easy to draft. An unusual tool result or an identifier copied from a live system may break the proposal immediately. A benchmark average can hide this mixture. It can also hide the metric that matters. An interactive assistant may care about latency for one request. A background agent pool may care about total completed jobs. A shared server may care about throughput under concurrency. One speculative setting will not necessarily improve all three.

The method changes the cost structure

The new vLLM analysis covers native MTP, Gemma 4 MTP, EAGLE 3, DFlash, and DSpark. Native MTP uses an auxiliary path built into the target architecture. EAGLE 3 drafts sequentially from target model features. DFlash predicts a block in parallel. DSpark combines parallel drafting with lightweight sequential correction. These are not interchangeable labels on the same mechanism. Sequential drafting can preserve dependence between proposed tokens, but each additional proposal adds sequential work. Parallel drafting reduces that loop, but later positions do not see earlier sampled outputs in the same way. Separate draft checkpoints also need memory headroom. The right method depends on the target model, available checkpoint, memory budget, and traffic shape.

Measure the rejection curve, not just tokens per second

A useful test records ordinary throughput first, then adds one supported speculative configuration and sweeps proposal length. Track total output throughput, mean accepted length, overall acceptance, and acceptance by position. The last metric is especially revealing. If positions one through three survive often while positions four through seven rarely do, the longer proposal is probably buying work rather than speed. Run the same sweep on distinct traffic classes instead of one blended prompt set. Code generation, tool calling, structured output, and general chat deserve separate rows. Keep quality checks beside performance checks even though target verification preserves the target distribution in principle. Deployment details still fail in ordinary ways.

Hardware portability is not configuration portability

The published numbers came from AMD Instinct hardware using ROCm. They are evidence that these methods can deliver material gains, not a promise for an NVIDIA workstation or any other server. Kernel paths, graph execution, memory pressure, tensor parallel layout, concurrency, and model support all change the result. On my RTX PRO 6000 setup, I would treat the AMD findings as a test plan, not a projected benchmark. The first question is whether the exact target and draft pair is supported. The second is whether both fit with enough headroom. Only then does speed testing begin.

Speculative decoding matters because decode is still a serial bottleneck, and the upside can be real. But turning it on without measuring acceptance is cargo cult optimization. Start with your current vLLM baseline. Choose one representative agent workload and one supported method. Sweep proposal length, inspect where acceptance collapses, and keep the setting only if completed work per unit of time improves. The recommendation is simple: never promote a speculative configuration because its draft is clever. Promote it because your production workload finishes faster.