vLLM 0.28 Turns KV Cache Into a Storage Hierarchy. That Changes How I Think About Local AI Memory

vLLM released version 0.28 this morning with 584 commits from 270 contributors. The headline items cover Kimi K3, DeepSeek V4, speculative decoding, a newer model runner, and a Rust front end. The feature I care about most is quieter: tiered KV cache offloading now includes disk. That sounds like a memory management detail. For anyone building local agents, it changes the architecture question from how much GPU memory do I have to which parts of model state deserve the fastest memory at this moment.

I run local inference through vLLM on a workstation with an RTX PRO 6000 Blackwell and 96 GB of VRAM. I also use two DGX Sparks for background work, with a Mac mini orchestrating the systems. That setup has made one constraint obvious. Model weights are only one claimant on memory. Long prompts, concurrent requests, and agent sessions create KV cache pressure that can become the real limit even when the model itself fits. Buying enough VRAM to load a model does not mean you bought enough memory to serve the workload you designed.

What version 0.28 actually adds

According to the vLLM release notes, the new tiered KV cache work adds disk offloading, external secondary tier managers loaded through a module path, partial results from a secondary tier, metrics for tiering, and a standard CPU layout that does not depend on the parallelism strategy. In plain English, vLLM is treating KV cache less like a pile that must live in one place and more like data moving through a storage hierarchy.

The fastest tier is still GPU memory. CPU memory can hold cache that does not need immediate access. Disk adds a slower but much larger tier for state that would otherwise be discarded or force the server to reject more work. The external manager interface matters too. It suggests this is not a single fixed offload trick. Operators can build policies around their own hardware and workload rather than accepting one universal eviction rule.

This does not make disk fast, and it does not turn a small GPU into a large one. A cache miss that travels to storage still has a latency cost. The important change is that capacity and speed can now be managed separately. Before this, the practical answer to KV pressure was often to shorten context, reduce concurrency, recompute state, or buy more VRAM. Those choices remain available, but they are no longer the entire menu.

Local agents make this more useful than chat does

A chat session usually has one person waiting and one active context. An autonomous agent can keep many contexts alive, revisit old tasks, branch into tool calls, and run while nobody is watching. That workload values reuse differently. An interactive request may justify keeping its cache in VRAM because every pause is visible. A background research task can tolerate a slower restore if the alternative is throwing away useful state and processing the prompt again.

This is the same separation I already use at the machine level. Interactive work belongs on the system optimized for responsiveness. Background work belongs where total capacity and availability matter more. Tiered KV cache extends that idea inside the inference server. Not every active context deserves identical treatment, just as not every agent belongs on the fastest GPU.

The founder lesson is easy to miss. Local AI economics are usually discussed as hardware cost against API cost. Memory policy belongs in that calculation. If better cache placement lets one machine carry more useful sessions, preserve more context, or avoid repeated prompt processing, software changes the effective capacity of hardware you already own. The result is not free performance. It is a better allocation of scarce performance.

The release also shows where inference is heading

The rest of version 0.28 reinforces the point. The release notes say shared expert sharding for Kimi K3 can save about 17 GiB per GPU in the project configuration. They also report an adaptive speculative token budget that improved time to first token on DGX Spark by about 60 percent in their test. These are project figures, not measurements from my systems, and I would not use them as buying guidance without reproducing them. But both optimizations attack resource placement rather than merely asking the chip to run harder.

vLLM also raised the default maximum number of batched tokens from 8192 to 16384 and raised the default Blackwell CUDA graph capture size to 1024. Defaults shape real deployments because many operators never tune them. A release that changes memory tiers, batching, graph capture, and model specific sharding at once is telling us that inference efficiency now comes from coordinating the whole memory and scheduling system.

That matters for model portability. A model agnostic stack is not just an API that lets you swap model names. It needs resource policies that survive the swap. Dense models, mixture models, long context agents, and multimodal workloads put pressure on different parts of the system. If cache placement and offload policies are modular, the infrastructure can absorb more of that variation without forcing the application to be rebuilt around each new model.

What I would test before enabling disk offload

I have not tested version 0.28 yet, so disk offloading is a proposed experiment, not a recommendation based on my own benchmark. I would compare three configurations on the same agent trace: GPU cache only, GPU plus CPU, and GPU plus CPU plus fast local storage. I would record time to first token, time per output token, prompt processing avoided, cache hit rate by tier, concurrent sessions completed, and tail latency. Average speed alone would hide the tradeoff.

I would also separate interactive and background queues. If disk restores make a person wait, the policy is wrong for that queue. If they let an overnight agent preserve useful state and complete more work, the added latency may be a good trade. The decision should be based on the cost of recomputation and eviction, not on whether an offload feature exists.

The useful takeaway from vLLM 0.28 is not that storage replaces VRAM. It is that local inference memory is becoming a hierarchy, and agent workloads need an explicit policy for it. Upgrade in a test environment first. Then replay a real agent trace and measure each tier separately. If you cannot see where the cache went and what the move cost, you are not ready to let the server make that decision for you.