The Most Important Part of an Inference Release Is the Known Issues Section

NVIDIA published TensorRT LLM release candidate 25 on August 31. The headline feature is a new KV cache manager enabled by default across a long list of current models, including DeepSeek V4, GPT OSS, Kimi K3, Qwen 3.8, Gemma 4, and GLM 5. NVIDIA calls the new manager the recommended architecture and says the old version will be deprecated.

That sounds like the story. It is not the part I would read first before putting an agent workload on it. The more useful section is the known issues list. It describes startup hangs, shutdown hangs, generation that can stall indefinitely, memory exhaustion, crashes, deadlocks, degraded visual output, accuracy loss, unsupported configurations, and combinations of features that can fail only when used together.

This is not an attack on TensorRT LLM. It is a useful, unusually candid map of what production inference actually looks like. The release notes expose a truth that model announcements usually hide: inference reliability is a property of a configuration, not a product name.

Your serving stack is a matrix

A team does not deploy TensorRT LLM in the abstract. It deploys one model, one precision, one GPU architecture, one parallelism strategy, one scheduler, one cache design, and one collection of optimizations. Add speculative decoding, adapters, structured output, or disaggregated serving, and you have changed the system again.

That distinction matters because two teams can say they run the same model on the same engine while operating systems with very different failure modes. One uses a single GPU with a plain decoder. The other uses multiple GPUs, expert parallelism, speculative decoding, cache transfers, and adapter serving. The second stack is not merely a faster version of the first. It is a different product with more boundaries where state can drift or work can stop.

Performance features multiply interactions

The release notes show the pattern clearly. Speculative decoding with chunked prefill can produce incorrect cache behavior and lower token acceptance. A Qwen 3 FP8 adapter workload may hang. Some distributed workloads can finish generation successfully and then crash or hang during shutdown. A sender abort during a disaggregated cache transfer can leave the transfer hanging. These are not simple model support questions. They appear at the intersection of features.

This is why I am skeptical when an inference comparison reduces the decision to tokens per second. Peak speed is measured on a path that completed. An autonomous agent cares about the paths that do not. A server that is ten percent faster but occasionally traps a worker until manual cleanup can deliver less useful work over a day than the slower server that stays boring.

For interactive use, a stalled request is annoying. For a background agent, it can be silent. The job may hold a lease, block a queue, repeat a tool call, or wait forever while the dashboard still shows a running process. Reliability is not separate from throughput. Completed work per day is the throughput that matters.

Default changes deserve migration tests

The KV cache manager change is especially important because it becomes the default for many popular model families. Defaults have enormous leverage. They make the common path easier, but they also change behavior for anyone who upgrades without pinning every relevant option.

A default migration should be treated like a database migration. Record the old configuration. Run the new path against the same prompts and traffic shape. Compare output quality, memory behavior, startup time, cancellation, restart behavior, and recovery after a worker failure. Then keep a documented rollback path until the new version has survived real workload conditions.

The temptation is to test only successful generation because that is easy to automate. The harder tests are the ones that resemble operational reality: cancel a request during a cache transfer, kill a worker during generation, restart repeatedly, fill memory, send long context, and shut down under load. A system is not production ready because it answered one prompt correctly.

Model support is not workload support

Release notes often say a model is supported. Builders tend to read that as permission to move. But support can mean that the basic path loads and generates. It does not guarantee that every quantization, adapter, tool parser, cache mode, GPU generation, or parallel execution plan is equally mature.

The right question is not whether the engine supports Qwen, DeepSeek, or Kimi. The right question is whether it supports your exact workload on your exact hardware with the exact features you plan to enable. If the answer lives across a model table, a release note, open issues, and test output, then that combined evidence is the actual compatibility document.

This also changes how I think about upgrades on local AI machines. A workstation is not a smaller cloud cluster. It often has less redundancy and fewer spare workers. A crash that a large service routes around can stop the only local endpoint an agent has. That makes conservative version pinning more valuable, not less.

Boring versions compound

The newest release can be the right choice when it adds required model support or fixes a problem you already have. It should not be the automatic choice merely because it exists. Release candidates are excellent for qualification. They are not a substitute for qualification.

For a founder running agents, stability has a compounding return. Every clean overnight run produces useful work and trustworthy data. Every unexplained hang creates investigation, retries, duplicate actions, and doubt about whether the next failure came from the model, the agent, or the serving layer. That uncertainty is expensive because it slows every layer above inference.

My recommendation is simple: read inference release notes from the bottom up. Start with known issues, then fixes, then default changes, and only then read the highlights. Build a small compatibility sheet for the configuration you actually operate. Pin the version that passes your failure tests, not the version with the most exciting feature list.

Before your next inference upgrade, run one test that interrupts the happy path. Kill a worker, cancel a long request, or force a restart under load. If the service cannot recover without manual cleanup, the benchmark number was never the deciding metric.