SGLang released version 0.5.20 on September 18 with the usual long list of model support, kernel work, and hardware optimizations. The feature I keep coming back to is much less flashy: a simulator that runs the real scheduler and cache logic on a CPU while replacing model execution
NVIDIA released AIPerf on September 18 as the designated successor to GenAI Perf. The headline is a new inference benchmarking tool. The more important point is less flattering to the benchmark culture around local AI: sometimes the machine generating the test traffic is the bottleneck, not the inference server being
NVIDIA released NemoClaw version 0.0.127 on September 17 with a change that sounds like plumbing but is really an architecture decision. OpenClaw and Hermes now own their native gateway processes, plugins, packages, child processes, hooks, and background work after onboarding. NemoClaw keeps control of the OpenShell sandbox, project
DeepSpeed released version 0.19.7 on September 16. It is labeled a patch release, and the change log looks like the usual dense pile of fixes. Buried inside it are two additions that matter well beyond model training: a prototype for continuous batching generation during rollouts, plus profiling that
A llama.cpp fix merged this morning deserves more attention than another model leaderboard. Pull request 24292 closes a use after free flaw in the RPC backend that the author says was reachable by an unauthenticated remote client. The bug could expose memory addresses and lead to remote code execution.
Colibri is getting attention for a claim that sounds almost absurd: run enormous mixture of experts models on ordinary hardware by treating storage, system memory, and video memory as one hierarchy. The project is written in pure C, its engine has zero dependencies, and it streams the experts a token
A model that needs more memory than your machine has usually produces a boring answer: buy a bigger machine. SGLang merged a more interesting answer this morning. Its new support for the NVIDIA Qwen3.8 Flash Next NVFP4 checkpoint lets a single DGX Spark serve a model package that otherwise
vLLM released version 0.29 this week with 594 commits from 277 contributors. The headline feature is not a new model or one impressive speed number. Model Runner V2 is now the default for all models. FlashInfer collective communication is also enabled by default for CUDA tensor parallel groups, and
Sakana AI released Fugu Max and Fugu Ultra v2 this week. Fugu is not one more general model competing for the top line on a leaderboard. It is a learned orchestrator that can break work apart, choose other models, coordinate their output, and return one answer through an OpenAI compatible
DeepSeek released V4.1 Flash today, and the headline numbers invite the usual model launch argument. It has 552 billion backbone parameters, native image understanding, and a context limit of one million tokens. DeepSeek also published benchmark tables that put it near much more expensive models on several agent tasks.
Hugging Face published Funes last week, an open source memory layer that turns coding agent session traces into a local, searchable dataset. It supports Hermes, Claude Code, Codex, and pi. The technical pieces are useful, but the bigger idea matters more: an agent's memory should belong to the
llama.cpp merged recurrent state rollback support for Kimi K3 on September 8. The code change sounds narrow, but the measurements attached to it expose a broader mistake in how people evaluate speculative decoding. We talk about speculation as a speed feature. On large recurrent models, it is also a