NVIDIA Just Shipped a Chip Built for Latency, Not Speed. The Compute Market Is Splitting Into Two.

On August 24, 2026, NVIDIA announced that Groq 3 LPX, an interactive AI inference accelerator, is now in full production. On its face this is a data center product for AI factories, the kind of announcement that normally glides past everyone who builds AI on their own desk. Read it again and it is more interesting than a new server chip. NVIDIA, the company that won AI on raw throughput, just shipped something whose entire design is a bet on latency, not on speed. That is not a footnote. It is the cleanest confirmation yet that AI compute is splitting into two separate markets.

What NVIDIA actually shipped

The Groq 3 LPX is not a GPU in the familiar sense. It is built around the logic of the Language Processing Unit that Groq pioneered: a chip with a large amount of on-chip SRAM used as primary weight storage rather than as cache, no dependency on slow off-chip memory for the hot path, and deterministic scheduling so latency is predictable step after step. NVIDIA describes a rack-scale build of 256 interconnected LPU accelerators tied into the Vera Rubin NVL72 platform, on the order of 315 petaflops of FP8 compute and 128 gigabytes of on-chip SRAM per rack. The pairing is deliberate. Rubin GPUs, with their high bandwidth memory, hold the large weights. The LPUs handle the low latency token generation that keeps an agent responsive.

The marketing claims are what you expect from a vendor on announcement day. NVIDIA says 4,767 median output tokens per second on its own SPEED-Bench, and 4x faster responsiveness for agents than the nearest alternative platform. Treat those as vendor figures on a vendor benchmark, not as independent law. The architectural direction is the trustworthy part, and it is unambiguous: determinism and low latency are the product, throughput is the supporting cast. NVIDIA calls it an interactive inference accelerator because interactivity, not speed, is the entire point.

The signal bigger than the chip

For the last two years the frontier race has been scored in tokens per second. The subtext of every model release and every accelerator spec was raw throughput. That metric is the right one for one half of the market and the wrong one for the other, and the split is now visible in silicon. Interactive agentic inference is latency bound. A single agent chains hundreds of sequential model calls; what kills the experience is not total tokens per run, it is the time to the next token at each step, the predictability of that time, and time to first token when context is huge. Throughput batch inference is the opposite. It is bandwidth bound. You want the most tokens per dollar and per watt for a queue that runs unattended, and latency barely matters because no one is waiting in the loop.

Those two workloads need different hardware. That is not a controversial engineering opinion anymore, it is the design logic behind a shipping NVIDIA product. When the most throughput-obsessed company in the industry builds an inference accelerator whose only reason to exist is low and deterministic latency, the interactive half of the market has its own category. The market did not coalesce around this because of theory. It happened because anyone running real agents discovered that a GPU tuned for batch throughput is the wrong tool when you are waiting on a model to think one step at a time.

What it means on your own desk

I do not run Groq 3 LPX racks, and neither do you. This is a data center product measured in AI factories, not a desktop card. But the same split is what I feel every day on my own hardware, and this announcement is the industry finally saying the quiet part out loud. On my rack I keep interactive work and background work on different machines deliberately. The RTX Pro 6000 handles the jobs where latency and responsiveness matter, the interactive tasks where I am in the loop. The twin DGX Sparks take the unattended queues, the batch and background agent work where throughput and the ability to run for hours matter more than any single step being fast. A Mac mini sits on top orchestrating the lot.

I stopped asking which single machine is fastest a long time ago, because fastest is not a question that has one answer once the split is real. The question that actually drives a buying decision is which constraint each workload binds on. Is this task latency bound, where first token and step time and determinism decide whether it is usable? Or is it throughput bound, where total tokens per hour and cost per token decide whether it pays for itself? The moment you sort your workloads by that question, your hardware stack sorts itself. You stop buying one machine to do everything and start buying two machines that each do one thing well, because that is now exactly what the two big compute markets look like.

The honest caveats

This split being real does not mean the desktop market flips overnight. Consumer inference is still largely a single-GPU world, though the same forces are creeping in: people already talk about an interactive card and a background card. And you should keep the vendor numbers in perspective. The 4,767 figure and the 4x claim are NVIDIA's measurements on NVIDIA's benchmark, publicized to sell a product line. Believe the architecture, treat the benchmark the way you treat any vendor benchmark, which is to say run your own. The thesis does not rest on whether Groq 3 LPX hits a specific token count. It rests on the direction, and the direction is that an interactive accelerator now exists in full production at the largest inference hardware company in the world.

The takeaway

Stop measuring your inference stack by peak tokens per second and start measuring it by the latency or throughput profile your workload actually binds on. NVIDIA just bet a shipping product on the idea that those are two different markets. When you plan the next machine you run agents on, ask which of the two markets each workload lives in. Buy for the constraint, not for the biggest number. The industry just publicly split compute into interactive and throughput. Your buying decisions should split right along with it.