NVIDIA AIPerf Exposes the Benchmark Bottleneck Nobody Measures
NVIDIA released AIPerf on September 18 as the designated successor to GenAI Perf. The headline is a new inference benchmarking tool. The more important point is less flattering to the benchmark culture around local AI: sometimes the machine generating the test traffic is the bottleneck, not the inference server being tested.
That sounds obvious once someone says it. It is still easy to miss. A quick Python script can send enough requests to make a chart, but that does not mean it can keep a fast GPU busy. At higher concurrency, a single process load generator can hit Python limits, spend too much time recording results, or produce traffic less quickly than the server can answer it. The result looks like a server ceiling. It may actually be a client ceiling.
A faster GPU makes bad measurement easier
I run local inference across an RTX PRO 6000 Blackwell workstation and dual DGX Sparks. That hardware makes the measurement problem more important, not less. When a model server is slow, almost any load script can expose it. When the server gets faster, the rest of the test setup has to keep up. Eventually the benchmark harness becomes another component that needs profiling.
NVIDIA says AIPerf is a clean architectural break from GenAI Perf. Worker processes generate load, separate record processor services handle results, and ZMQ coordinates the pieces. The stated goal is to stop the client from becoming the limit under real concurrency or request rate. This is not a glamorous optimization, but it changes whether a throughput number describes the inference server or the laptop running the benchmark command.
I have become skeptical of benchmark results that name the model, quantization, GPU, and token lengths but say nothing about the load generator. Those details describe only one side of the experiment. If the client cannot saturate the server, the published maximum is just the highest number that a particular test script managed to request.
Traffic shape matters as much as traffic volume
AIPerf also supports constant, Poisson, and gamma arrival patterns, along with adjustable burstiness and gradual load ramps. That matters because production traffic does not arrive as a neat procession of identical prompts. Agents are even messier. One tool call can return immediately, another can inject a large document, and several background jobs can wake up together. Average requests per second hides the queue that users and agents actually experience.
NVIDIA's walkthrough shows the difference with a static workload and a Poisson workload. The static run pins input and output lengths. The Poisson run varies arrival times and token lengths. As expected, the second run produces a wider spread in time to first token because requests compete for GPU access while prefill and decode overlap. That wider spread is not noise to clean up. It is often the result that resembles the system people will use.
This is why a single tokens per second number tells me very little about an agent machine. Interactive use cares about time to first token and inter token latency. Background agents care more about aggregate throughput and tail behavior when several jobs overlap. A system can look excellent at one active request and fall apart when bursty prompts compete for memory and scheduling attention.
The tail is where architecture shows up
AIPerf reports time to first token, inter token latency, full request latency, and output token throughput. It also reports percentile distributions through the ninety ninth percentile. That last part is where architecture decisions become visible. A healthy average can coexist with a miserable tail, especially when long prompts arrive beside short interactive requests.
For a local setup, this can expose whether interactive and background workloads should share one server. If the median remains good but the tail jumps whenever an agent batch arrives, buying a faster GPU may not be the cleanest answer. Separating workload classes, changing queue policy, or routing background work to another machine may matter more. That is the reason I separate interactive inference from background agents in the first place. The architecture is trying to protect responsiveness, not win a peak throughput contest.
AIPerf can also collect GPU power, utilization, and memory data when DCGM or pynvml is available. Correlating a latency spike with memory pressure is more useful than staring at utilization in a separate window and guessing whether the timestamps line up. Again, the value is not another number. It is a test where the numbers describe the same event.
Reproducibility needs more than a command
The tool supports more than fifteen endpoint types, public datasets such as ShareGPT, production trace replay, fixed random seeds, and exported CSV and JSON results. Those features make runs easier to compare, but they do not automatically make a benchmark honest. A reproducible synthetic workload can still be irrelevant to the application. A production trace can still omit the failure condition that matters. A powerful harness does not choose the right experiment for us.
Still, this release raises the minimum standard. A credible inference result should identify the server, model, precision, input and output distributions, concurrency, arrival pattern, warm up behavior, client hardware, and evidence that the load generator had room left. Without that last piece, a chart may be measuring the wrong computer.
The local AI conversation spends a lot of time debating models and GPUs. The quiet lesson from AIPerf is that measurement infrastructure is part of the inference stack too. Before I trust the next impressive throughput chart, I want one answer: what proved the load generator was not the slowest part of the test?