A serving metric worth arguing about
A CNCF post published on July 20 by Graziano Casto of Akamas makes a pointed case to anyone running large language models in production: the throughput number everyone quotes is the wrong headline. The argument centers on goodput, which Casto defines as "the number of completed requests per second that also meet your latency targets." Raw throughput counts every completed request regardless of how long users waited, and Casto's verdict on that framing is blunt: "A throughput number with no SLO next to it is marketing, not engineering."
For platform teams that have inherited LLM serving as a new responsibility, this reframing matters because it changes what faster means. A serving stack tuned to maximize requests per second can post impressive dashboards and still leave a meaningful share of users waiting through unacceptable delays. Goodput folds the latency SLO directly into the capacity metric, so the number a team reports upward reflects service quality. That alignment is what makes goodput useful as a planning input instead of a vanity figure.
The experiment that exposes the gap
Casto grounds the argument in a concrete benchmark. Across a sweep of serving configurations, one experiment pushed combined throughput 50 percent higher than the baseline. On a naive reading, that configuration is the clear winner. The catch appears when you look at the latency distribution underneath the aggregate, which is exactly where throughput-first tuning tends to hide its costs. The higher-throughput configuration was seating more requests, and the price landed on the users already in the queue.
The vivid version of the finding comes from Casto's kitchen analogy: "Constrain how fast the kitchen seats you and brings the first course, and the wait gets pushed onto the gap between the courses that follow instead." Translated to a serving stack, packing more concurrent requests onto a GPU can keep the first response quick while stretching the time between subsequent tokens. The aggregate throughput climbs, and the per-user streaming experience degrades. A single top-line number cannot capture that trade, which is the whole point of the piece.
Why the averages lie
The measured numbers make the effect concrete. Average time to first token barely moved between the baseline and the high-throughput run, sitting around 1,395 milliseconds in the first experiment and 1,403 in the twentieth. A team watching first-token latency alone would conclude nothing had changed. The damage was entirely in decode speed: p95 time per output token rose from roughly 50 milliseconds to about 494 milliseconds, close to a tenfold increase, even as first-token latency held steady.
That split between first-token and per-token latency is the crux for anyone tuning a serving fleet. Time to first token governs how responsive the interface feels at the start, and time per output token governs how quickly the rest of a streamed answer arrives. Optimizing for throughput can hold the first metric steady and still push the second far past its target, which no average will reveal. Casto's data is a reminder to track tail latency on both dimensions, because the p95 on decode is where a good demo turns into a bad production experience.
The toolchain behind the measurement
The setup is reproducible with open tooling, which is part of why the piece is useful to practitioners. Serving ran on vLLM, load was generated with GuideLLM, and the metrics were collected through Prometheus and Grafana with the NVIDIA DCGM exporter supplying GPU telemetry. The hardware was deliberately modest: a single NVIDIA A10G with 24 GB of memory on an EKS cluster, running the Qwen2.5-7B model. Nothing in the stack is exotic, so a platform team can stand up the same measurement harness against its own models.
The choice of a single mid-tier GPU is a feature of the demonstration. It shows that the goodput-versus-throughput gap appears on commodity hardware and does not require a large multi-GPU node to reproduce. For teams evaluating serving configurations, the same combination of a load generator, a Prometheus scrape of vLLM metrics, and DCGM GPU stats is enough to compute goodput for their own latency SLO. The measurement discipline is portable, and that portability is what turns the argument into something actionable.
What goodput means for capacity planning
The practical consequence lands on capacity and cost decisions. If a team sizes its GPU fleet on throughput, it will systematically overestimate how much useful traffic each card can carry, because some of that throughput violates the latency SLO and never counts as served. Sizing on goodput yields a smaller, honest number: the traffic the fleet can actually serve within the promise made to users. That honest number is the one that belongs in a capacity model and a cost-per-request estimate.
We see this as directly relevant to the build-versus-buy math around inference. When a vendor or an internal benchmark quotes tokens per second, the immediate question should be which latency SLO that figure respects. Two stacks with identical throughput can deliver very different goodput once a p95 decode target is applied. For leaders comparing serving platforms or negotiating GPU budgets, insisting on goodput at a stated SLO is the discipline that keeps a purchasing decision grounded in the experience users will receive.
Wiring goodput into your SLOs
Adopting the metric starts with defining the latency targets that goodput is measured against. A team needs an explicit SLO for time to first token and a separate one for time per output token, because Casto's data shows the two move independently. With those targets fixed, goodput becomes the count of requests per second that satisfy both, and it can be dashboarded alongside raw throughput to expose the gap between them. The two numbers side by side tell the real story of a serving configuration.
The larger point for platform engineering is that LLM serving deserves the same SLO rigor already applied to web services. Casto's piece supplies a clear definition, a reproducible harness, and evidence that the popular metric misleads. For teams standing up internal inference platforms, encoding goodput into the service's SLOs and capacity models is a small amount of work that prevents a large class of bad decisions. Goodput is how a team keeps score of the diners, and that is the number worth optimizing.
