docs/vllm-online-serving.md

vLLM, online serving

The binding comparison. vLLM runs its production graphed config, never --enforce-eager. Every 0.25.0 row below is SUPERSEDED and OPTIMISTIC, on two counts that both flattered us: the harness enforced the 0.25.0 ROLLBACK and raised on the pin until 2026-08-12 (#520 ), and the same runs never passed --language-model-only, so the denominator ran vLLM’s UNFUSED QK-norm+RoPE+gate path (#414 ). Nothing is withdrawn. The first series free of both, at the pin, graphed, and at a pinned clock is in the benchmark record .

THE PIN MOVED UNDER THESE ROWS, AND THEY HAVE NOT BEEN RE-MEASURED. The parity pin advanced to a7c23ac96d on 2026-09-26 (#2817 ). Every row below that says “at the pin” was measured against the PREVIOUS pin 555967922, with FlashInfer 0.6.15.post1. FlashInfer moves to 0.6.18 at the new pin and is on the executed path of these numbers on both sides — it is the NVFP4 GEMM under the denominator and the CUTLASS source tree our own arm compiles against. Four rows here owe a re-measurement: the 27B NVFP4 nvidia and 35B-A3B NVFP4 rows, the Qwen3.8-27B bf16 c4 row, and the cold-to-/health startup row, which is also where nvidia-cutlass-dsl 4.6.0 to 4.6.2 can move a published number. Owed by #2818 . The advance preceded the re-measurement by a developer ruling that inverts the sync cycle’s step order; a red on the re-measurement requires reverting the pin, not withdrawing these rows on their own. Nothing here is withdrawn today.

GDN prefill kernels by GPU#

Qwen3.6-27B by concurrency#

Medians of three interleaved repetitions, 1,024 in / 128 out, cache off, closed loop. Output is token-for-token identical to vLLM at every point. The nvidia ModelOpt table that follows is a DIFFERENT checkpoint on a DIFFERENT axis and must not be compared against this one: a 7-token prompt scored on OUTPUT tokens per second, with method, attribution and the owed 1,024 in / 128 out axis in the benchmark record .

TOTAL tokens per second (prompt plus output), not a decode rate: c1’s 86.05 is about 10 tok/s of decode plus about 940 of prefill. The nvidia table below counts OUTPUT only, hence its c1 of 10.41. We are nominally ahead at all six, but only c1 means anything. Our run-to-run noise band is 0.5% and c2 through c32 land between 0.7% and 1.7%, so treat those five as ties, not as wins. The nine axes that fail in both grids are one tradeoff, not nine problems: our synchronous deterministic forward loses on low-concurrency median decode and TTFT, and wins the corresponding tail and the same metric at higher concurrency (c8 p99 ITL 0.86x, but 1.055x at c16 and 1.078x at c32).

Qwen3.6-27B NVFP4 nvidia @0893e160 by concurrency (ModelOpt)#

| Noise band, measured BEFORE any delta | ±0.03% c1 back-to-back; 0.29-1.85% leg-to-leg with reload, drifting down on BOTH arms so it cancels in the ratio | | | | Startup, cold to /health | 33.38 s vs vLLM 182.41 s = 5.46x faster | | | | Peak host RSS | 21.10 vs 13.09 GiB = 1.612x, BELOW FLOOR, open gap | | | | Peak GPU memory | PENDING: nvidia-smi returns N/A on this unified-memory part | | | | Step attribution (nsys, node-level, both arms same tool) | ours 98.906 vs vLLM 81.577 ms/step, 99.2/99.3% GPU-busy; lm_head 8.6414 + fp8 tower 7.6068 + splitK 0.0532 + other 1.0279 = 17.3292 vs measured 17.3292 | | | | Lever 1, lm_head | ships U8/NVFP4 (0.666 GiB), we read 2.368 GiB BF16: +1.702 GiB/step, 11.183 ms. Marlin efficiency is EQUAL (207.9 vs 210.0 GiB/s), only bytes differ | | | | Lever 2, GDN fp8 in_proj | identical 6.7188 GiB/step both arms; ours 96 GEMMs at 165.9 GiB/s vs vLLM 48 merged qkvz at 204.3; in_proj_qkv at 129.3 vs 213.6 GiB/s | | | | Lever 2b, packed GDN decode (#365) | separate c1 in16/out256 harness: 0.977x -> 0.984x, INDICATIVE not binding (arms not interleaved); see the record | | | | | | | OPEN: host-memory-state sensitivity | same binaries read c1 0.7604 pre-reboot vs 0.8289 post; vLLM barely moved. Protocols also differed, variables not separated | | | | Spread, ours / vLLM | 1.000 / 1.006 | 1.009 / 1.069 | 1.001 / 1.001 | 1.005 / 1.003 | | Method | medians of 3, warm servers, one flock, greedy, ignore_eos so both emit exactly 128 tokens, --gpu-memory-utilization 0.55 --max-model-len 4096, vLLM in its production graphed config | | | | | Tokens | greedy continuation IDENTICAL between engines, captured from the same warm processes as these numbers | | | | | Reading | still a real loss, not a tie, and still FLAT across the sweep, so batching and the scheduler are not the cause | | | | | Roof | 20.42 GiB over about 273 GB/s: vLLM’s 79 ms/token is about 95% of the bandwidth limit, ours 96 ms/token about 78% | | | | | Peak host RSS | 21.0 GiB, down from 24.2 GiB, because the FP8 tower is no longer expanded to BF16 | | | |

NVFP4 lm_head kept packed (PERF-27B-LMHEAD-FP4, #213)

Qwen3.6-35B-A3B by concurrency#

There is no isolated c2/c8 weakness. Binding grid at a0fa12c7 (2026-08-10), 3 reps, binding-eligible 12/12; the prior c2 0.87x / c8 0.92x “weak cells” came from a different harness and were never comparable. The deficit is a flat mid-band (c2 to c16 within 0.935x-0.949x, CoV under 0.81% on both engines) and is entirely MARGINAL per-token work, since our FIXED per-step cost already beats vLLM (9.02 ms against 9.43 ms). Two glue levers land against it, both default ON: the fused shared-expert gate_up sink off the MoE-marlin route (+1.31% c8 / +1.38% c4, VT_MARLIN_DENSE_PAIR) and the shared down_proj emitted as bf16 rather than f32 (+2.05% c8 / +0.79% c4, bit-identical, VT_SHARED_DOWN_BF16).

The SiLU lever is NEGATIVE (spec ): the 9.2x per-launch gap that motivated it was a MEAN over a bimodal kernel (min 1.34 us, max 979 us) and our decode-phase SiLU is already faster than vLLM’s, so the remaining band is unattributed. The marlin block-size lever is REFUTED (block 8 at c8 is 1.16% SLOWER against a +0.29% control), and a per-launch marlin gap is WITHDRAWN as cross-tool uncertainty. Memory passes decisively: peak PSS 3.81x, peak GPU 1.40x. Detail in .agents/benchmark-record.md.

Device-resident sampled tokens on integrated (VT_ASYNC_DEVICE_MIRROR) A/B’d 2026-08-06, speed-NEUTRAL (same-binary): c16 OFF median 2305.8 vs ON 2303.3 (0.999x), c32 2928.9 vs 2919.1 (0.997x), bands overlap. It is a drain MOVE (relocate the drain past the host prep, not remove it), so it overlaps only the small host prep; the drain still serializes GPU input staging, so c16 does not recover. The drain REMOVAL, VT_ASYNC_EXECUTOR Option A (row/SERVE-ASYNC-OPTION-A, default-OFF, input H2D staged out of capture, the faithful vLLM structure), is GREEN + RED-reproducing but binding-A/B speed-NEUTRAL (c8/c16/c32 +0.0-0.1%), so the c16 gap is NOT the async input path. Detail in .agents/benchmark-record.md.

But the mirror FIXES a shipping correctness bug, so it is now DEFAULT ON (ROW-SERVE-ASYNC-LLM, 2026-08-06). Baseline async (AsyncLLM depth-2) batch-1 greedy decode degenerated into nondeterministic token-0 garbage; the mirror is deterministic and coherent. The missing gate now exists, test_qwen36_async_serving (depth-2 AsyncLLM, batch-1 + concurrency, token-exact vs the SACRED oracle): RED on =0, GREEN on the default. c16 re-checked on the default: 2312.9/2303.9/2294.4 (median 2303.9), c32 2942.7 (no regression). Root cause + file:line in the benchmark record. The same P0 hit classic dense Qwen3ForCausalLM (quant-independent), fixed by ROW-SERVE-ASYNC-DENSE-MIRROR (see the MXFP4 Qwen3-8B row). The intake-drain lever likewise measured NEUTRAL (2026-08-06, VT_INTAKE_DRAIN A/B 3+3 reps): admitting during the forward wait collapses intake -91% but shifts it into queued, arrival-to-scheduled invariant, so the recorded INTAKE term is an attribution boundary over a GPU-bound prefill wait, not reducible; lever reverted, byte-exact VT_LOOP_TRACE probe kept.

Qwen3.8-27B (bf16) by concurrency#

Qwen3.8-27B quantized arms, both gates PENDING and no number quoted#

DeepSeek-V2-Lite (MLA)#

Medians of 3 reps, 1,024 in / 128 out. The vLLM arm runs --moe-backend triton, which is its best stable graphed config on GB10: the auto-selected FlashInfer CUTLASS MoE backend hard-rebooted the box five times. The substitution does not flatter us, we lose against it.

Peak memory is the decisive win: 31.38 GiB against vLLM’s 68.5 GiB, with the caveat that vLLM pre-reserves a fixed fraction up front while we allocate the KV blocks the workload needs. Real difference in operating footprint, not evidence of a lower per-token KV cost.

Laguna-S-2.1 (NVFP4)#

poolside/Laguna-S-2.1-NVFP4, 118B total / ~8B active MoE, 48 layers, 256 experts, ~67 GiB. Both arms NVFP4, single request, batch 1, GB10. The local checkpoint directory is named laguna-xs-nvfp4, which is where the “Laguna-XS” label in earlier revisions of this page came from; the measured model is S-2.1.

The gap is CLOSED (2026-08-04, same-tool nsys graph-node tracing on both engines). Root cause: the bf16 M=1 projection GEMVs (o_proj, qkv, router, dense, lm_head) read their weights from GB10 UNIFIED/ATS host memory (a w.View() device retag, no cudaMalloc staging), which the GPU reads slower than true device memory. VT_LAGUNA_RESIDENT_BF16W (default-ON parity enabler) stages every projection device-resident (one H2D copy at load); byte-exact (ids bit-identical to the =0 retag arm). Binding new-default decode is 44.46 tok/s (median-of-3), 1.03x vs vLLM’s graphed 43.10.

Per-call the residency recovers o_proj 194 to 131 us (about 168 to 249 GB/s), qkv 245 to 225 us, and lm_head 2410 to 1620 us/call. This supersedes the earlier invocation / bf16-output-cublasGemmEx framing (measured a wash): the invocation was never the cause, the weight’s memory RESIDENCY was. Full forensics in .agents/benchmark-record.md.

Memory: the roughly 2.6 GB of bf16 device copies sit in the 119 GB GB10 unified pool; peak host RSS is unchanged at about 40.7 GiB (device/unified allocations are not counted in ru_maxrss), so there is no memory regression.