docs/qwen38-27b-exl3-variadic-gb10.md

qwen38-27b-exl3-variadic-gb10 — Qwen3.8-27B EXL3 under a mixed-length serving load

Their engine and ours, on one GB10 board, behind one client, over a prompt-length distribution that spans short questions to long code reviews, swept over concurrency, reported as percentiles.

Its method is variadic-load-methodology . Read that before you quote anything here. Its predecessor is qwen38-27b-exl3-gb10 , which measured one prompt band at concurrency 1 and published means; nothing on this page supersedes its ratios, because the two pages measure different loads.

Disposition#

Measured. Between 5 and 10 September 2026 one client drove both engines on one GB10 board over a mixed prompt-length corpus at concurrency 1, 4, 8, 16 and 32. Every cell has two rounds, in different boots, and the spread is published beside every value.

Three results, warm only, warmup discarded:

• Their aggregate throughput does not move with concurrency and ours does. 33.12, 33.05, 32.84, 32.88, 31.95 output tok/s at c = 1, 4, 8, 16 and 32 against our 35.81, 53.17, 53.93, 53.53, 50.62. Thirty-two times the offered load buys them nothing and buys us 41% over our single-stream rate. At c = 16 and 32 the ratio is 1.63x and 1.58x.
• We prefill long prompts about half as fast as they do, and the gap widens with prompt length. This is the finding the predecessor’s 93-to-457-token workload could not produce. The gap is a c = 1 phenomenon; at c = 16 and 32 it is buried under their queue time.
• Both engines now report acceptance, for the first time in this comparison: 0.789 to 0.799 accepted draft tokens per output token on our side against 0.773 to 0.778 on theirs.

These legs ran the SHIPPED paged draft route. Every previously published number on the predecessor page ran VT_DFLASH_PAGED=0, because that route faulted eagerly until 42b309508 closed #2274 . This is the first measurement of the default configuration, so no number here is directly comparable to a number on that page — a different code path served it.

Two things that are true of every number here#

No correctness gate covers this run, and none can. Both engines sample at T = 0.6, so the two token streams are not expected to match and a token-exact comparison cannot run on them. Nothing scores the text either engine writes.

The comparator serializes generation. tools/serve_openai.py:45 creates gen_lock and line 315 holds it around the entire generator.iterate() loop. Their own docstring at line 18 says “requests are serialized (batch-1 draft); concurrent callers queue.” Above concurrency 1 their figures measure queueing and ours measure batching, and the flat throughput row above is that lock. A ratio at c = 8 is a serving result and not a kernel result.

Subject#

G-BYTES PASS: all three checkpoint shard sha256 values were recomputed on the device and matched the pins (7b77214f…, 411c83bb…, 6b2e3afc…).

G-RESOLVED PASS: the server reported max_num_seqs=8 resolved against a 2048-block KV pool on the c = 1-8 boots, and max_num_seqs=32 resolved against an 8192-block pool on the c = 16-32 boots, so no leg ran below its rung.

The startup line also reports a term gpu_memory_utilization does not bound: 1282.5 MiB of draft speculative context at max_num_seqs 8 (5130 MiB at max_num_seqs 32), 160.312 MiB per concurrent request. That is #2993 . This run pinned --num-blocks explicitly so the ladder would not measure it as noise.

The load#

Four length bands from three corpora, each pinned by sha256:

M is the predecessor’s whole workload, kept so the two pages share one band.

The corpus is built by benchmarks/variadic/build_corpus.py from those bytes, that seed and those weights, and nothing else. The realised token-length histogram is read back from each server’s own usage.prompt_tokens.

Matched on both sides: the box, the client, the corpus, max_tokens: 192, temperature: 0.6, top_p: 0.95, top_k: 20, seed: 0, the chat endpoint, each model’s own chat template, thinking enabled, and no ignore_eos.

Concurrency rungs 1, 4, 8, 16 and 32. Two rounds. Round 2 reverses both the arm order and the rung order, so a monotone drift over the session appears as a rung-to-rung disagreement between the rounds rather than as a shape in the sweep.

The c = 1-8 legs used a 144-prompt corpus (sha256 6d6420fc...); the c = 16-32 legs used a 192-prompt superset (sha256 b52ef36c...). The first 144 prompts are identical (deterministic seed), so the measured 128 requests are the same prompts at every rung below 32. At c = 32 the warmup is 32, so 16 prompts that were warmup at lower rungs enter the measured window; they are from the same bands and the distribution does not shift.

Warmup is max(C, 4) requests, run to completion before the measured window opens, and recorded rather than discarded, so both views come from one set of records.

The percentile method, applied to the predecessor’s own records#

benchmarks/variadic/adapt_headtohead.py converts the four legs of qwen38-27b-exl3-gb10 into the record shape report.py reads. The field definitions already agreed: ttft is the wall time to the first content-carrying chunk, itl is a gap between streamed chunks, latency is client-side, and the token counts come from each server’s usage. Request i = 0 is tagged as warmup, which is that page’s own stated rule and the one its head-to-head legs did not apply.

The percentile tables that came out of it are published on that page, under time to first token , and they are not repeated here. In short: its mean-TTFT conclusion held at the median and reversed above p90, and one request out of 164 was most of the difference.

Two readings from the same conversion are not on that page.

Inter-token latency measures the same thing on both sides once the chunking is removed#

Their p50 inter-token latency is 0.0 ms. That is not a fast engine; it is a server wrapper that holds back 16 characters (tools/serve_openai.py:74) and then releases several chunks with no gap between them. Ours is 82.2 ms because one chunk carries 4.47 tokens.

Divide each engine’s mean chunk gap by its own tokens per chunk and both land on that engine’s mean time per output token, 18.4 against 18.6 and 22.3 against 22.3. That is why this harness reports time per output token as the primary inter-token axis and prints the chunk statistics beside the raw gaps.

A bound on what the chunking costs our time to first token#

The harness’s own correction needs the first chunk’s character count, which the predecessor’s client did not record, so report.py prints that column as absent for these legs. A weaker correction is available from what the files hold, and it is a bound, not the harness’s estimate: subtract (mean tokens per chunk - 1) x mean time per output token, which is 64.5 ms from ours and 1.7 ms from theirs. It assumes the first chunk carries that engine’s average number of tokens, which the harness does not assume.

Chunking accounts for about two thirds of their median lead and none of our tail lead. Their median advantage survives the correction, at 5% rather than the 12% the raw figures show.

The bound does not neutralise their hold-back, and should not be read as doing so. Their wrapper emits pending[:len(pending) - HOLD_BACK], so its first chunk contains the text minus the 16 characters that caused the delay. The correction removes a chunk SIZE from both engines. It does not restore the time either engine spent before its first chunk left the server.

What this section does not do#

It adds no samples, runs no engine and changes no configuration. Every unmatched axis and every limitation carries over from the predecessor page unchanged.

Realised prompt-length histogram#

Read back from each server’s own usage.prompt_tokens, over the measured requests of every leg. Each engine renders its own chat template, and both rendered the corpus to the same counts.

The corpus itself measures 26 to 3,233 tokens with the target’s own tokenizer (benchmarks/variadic/corpus_tokens.py, seed 0). The c = 1-8 legs used a 144-prompt corpus (sha256 6d6420fc4c0a55019dc24da3dd5b410e2743a6b268f2ee2fb2c89ae21d3cfc5e); the c = 16-32 legs used a 192-prompt superset (sha256 b52ef36c4c0a55019dc24da3dd5b410e2743a6b268f2ee2fb2c89ae21d3cfc5e); see the load above. The chat template adds about 50 tokens to every prompt, which is why the served minimum is 78 rather than 26. The predecessor’s entire workload is the M band alone.

Throughput, and the concurrency the comparator cannot use#

Warm only. Aggregate output tokens divided by the measured wall clock.

Ours moves 50.6% from c = 1 to c = 8, holds through c = 16, then drops 6% at c = 32 where the per-stream rate has risen to 480 ms per token. Theirs moves nothing across the whole sweep. That is the gen_lock above, measured: their engine serves one request at a time whatever the client offers it, so its aggregate rate is its single-stream rate and the extra requests wait.

Both engines’ two rounds agree on different server boots. Ours agrees to 1.0% or better at every rung. Theirs agrees to 0.1% at c = 1 but widens to 4.2% at c = 8, where the second round served into heavier queueing.

Percentiles#

Warm only, pooled over both rounds. Each cell pools 256 requests. Percentiles interpolate linearly between order statistics, as numpy.percentile and vllm/benchmarks/serve.py:739 do.

Time to first token, ms#

At concurrency 1 their engine wins every percentile, and its tail wins by more than its median: 1.09x at p50 and 2.11x at p95. At every higher rung their figures are queue time and ours are not. At c = 32 their p50 time to first token is 164 s against our 12.8 s.

Time per output token, ms#

This table is the other half of the throughput table and should be read with it. Their per-stream rate is flat because every request runs alone. Ours degrades under batching, which is what buys the aggregate. An operator picks a point on that trade; the harness does not pick one for them.

End-to-end latency, ms#

Where the time to first token goes: prefill, by band, at concurrency 1#

This is what the mixed corpus was built to expose, and the predecessor’s 93-to-457-token workload could not.

The implied rate divides the mean prompt tokens by the mean time to first token, so it carries each engine’s fixed per-request overhead and understates both at short lengths. The trend is the result: their rate keeps climbing with prompt length and ours plateaus near 300 tok/s. At the longest band they are 2.02x faster to the first token.

This is why our time-to-first-token tail at concurrency 1 is worse than theirs while our median is close: p90 and above are the L and XL prompts, which are 17% of the corpus.

The mechanism is now identified: their engine reconstructs EXL3 weights to fp16 on-device and calls cuBLAS when the prefill batch exceeds 144 tokens (AUTO_RECONSTRUCT_THRESHOLD in exllamav3/modules/quant/exl3.py:132), while this engine always calls the EXL3 cooperative kernel regardless of batch size (dense_attn_block.h:297). The cuBLAS fp16 path is faster at large M, which is why their rate climbs and ours plateaus. That gap is tracked in #3124 .

Streaming granularity, and why the correction is zero on this run#

The predecessor inferred that our first streamed chunk carries a whole accepted speculative block, and that this inflates our time to first token at the median. Measured here, it does not. Our first content chunk is exactly two characters on every one of the 1,280 measured requests, so the estimated first-chunk token count is 0.58 and the correction is nil. Our later chunks do carry blocks, which is where the 4.81 comes from.

So the corrected and raw figures are equal on both engines in this run, and every time-to-first-token number above is a time to first token rather than a time to first block. The correction stays in the harness because the property it removes is real on other configurations, not because it did anything here.

Cold start#

The cold start is per process, not per leg, and the table shows it. Our first leg on a fresh server paid 27,106 ms; the round-2 c = 1 leg, which ran third on its server, paid 791.2 ms. That is why the discard is positional and why both views are published: on a leg that follows another the two views agree, and on the first leg of a boot they differ by 9%.

At c > 1 the first request by dispatch order still competes with its peers, so those cells contain queueing as well as any cold start. They are not a cold-start measurement and should not be read as one.

Acceptance, from both engines#

Ours comes from GET /metrics, which serves vLLM’s four speculative-decoding families since 58a7162ba closed #2770 . Theirs comes from the Job counter its server wrapper forwards. The predecessor had one side only, and that gap is closed.

Read the first column, not the second: their engine exports no count of proposed draft tokens, so accepted-per-output-token is the only denominator both engines share. On it we accept about 2% more per output token than they do, and per-output-token acceptance is flat across concurrency on both sides. Our accepted/proposed ratio falls from 0.54 at c = 1 to 0.52 at c = 32, tracking the rising per-stream latency.

What is not matched#

Each engine runs at its own published configuration, which is the shape this comparison was built for and is not one configuration.

• Context and KV cache. Theirs is -cs 262144 with an NVFP4 KV cache. This engine refuses an NVFP4 KV cache by name (#2620 ), so their cache configuration is not one our arm can take.
• Prefix caching. Disabled explicitly on our side. Their paged cache reuse is not configurable from their server wrapper, so leaving ours on would be an unmatched cache rather than a matched one.
• Their engine revision is our choice. Their card pins none.
• Their stop rule. Their generator ends at max_new_tokens - 1 - num_draft_tokens, so a request for 192 returns 184 from them and 192 from us. Each side is divided by the tokens it produced.

Owed#

• #3124 . The band table shows their rate climbing with prompt length while ours plateaus near 300 tok/s. The root cause is identified: their engine reconstructs EXL3 weights to fp16 and calls cuBLAS when M > 144, a path this engine lacks. The fix is a code change, not a measurement.
• A matched-configuration leg. Each engine still runs its own published recipe, and this engine still refuses an NVFP4 KV cache by name (#2620 ).
• #2993 . gpu_memory_utilization does not account for the draft speculative context.

Evidence#

docs/bench-evidence/qwen38-27b-exl3-variadic-20260905/ . Per-request records stay on the share at /mnt/nas_share/rc/exl3-variadic/out/.

Reproduce this run#

See the methodology page . The checkpoint download and verification steps are the predecessor’s (qwen38-27b-exl3-gb10 ) and are not repeated here.