docs/at-a-glance.md
At a glance: W5/W6 green; validated release artifacts pending
| Reference | Workload | Headline | Tokens |
|---|---|---|---|
Weight load (ENG-LOAD-DIRECT-UPLOAD, #150) | Qwen3.6-27B bf16 (50.098 GiB), GB10 Vulkan, same binary both arms, interleaved under one GPU lock | Load phase 1.54x warm (19.27 -> 12.48 s), 1.61x cold (52.62 -> 32.75 s); bytes moved 100.196 -> 81.260 GiB. Every ON leg beat every OFF leg | byte-identical; 6/6 token-exact |
Expert streaming vehicle (ENG-EXPERT-STREAM, #912
) | Qwen3.8-2.4T-A95B UD-Q1_0 (370 GiB) on ONE GB10, 119 GiB, c1 | Loads and generates, streaming OFF: resident 62 GiB, decode 66.7 s/tok. Experts BORROW the mmap; ~6.7 GB/token, ~50x off NVMe. Streaming-ON decode VOID (#912 F1); re-measured LIVE by ENG-EXPERT-STREAM-DEVICE | correct answer; no oracle runs this |
Device expert slices (ENG-EXPERT-STREAM-DEVICE, #1124
, #1299
) | Qwen3.8-2.4T-A95B UD-Q1_0 370 GiB on cuda vs cpu, one GB10, greedy 32 tokens, 4000 expert-stream slots (8000 is 3.6-4.1x slower), record | CPU 11.05 s/token steady, replacing #912 F1’s VOID. CUDA G0-LIVE PASS: 32/32 steps, exhausted delta 0, peak RSS 97.75 GiB. G0-CORRECT FAIL: router input differs at block 0 while embedding output is bit-exact; cause remains open. G0-SPEED VOID | correct; no oracle |
BPE merge loop (SPEC-BPE-QUADRATIC-MERGE, #1365
) | 65,536 bytes in ONE word, both tokenizer goldens; one harness source, both sides, 20-core host at load 1.98 | Quadratic to linear. Mistral 23,620.7 -> 7.797 ms; Qwen3.6 22,813.1 -> 10.563 ms. Per 4x input: 16x before, 4x after. At matched load, PARITY with HF tokenizers (10.563 vs 10.546 ms), not faster | 320/320 ids match HF tokenizers 0.22.2; re-derived at the close with the eight tokenizer suites green |
Request-length guard (SERVE-REQUEST-LENGTH-GUARD, #1541
) | No number owed, and none taken: a refusing byte bound before tokenization, with no throughput, latency or memory gate declared | Not a rate. 67823aee2 already took 64 KB in one word to tens of milliseconds, so this bounds a 100 MB body and a future regression, not a measured cost | n/a |
| Structured state record (active) | v1 scalar + relational + Git-history contracts | No benchmark. At 776c56f1: 157 imports = 3,231,342 exact bytes; append preserved all 156 wrappers/rows. 95 tests: validator/core 44 (checker 20 + core 24), NOW 18, migration 22, cutover 11. New raw-row mutation guard. | n/a |
| Binary release (ACTIVE; Windows pre-alpha pending) | v0.0.2 shipped eight primary archive/checksum/provenance triplets + two indexes (26 assets) from source SHA 7020de93652ca920424a10ac5255b34810dd2f24, run 31466516224 | Windows W14-W16 implemented. PENDING: native hosted gates, merged-SHA ten-tuple dry run, matching-hardware evidence, v0.0.3-pre.1 publication, 32-asset audit | W12 optional/non-primary |
| Container images (ACTIVE; arm64 cuda verified on GB10 + Orin 2026-08-11) | ENG-RELEASE-CONTAINERS (spec
) | cpu amd64 783 MB; cuda arm64 1.71 GB. GB10 sm_121a: /health+/version+SIGTERM on --gpus all. Orin sm_87 (Tegra): Qwen3-0.6B generates, GPU GR3D 95-97% | n/a |
| Developer/row protocol | Contribution entry point; ENG-NOW-DERIVED #374 @dbd0d51c | Entry-point gates retained. #374 W1-W5 DONE; benchmark/runtime/parity VOID; row specs now carry ## Now | n/a |
NemotronH host re-expansion / decode token (A2-Q2b, #810
) | Real Nemotron-3.5-Lightning-30B NVFP4 (rev 29f2d174), T=1 decode step, counted at the DenseBf16 seam via examples/nemotron_h_gen | Full host arm 3 078 512 640 elements (6.157 GB bf16), 369 calls. lm_head 352 321 536 (11.44%), 28.35% of the post-A2-Q2a residue, 100% once the mamba arm lands | Per-group table in the record
. Attribution only, no speed number. Device lm_head gate PENDING a dgx:gpu0 window |
Record-anchor ratchet (ENG-RECORD-ANCHOR-RATCHET, #632) | No number owed: a record checker. At 8daa67b39, 832 of 867 in-scope citations (96.0%) were already parsed; no symbol test and no report ran. Rot 38 (32 stale, 6 broken), 32 in range | ||
NemotronH paged forward (MODEL-NEMOTRON-H-ABI-A2P, #810
) | Decode 0.001392x (10.3194 vs 0.014369 s/token, batch 1). Load 2.12x FASTER (280.9 vs 596.3 s). vs pinned vLLM, CUDA graphs ON (#1250 ) | 96/96 STRICT PASS both legs on 5325b7b97, NOT current main; oracle 180/192 vs its OWN golden. Clock gate REFUSES the pair | GPU idle 2,019 of 2,155. A2-Q1’s device arm is 7.28x faster, UNGATED at 95/96 (#1388 ) |
LoRA runtime W2 (LORA-RUNTIME, #278) | No number owed: correctness-only; a grid PENDS the W7 model gate | ||
| ARCH audit: ABI is text-only | 4 capabilities (H3 video, Laguna, Kimi-Linear, DeepSeek-V4) reachable only from examples/, none registry-backed. No gate asks whether a CONSUMER can reach a capability. Documentation only | ||
CUDA-graph break seam W1 (ENG-CUDAGRAPH-BREAK, #1192
) | No number owed, and none taken: coverage and correctness row, no throughput gate declared | Capability, not a rate: mid-forward capture re-begin holds on a leased GPU; scoped forward matches eager, 500 logits, 0 differing. Probe committed, recipe and sha256 in the record | |
| DSR fix: server TU profiler guards (2026-08-09) | No number owed: comments only. #189 moved the server body into the shared layer with its 5 VT_BENCH_PROFILE_CONTROL guards, taking DSR 32 -> 37; they are DSR-ALLOW’d per site, baseline unchanged at 32 | ||
| DSR fix: async readback capability (2026-08-08) | No number owed: behavior-neutral (CPU/CUDA async-ON, discrete non-CUDA async-OFF, unchanged); moves a kCUDA check onto Backend, unblocking red CI on #127/#154/#155 | ||
ROAD-V1-MEM M1+M2 (2026-08-08) | KV auto-sizing CPU brick: --kv-cache-memory sizes the pool from a byte budget via the group-aware KVBytesPerBlock divisor (ABI v16, CPU-gated). M3 profile run dgx-gated | ||
| Record/checker repair 2026-08-07–08 | Gates fixed. Public: VT_GEMMA4_EXPERT_VRAM_MB (positive-MiB LRU cap; unset/0 unlimited), VT_SERVER_MAX_{PROMPT_CHARS,NEW_TOKENS} (200000/4096; 0 disables); nine Gemma4/ROCm tuners internal. No runtime/perf change. | ||
Clock attribution (BENCH-ASSERT-CLOCK-STATE, #543) | dgx.casa GB10, driver 580.159.03, no throttling either side | A CLOCK-CONTROLLED series now exists: flat 2184 MHz (requested 2190), one boot_id, one leg n=861 / min 2158. Every OTHER figure predates clock assertion, where med 2470 vs 2190 repriced marlin::Marlin +9.65% | n/a, nothing withdrawn |
| vLLM | Qwen3.6-27B NVFP4 unsloth @890bdef7, GB10 | ahead 4.5% at c1, tie at c2 to c32, all SUPERSEDED: rollback oracle (#520) + unfused denominator (#414), both flattering us. No pin re-run yet | identical |
| vLLM | Qwen3.6-27B NVFP4 nvidia @0893e160 (ModelOpt modelopt_mixed), GB10 | BINDING at the pin, clocks pinned: 0.976x c1 / 0.946x c4 decode TPOT, TTFT 0.944x c1. c16 VOID (#577). The 0.937x-0.956x canonical is SUPERSEDED and optimistic: rollback oracle (#520) + unfused denominator (#414) | near-tie |
| vLLM | Qwen3.6-35B-A3B NVFP4 nvidia @491c2f1e, GB10 | BINDING at the pin, clocks pinned: 0.995x c1 / 0.946x c4 decode TPOT; TTFT 0.920x c1, 0.849x c4; c16 NOT ESTABLISHED. The 0.918x-0.972x canonical is SUPERSEDED and optimistic (#520, #414) | near-tie |
| vLLM | DeepSeek-V2-Lite (MLA), GB10 | 0.86x to 0.95x throughput, TTFT wins at c4/c8 | identical |
| vLLM | Laguna-S-2.1 NVFP4 (118B/8B MoE), GB10 | parity+, 1.03x (44.46 vs 43.10 tok/s, byte-exact, default config; bf16 weights now device-resident) | near-tie |
| llama.cpp | Qwen3.5-2B GGUF, CPU aarch64 | 20-core Arm/i8mm: prefill 1.18x ahead, decode tie, memory parity. RPi5/A76: vllm.cpp is 0.461x prefill / 0.653x decode+E2E, but uses 24.2% less RSS | byte-identical on both Arm lanes |
| MLX-LM | Qwen3-0.6B, Apple M4 | 97.6% warm total, prefill ahead | near-tie |
exllamav3 (MiaAI-Lab fork @ 63b32f00) | Qwen3.8-27B EXL3 3.5bpw with its DFlash2 draft, GB10, one client over HTTP driving both engines, 164 HumanEval problems at T = 0.6, four interleaved legs | 1.197x decode only (53.63 vs 44.82 tok/s) and 1.123x whole run (37.35 vs 33.26 tok/s). Discard the cold first request of each leg, as the detail page’s method requires, and time to first token splits. The warm mean is 1.029x (929.8 vs 904.0 ms), p50 is 0.884x to them, and p90 and p99 are 1.427x and 1.856x to us. The 0.901x published on 5 September 2026 kept that cold request; #2968 corrects it. Our stream carries 4.47 completion tokens per chunk against their 1.08, so the median is not a matched comparison. Their acceptance 0.774 per output token; ours is not exported at all (#2770 ). Detail | none: both arms sample at T = 0.6, so no token gate can run |
| DwarfStar | DeepSeek-V4-Flash GGUF, GB10 | beats ds4, 1.144x (18.69 vs 16.33 tok/s, byte-exact, default config) | n/a, GGUF peer |
| vLLM | Kimi-Linear-48B-A3B, GB10 | no binding number: the published checkpoint is tiktoken-only, so it cannot drive the warm-server harness | golden 122/128, near-tie profile |
| Muse Glimmer 30B (#268) | no vLLM denominator (pin lacks muse_glimmer); SECONDARY llama.cpp, same GGUF, idle GB10, after the #391 fix | vLLM axis is an OPEN GAP. vs llama.cpp: in128 prefill 1.023x (was 0.878x); in512 decode 0.194x (3.41x), prefill 0.175x flat. Denominator stock 7044859, SUPERSEDED (#1003) | coherent, NOT token-exact |
| #1047 prefill-peer Finish barriers (item 3) | Gemma-4-26B-A4B-it-fp8, dual R9700, isolated :8012, T=2029, PEER_ACT=1 | Attribution only, not a ship claim. Two Finish success-path barriers cost 2.55% (1122.10 vs 1094.24 tok/s; 46.05 ms/req) vs retained RetirePinThenUnpin wait. record | ident-equal 4/4 |
Reading the ratios: throughput is ours/reference, latency is reference/ours, so 1.0 or higher is a win everywhere on this page. Which architecture each number is measured on, and the per-architecture correctness gate behind it, is the registry-bound list in FEATURES.md .