docs/WEIGHT-OFFLOAD.md

Weight offload

Weight offload keeps model weights outside device memory so that a model larger than the device can still load. It is a different subject from KV offload , which keeps KV blocks outside the paged cache. The two use separate flags and do not interact.

The --offload-config '<json>' flag accepts vLLM’s OffloadConfig fields. It also accepts a vllm_cpp extension for disk residency and device placement. A config that contains only vLLM fields remains valid here.

Enabling vLLM-style weight offload fails startup today, on every model. No loader consults that offloader yet. The engine refuses the configuration instead of accepting a budget that would free nothing. The vllm_cpp extension is live and is not subject to this refusal. Read What works today before you enable anything.

What works today#

So the honest summary is: you can write and validate a weight-offload configuration, and you cannot yet run with one enabled. A budget that silently frees nothing is worse than a budget the engine refuses, so the engine refuses it.

A configuration that leaves offloading disabled is unaffected: the guard only fires when is_offloading_enabled() is true, so parsing, validation, and the resolved-backend report all still work for inspecting a config.

Progress is tracked in issue #797 .

The flag#

vllm-server --model /path/to/model \
  --offload-config '{"offload_backend":"uva","uva":{"cpu_offload_gb":10}}'

The same document is available through the C API as the offload_config field of vllm_model_params (ABI v21). NULL or an empty string means no offload, which is the default engine.

Place routed experts on the CPU#

The vllm_cpp.placement object keeps attention and dense layers on the selected accelerator. It moves routed-expert computation to the CPU. This can reduce device-memory use on a discrete GPU, but the engine has not measured the end-to-end speed yet.

The following example places every routed expert on the CPU:

vllm-server --model /path/to/model \
  --offload-config '{"vllm_cpp":{"placement":{"cpu_moe":true}}}'

The same flag works with vllm-cli. C callers set the identical JSON string in vllm_model_params.offload_config.

Placement supports these fields:

An explicit override contains a tensor-name pattern and a device name:

{
  "vllm_cpp": {
    "placement": {
      "overrides": [
        {"pattern": "blk\\.0\\..*ffn_.*_exps", "device": "cpu"}
      ]
    }
  }
}

The placement target must be cpu. The engine refuses accelerator targets. It also refuses placement for eagerly resident FP4 experts. Placement is wired for Qwen3-MoE, Qwen3.5/3.6, Nemotron-H, and DeepSeek-V2. Laguna and DeepSeek-V4 do not use the shared placement seam. Kimi-Linear already runs its MoE on the host, so this setting does not move its computation.

GGUF mmap residency is often active when weights remain quantized. The engine warns when mmap and CPU placement are both active because routed experts can then cause slow, router-ordered page faults. The combination remains valid.

Unit and device-boundary round-trip tests cover the placement mechanism. No model has completed an end-to-end run with placement enabled. Treat it as an experimental capacity feature until the token and speed gates finish.

Fields#

Targeting by name segment#

cpu_offload_params matches whole dotted segments, not substrings. For the parameter mlp.experts.w2_weight:

The last row is the reason the rule exists: w2_weight must not also match mlp.experts.w2_weight_scale, because a quantized weight and its scale are different tensors with different sizes.

The two empty-set defaults are not the same#

An empty uva.cpu_offload_params means “offload without selecting, until the byte budget is spent”. An empty prefetch.offload_params means “offload every parameter of each offloaded layer”. The first is bounded by bytes and the second by layer position.

What the engine refuses, and what it only warns about#

The engine refuses a configuration it cannot honour, at startup, before it reads any model file:

server: fatal: offload_num_in_group (5) must be <= offload_group_size (4)

Refusals cover a malformed document, an unknown backend name, a field of the wrong type, a negative budget, offload_num_in_group above offload_group_size, and offload_prefetch_step below 1 while prefetch is enabled.

The model has to claim support, and none does yet#

A valid configuration is still refused when the resolved architecture’s loader does not consult the offloader. This is the refusal you will actually hit, since no model declares support today:

weight offload is configured but architecture "Qwen3MoeForCausalLM" does not
support it: its loader does not consult the weight offloader, so every weight
would stay on the device and the budget would free nothing
(ENG-WEIGHT-OFFLOAD W2c). Remove --offload-config, or wire this model's loader.

It is raised after the architecture resolves and before any weight I/O, so nothing is read from disk before you are told. The support flag defaults to off, which means a newly added model is refused until someone wires its loader, the default is the mechanism, not an oversight.

A model that claims support and then never asks the offloader about a single weight is refused too, after load, and reported as a defect in that loader rather than as a configuration error. Zero consulted weights is the only count that can prove that particular lie.

A backend that disagrees with the fields you set is a warning, not a refusal, which mirrors vLLM. The named backend wins and the other fields are ignored:

[vllm.cpp] offload_config: Prefetch offload fields are set but offload_backend='uva'. Prefetch settings will be ignored.

Why the budget can overshoot by one weight#

The engine tests the budget before each weight, never against that weight’s size, which is what vLLM does. A 10 GiB budget can therefore be exceeded by the size of one weight. Treat cpu_offload_gb as the point at which offloading stops, not as a hard ceiling on host memory.

Limitations#

• No weight moves yet. See What works today .
• On unified memory the feature cannot help. On a device such as GB10 the host and the device draw on one physical pool, so moving a weight to “CPU” frees no device memory. Offload is useful on a discrete GPU, where host memory and device memory are separate.
• The prefetch backend is not built.
• Offload trades throughput for capacity. A weight that is not on the device is read across the bus when it is needed. Expect a model that fits without offload to run faster without it.

Consuming it programmatically#

vllm_model_params p = vllm_model_params_default();
p.model_path = "/path/to/model";
p.offload_config =
    "{\"offload_backend\":\"uva\",\"uva\":{\"cpu_offload_gb\":10,"
    "\"cpu_offload_params\":[\"experts\"]}}";

vllm_engine* engine = NULL;
if (vllm_engine_load(&p, &engine) != VLLM_OK) {
    fprintf(stderr, "%s\n", vllm_last_error());
}

A configuration error returns VLLM_ERR_INVALID_ARGUMENT and sets vllm_last_error(). That is a different result from VLLM_ERR_MODEL_LOAD, which means the configuration was accepted and the model itself failed to load.