docs/multimodal-input.md

Send multimodal input

The OpenAI-compatible server routes supported image and audio content parts through /v1/chat/completions to the model. Support depends on the model:

These are implementation ceilings. --limit-mm-per-prompt can lower them; --language-model-only sets them to zero. Dots3-note can mix images and audio in one prompt. Qwen3-VL serves one sequence per model step.

CPU tests exercise the serving path with synthetic weights. They establish that media reaches the model, not token parity on real checkpoints: Qwen3-VL tests and dots3-note tests . No real-checkpoint token-parity or accelerator-performance claim follows from these tests. Other multimodal architectures do not gain HTTP support from registration alone. The text CLI and vllm_chat C API do not accept these media requests.

Send an image#

The server accepts square RGB images as packed, unsigned 8-bit bytes, with three channels per pixel and no file header. Use image/x-raw-rgb in the data URI. PNG and JPEG files are refused; changing their MIME type does not convert them to raw RGB.

Start a server with a local Qwen3-VL safetensors checkpoint:

./build/examples/vllm-server \
  --model /models/Qwen3-VL-4B-Instruct \
  --served-model-name qwen3-vl --port 8000

Save a square raw RGB image as image.rgb. For a 448 by 448 image, the file must contain exactly 602,112 bytes. Run this request with Python’s standard library; it reads the image from disk and sends its bytes:

import base64
import json
import math
from pathlib import Path
from urllib.request import Request, urlopen

rgb = Path("image.rgb").read_bytes()
side = math.isqrt(len(rgb) // 3)
if side == 0 or side * side * 3 != len(rgb):
    raise ValueError("image.rgb must be a square HxWx3 raw RGB buffer")

image_url = "data:image/x-raw-rgb;base64," + base64.b64encode(rgb).decode("ascii")
payload = {
    "model": "qwen3-vl",
    "max_tokens": 32,
    "messages": [{"role": "user", "content": [
        {"type": "text", "text": "Describe this image."},
        {"type": "image_url", "image_url": {"url": image_url}},
    ]}],
}
request = Request(
    "http://localhost:8000/v1/chat/completions",
    data=json.dumps(payload).encode("utf-8"),
    headers={"Content-Type": "application/json"},
)
with urlopen(request) as response:
    print(json.load(response))

Dots3-note uses the same image format. Its audio path accepts input_audio or audio_url parts carrying PCM16 RIFF/WAVE data. It mixes channels to mono and resamples to the checkpoint’s configured sample rate. MP3, FLAC, and Ogg are refused. Parsing a part type does not establish model support: both registered chat paths refuse video_url.

Add a clip multimodal projector to GGUF#

A GGUF multimodal model has two files: the language .gguf and a clip-architecture mmproj-*.gguf carrying the vision tower. Name the second one with --mmproj (vllm-server) or vllm_model_params.mmproj_path (C ABI v22); it is never auto-discovered from a sibling filename, because a directory holding two unrelated models must not silently fuse them.

./build/examples/vllm-server \
  --model /models/Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj /models/mmproj-BF16.gguf

The loader opens the projector and reads its clip.* metadata and v.* and mm.* tensors. The engine then holds the same vision tower that the safetensors path builds. No forward pass consumes this tower yet. Neither the server nor the C ABI has a multimodal GGUF request path, so the flag validates and loads the tower but does not produce an image answer (#821 ).

The loader refuses four conditions by name before it reads the tokenizer or any language-model weight bytes:

• --model is not a .gguf. A safetensors checkpoint carries its tower in its own shards and needs no projector file.
• the file’s general.architecture is not clip (this is what you get for passing the language file twice).
• its clip.projector_type is not qwen3vl_merger. A muse-glimmer projector is routed to MuseGlimmer’s own recorded refusal instead, which names the missing axis.
• it carries v.patch_embd.weight without v.patch_embd.weight.1. llama.cpp writes the temporal patch embedding as two halves; with one of them absent, loading would mean inventing the other, and the result would be a fluent, wrong model rather than an error.

The checkpoint registry owns the exact file sizes, repository revision, and checksums. The Qwen3.8 27B model recipe owns current arm support and limitations. Loader and gate evidence remains in the Qwen3.8 quantized-arm spec .

Per-prompt input limits#

vLLM caps how many items of each modality one prompt may carry (--limit-mm-per-prompt), and --language-model-only is sugar for setting every one of those limits to 0. Both flags are accepted (#607, waves L1+L2) and both are enforced on this server’s chat path, which is the one place that installs the multimodal chat seam the check runs behind.

Both are also C ABI fields (vllm_model_params.language_model_only / .limit_mm_per_prompt, ABI v19), and there they configure the engine, including a server built on it, but they do not change what a vllm_chat call returns: the C ABI has no multimodal request path yet, so an image_url content part sent through it is dropped and answered as text. The refusals below are the server’s. vllm_model_params.mmproj_path (ABI v22) is in the same position: it loads and validates the projector, and no C-ABI call can feed the tower an image yet.

The flag sets all modality limits to zero and refuses media requests. Qwen3-VL, MuseGlimmer, and clip projector loaders also skip a tower when all modalities it serves have zero limits. The dots3-note loader still loads its supported vision and audio towers, even with zero limits.

With default limits, a Qwen3-VL server refuses three images with HTTP 400:

{"error":{"type":"BadRequestError",
          "message":"At most 1 image(s) may be provided in one prompt."}}

Two things follow from how the limit is computed (min(user limit, what the model/seam supports)):

• A user limit can only lower the ceiling, and what it lowers is declared per architecture by the chat seam that architecture registers. On a Qwen3VLForConditionalGeneration server, the option --limit-mm-per-prompt '{"image": 99}' still refuses a second image, because that seam declares {"image": 1} and routes no video or audio part at all, so those limits are 0 and such a part is refused by name rather than dropped, which is what closed #686 . Dots3NoteForCausalLM is the exception, since W8a (#2860 ): its seam applies every item of every modality in ONE pass over the prompt, so it declares upstream’s own {"image": 512}, plus {"audio": 128} when the checkpoint carries an audio_config. There '{"image": 99}' resolves to min(99, 512) = 99 and the second image is SERVED, and one request may carry an image and an audio part together. video is absent from that seam too, so a video part is still refused by name at limit 0. --limit-mm-per-prompt '{"image": 1}' is how an operator puts the single-image ceiling back.
• The Set `--limit-mm-per-prompt` to increase this limit. hint appears only when raising the limit would actually help, that is, when the seam could take the items and the configuration is what refused them. Its absence is currently the only way to tell an unimplemented arm from a configured limit; the refusal message itself does not say which (#758 ).

What the zero limits now free. In the Qwen3-VL, MuseGlimmer, and clip projector loaders, a tower whose every modality is at limit 0 is constructed but never loaded: its geometry is still parsed from vision_config, so a refusal can still name what is missing, and its checkpoint tensors are never read. This mirrors vLLM’s _mark_tower_model (vllm/model_executor/models/interfaces.py:288-293), and it follows from the LIMITS rather than from the flag: --limit-mm-per-prompt '{"image":0,"video":0}' skips the same tower, and one non-zero modality keeps it.

Tower-skip tests cover the checkpoint loaders for MuseGlimmerForConditionalGeneration and Qwen3VLForConditionalGeneration, and the --mmproj projector, which is the Qwen3-VL tower read out of a second clip GGUF beside a .gguf language file. On the --mmproj path the file is still opened and still partly validated at zero limits, and only its tensors go unread.

Which validation survives the zero limits is worth being exact about, because the two classes behave differently:

• Refused whatever the limits are — the checks that run before the read: a projector whose architecture or clip.projector_type this build cannot use (RefuseUnsupportedClipMmproj), and one carrying tensors the reader would not consume (RefuseUnaccountedClipMmproj). Both sit above the skip, alongside ClipMmprojVisionConfig, so the geometry still resolves and the file is still named in the error.
• NOT reached at zero limits — the checks that live inside LoadQwen3VLVisionFromClipMmproj, which is the call the skip removes. A projector missing a tensor the tower needs, such as the temporal half v.patch_embd.weight.1, is refused at default limits and walks straight past the loader with --language-model-only. That is the construct half of construct-without-initialise doing what it says: what stops is the storage, and the reader’s own missing-tensor refusals stop with it.

The server prints one line naming what was skipped, read back off the loaded model rather than off the flag:

$ vllm-server --model /path/to/muse-glimmer-30b --language-model-only
server: multimodal limits language-model-only=ON audio=0 image=0 video=0
server: multimodal towers NOT loaded (every modality they serve is at limit 0): vision_tower

Nothing is printed when nothing was skipped, so a text model and a multimodal model at their default limits both look exactly as they did before.

How many bytes that saves, measured on one model. On Qwen3-VL-4B-Instruct the flag frees 0.770 GiB of host RSS at load (826,916,864 bytes: peak 9,381,281,792 B without it against 8,554,364,928 B with it), and the swapped repeat agreed to within 339,968 B. Measured 2026-08-28 on dgx:gpu0 under an rc lease, --device cpu, at 525d2b991, against a threshold of 747,625,881 B — 90% of the 830,695,424 B tower the checkpoint ships — that was derived before the run.

An earlier run of the same procedure, on 2026-08-24 at 41ab550b9, read 1.542 GiB. That is not a regression between the two, and the difference is not noise: about half of the older figure was a widening defect of ours, described in the second bullet below, and fixing it moved the saving to 0.499x of what it was. Both runs MET their own pre-derived threshold on both pairs.

Three things that figure is not, all of which matter before you quote it:

• It is one model’s tower, not a general saving. How much a skip frees is how big that model’s tower is, and nothing else. muse-glimmer-30b’s tower is 4.6x larger on disk, still held in host f32, and still unmeasured, so the number above says nothing about it. Do not scale one into the other: the 4.6x is an ON-DISK ratio, and because muse-glimmer-30b still widens while Qwen3-VL no longer does, its RESIDENT tower is about 9.3x the 0.770 GiB above, not 4.6x. Its threshold is derived from its own headers, 90% of 7.161 GiB, and is unchanged by any of this.
• About half of the older figure was a defect of ours, and that defect is now fixed and the fix is measured. Qwen3-VL’s tower is 0.774 GiB on disk in bf16, and our loader used to widen it to host f32 (#1359 ). The Qwen3-VL half of that landed, and the 2026-08-28 rerun is what confirms it: the default arm — which is the arm that pays for the tower — dropped 828,219,392 B (0.771 GiB), from 10,209,501,184 B to 9,381,281,792 B, or 99.7% of the 830,695,424 B the fix predicted. The --language-model-only arm loads no tower and so should not have moved, and it did not: +655,360 B, or +0.0077%. That control is why the drop can be attributed to the fix rather than to the change of host. The saving the flag reports therefore fell to 0.499x of the older figure, and the smaller number is the honest one — the flag now frees the tower the checkpoint ships rather than the tower plus our widening. muse-glimmer-30b still widens, blocked on #2166 , and Gemma-4’s vision tower is unreached, tracked by #2173 .
• It is load-time residency, and it is host RAM. The measured window ends at server readiness, and the build was CPU-only, so this is not a steady-state serving figure and not a VRAM claim.

The procedure and its pre-declared thresholds are scripts/mm/tower_skip_rss.sh and .agents/specs/multimodal-track.md §1.5 L3 — one threshold per model kind, muse-glimmer and qwen3-vl, each derived from that checkpoint’s own safetensors headers, because one model’s tower size does not describe another’s. muse-glimmer has not run: it needs about 56 G staged to local disk on a leased device (#607 , #1358 ). See Memory benchmarks .