vLLM wouldn't repeat itself.
At seed 42, one request at a time, LM Studio's Lightning gave one answer 200 times in 200, and vLLM's answered one prompt at seven different lengths. Tristan was sure a setting was missing. vLLM has one, and in 0.29.0 it refuses every model Continuum runs.
Broken record
At a fixed seed, vLLM answered one prompt at six different lengths.
The morning's durable-inference study ran Nemotron 3.5 Lightning 30B-A3B on
two runtimes, each alone on an RTX 5090: NVIDIA's NVFP4 checkpoint on vLLM 0.29.0, and
Unsloth's UD-Q4_K_S GGUF on LM Studio. Every arm ran 200 times at seed 42, one
request at a time, with the same prompts and the same sampling.
LM Studio gave one answer per prompt, 200 times in 200. vLLM's long prompt came back at six different lengths, the most common 118 times in 200, and its 4K document at seven. Nemotron 3 Nano 4B, the inference model, had repeated one answer 100 times in 100 on vLLM the day before.
The durable workflow's own cost was 13-16 ms outside the activity on both runtimes, so the latency figures held. The extraction study that followed is where it showed: across three repeats, LM Studio gave the same facts on 43 of 48 sections, and vLLM on 3. Continuum signs every fact with the model and the exact configuration that produced it, and on vLLM that signature names a configuration that rarely produces the same facts twice.
Set in stone
The setting exists, and no run had set it.
Tristan
"I need to know about the variability, and I promise you, it's an option parameter or environmental config we are not using - we are supposed to be using. Research this DIRECTLY."
The research went to primary sources: the vLLM 0.29.0 package inside the pinned image,
and vLLM's own documentation. The reproducibility doc settles it early: "vLLM does not
guarantee the reproducibility of the results by default, for the sake of performance."
For online serving it names one way out, batch invariance, switched on with
VLLM_BATCH_INVARIANT=1. The mode installs deterministic matmul kernels, turns
TF32 off, fixes the MoE tile configuration and refuses at startup any kernel that doesn't
declare support. Tristan was right: it is the one setting vLLM documents for reproducible
serving, and none of the runs had set it.
Computer says no
The switch refuses every model Continuum runs.
Started on Lightning with the study's flags and the switch, vLLM refused at once.
# the study's vLLM flags, plus -e VLLM_BATCH_INVARIANT=1
VLLM_BATCH_INVARIANT forces NVFP4 linear to use the CUTLASS backend for deterministic execution.
ValueError: CutlassNvFp4LinearKernel does not support W4A16NVIDIA's checkpoint stores its NVFP4 weights as weight-only, W4A16, and batch-invariant mode forces NVFP4 layers onto a CUTLASS kernel that handles only full W4A4. Past that check sits a second refusal. No recurrent backend in 0.29.0 supports the mode, so Mamba2, which Lightning and the Nano use, raises at startup, and so does Gated DeltaNet, which is 48 of Qwen3.8-27B's 64 layers. All three of Continuum's models are hybrids, and vLLM 0.30.0, out on 22 September, doesn't change it. Pull requests adding batch invariance to both backends are open upstream.
Claude's first draft of the write-up recommended the switch for the Qwen3.8 judge, because vLLM's batch-invariance doc lists Qwen3-Next among its tested models. A research agent reading the same image found the Gated DeltaNet backend refuses it. Claude confirmed it in the source and corrected the write-up and three other docs the same hour. The doc's tested list and the code disagree, and the code is the one that starts.
Process of elimination
With every configurable source switched off, it still gave two answers in twenty.
Reading the code cleared three suspects. A request with a seed gets its own generator, and
vLLM then falls back from FlashInfer's sampler to its native one. Marlin's atomic-add
reduction is off by default. Prefix caching stores only full blocks, 2,128 tokens for
Lightning, and these prompts never fill one. The same reading showed that 0.29.0 turns
prefix caching on by default, so the --enable-prefix-caching flag on every run
had changed nothing.
Then 20 identical requests, one at a time, on the long prompt with the extraction sampling at seed 42:
| Flags beyond the study's | Answer lengths in tokens (count) | Distinct answers |
|---|---|---|
| the prefix-caching flag removed, so the default keeps it on | 1,021 ×5, 1,024 ×1, 1,054 ×6, 1,060 ×8 | 4 |
--no-enable-prefix-caching --enforce-eager | 973 ×9, 993 ×11 | 2 |
In eager mode with prefix caching off, one sequence at a time and a fixed seed, the two answers arrived in random order, so the variation is in a kernel. The Nano repeats itself on vLLM with the same Mamba2, FlashAttention and CUTLASS FP8 kernels. Only Lightning runs Marlin's W4A16 GEMMs and MoE, which makes Marlin the suspect.
HUMMING, the other W4A16 backend, would have settled it. On its first start it overran the 32 GB card into shared system memory, from 1.97 to 5.28 GB, until the watchdog stopped it, and its retry failed with a CUDA driver error. The watchdog now also trips once shared GPU memory passes 1.5 GB.
Head count
Distinct answers are counted before any quality run.
The benchmark brief gained a reproducibility gate. Before a quality run, the model gets 20
identical requests at the fixed seed, one at a time, and the number of distinct answers is
reported beside every result. On vLLM the gate starts with
VLLM_BATCH_INVARIANT=1, which refuses at startup any model it can't serve
deterministically. More than one answer means every repeat is judged, and the spread is
reported with the result.
For extraction the numbers made the choice. The judge graded the two builds a tie, LM
Studio's 66.0% clean against vLLM's 69.0% with a standard error of 3.8 points, and vLLM ran about 15% faster.
LM Studio's UD-Q4_K_S stays Continuum's extractor, because it gave the same
facts again after a reload 40 minutes later, all 46 extracted sections identical.
Both builds, run for run, are on the Nemotron 3.5 Lightning benchmark page.