LM Studio is faster one at a time, vLLM NVFP4 under load.
Nemotron 3 Nano 4B as Unsloth's GGUF on LM Studio and as an NVFP4 build on vLLM, with the BF16 weights it came from as the control: quality, one request at a time, Continuum's cost on each runtime, and up to eight in flight.
UPDATED 2026-09-26 · VERSION 0.1
The benchmark
The runtime comparison runs Continuum's model, Nemotron 3 Nano 4B, on two runtimes and three builds. LM Studio runs the Unsloth GGUF on Windows. vLLM, in WSL2, runs an NVFP4 build made from NVIDIA's BF16 weights, and the BF16 file itself as the control. LM Studio can't load the NVFP4 build: llama.cpp's converter refuses a checkpoint that mixes FP8 and NVFP4 layers. So the headline pair is each runtime's best format, which measures runtime and format together, and vLLM's BF16 against its NVFP4 isolates the format. Every run is kept, and the excluded ones are marked with the reason.
Quality control
No build fell measurably behind on 98 checked answers.
The quality gate ran before any speed test: Continuum's 60-question battery, with 49 questions checked automatically (10 from GSM8K, 10 from IFEval, 29 of Continuum's own). Each build answered them under two sampling settings at the same seed, 98 checked answers per build. The checks graded them alone, with no judge model, so no second model had to be loaded.
LM Studio Q4_K_S | vLLM NVFP4 | vLLM BF16 | |
|---|---|---|---|
| Checks passed | 89 of 98 (91%) | 92 of 98 (94%) | 94 of 98 (96%) |
| GSM8K | 20 of 20 | 20 of 20 | 20 of 20 |
| IFEval | 19 of 20 | 18 of 20 | 18 of 20 |
| Continuum's own | 50 of 58 | 54 of 58 | 56 of 58 |
The order, BF16 then NVFP4 then Q4_K_S, is what precision predicts. Paired
answer by answer, no difference is significant at this size, by an exact McNemar test:
NVFP4 against Q4_K_S, the headline pair, is 4 answers to 1 (p = 0.375), BF16
against Q4_K_S 6 to 1 (p = 0.125), and NVFP4 against its own BF16 source 2 to
0 (p = 0.5). BF16's one truncated answer, 8,192 tokens of reasoning
with no reply on a long counting question every build gets wrong, is counted as a fail.
One at a time
One request at a time, LM Studio decodes 1.45 to 1.48× faster and vLLM reaches the first token 2.5 to 2.8× sooner.
All three builds were loaded at once and served strictly in turn, one request in flight on the whole machine, with the order rotating every run. Each prompt starts with a unique run-id line, so no prefix cache serves an earlier run. Each build ran 200 times per prompt: 1,800 requests, with no errors.
Table view
| Build | p50 | p95 | p99 | Runs |
|---|---|---|---|---|
| LM Studio Q4_K_S | 292 ms | 300 ms | 306 ms | 200 |
| vLLM NVFP4 | 110 ms | 114 ms | 131 ms | 200 |
| vLLM BF16 | 168 ms | 177 ms | 186 ms | 200 |
| ms per output token, p50 | LM Studio Q4_K_S | vLLM NVFP4 | vLLM BF16 |
|---|---|---|---|
| Short prompt | 2.87 | 4.26 | 6.02 |
| Long prompt | 2.69 | 3.98 | 5.64 |
| 4K document | 2.78 | 4.02 | 5.72 |
A single sequence's decode is bound by memory bandwidth, and the GGUF is the smaller file: 2.8 GB against 4.8, because only 31% of the NVFP4 build's weights are 4-bit, with the rest in FP8 (44%) and BF16 (25%). The prefill is compute-bound, where NVFP4's FP4 tensor cores count: vLLM reaches the first token of a 4K document in 110 ms against LM Studio's 292. At equal output, vLLM NVFP4 takes 1.27× LM Studio's time on the short prompt, 1.43× on the long one and 1.12× on the 4K document.
On the 4K document, the shape of Continuum's requests, a long context and a short answer, the two fitted lines cross at 149 output tokens: LM Studio at 290.7 ms plus 2.775 a token, vLLM NVFP4 at 103.6 plus 4.033. A shorter answer finishes first on vLLM, a longer one on LM Studio.
NVFP4 decodes 1.4× faster than BF16 on the same runtime.
The format alone, vLLM NVFP4 against the BF16 it was built from: NVFP4 decodes 1.41 to 1.42× faster, prefills the 4K document 1.53× faster (110 ms against 168), and at equal output BF16 takes 1.37 to 1.43× NVFP4's time on every prompt. On the short and long prompts BF16 reaches its first token sooner, 22-23 ms against 34-35, because NVFP4 quantises every request's activations and a short prefill has too little work to repay it.
Every prompt: totals, first token and the fitted totals
| ms, p50 / p95 / p99 | LM Studio Q4_K_S | vLLM NVFP4 | vLLM BF16 |
|---|---|---|---|
| Short: total | 569 / 761 / 850 | 533 / 725 / 785 | 727 / 969 / 999 |
| Short: first token | 98 / 113 / 120 | 35 / 40 / 46 | 23 / 26 / 27 |
| Short: fitted total at 136 tokens | 481 | 610 | 835 |
| Long: total | 2,012 / 2,743 / 3,037 | 2,575 / 3,361 / 3,831 | 3,362 / 4,739 / 5,025 |
| Long: first token | 86 / 93 / 102 | 34 / 37 / 39 | 22 / 25 / 26 |
| Long: fitted total at 649 tokens | 1,829 | 2,611 | 3,680 |
| 4K: total | 999 / 1,254 / 1,339 | 1,148 / 1,511 / 1,594 | 1,468 / 1,919 / 1,951 |
| 4K: fitted total at 241 tokens | 960 | 1,076 | 1,540 |
Three runs looped: LM Studio's long run 132 and NVFP4's long runs 160 and 188 repeated a Redis command name to the 8,192-token cap, at a repeat penalty of 1.0, with decode speed normal throughout. They stay in the data. At 200 runs the nearest-rank p99 is the 198th value, so NVFP4's two loops, its two slowest runs, fall past it. The absolute times drifted by up to 10% over the evening, cause unknown; the builds took turns, so the ratios between them stand.
Home or away
Continuum's durability costs the same on either runtime.
Each runtime made the same model call directly and inside a durable Temporal workflow, 200 runs per arm. Durability added 16 to 21 ms at p50 on both, and its own time outside the call was 25 to 32 ms at p99.
Table view
| Runtime | p50 | p95 | p99 | Runs |
|---|---|---|---|---|
| LM Studio Q4_K_S, 658-token answer | 14 ms | 26 ms | 30 ms | 200 |
| vLLM NVFP4, 700-token answer | 14 ms | 24 ms | 26 ms | 200 |
Every build was co-resident for these runs. Run again on the short prompt with each runtime alone on the card, Continuum's time was unchanged: 13-14 ms at p50 and 25-26 at p99, against 13 and 25 co-resident. The model calls ran 7-8% faster alone; the earlier runs point to the time of evening rather than co-residence, and the test that would separate the two wasn't run.
The whole platform, which is wired to LM Studio, added 35 to 46 ms at p50. Every stretch of both is on the durable inference benchmark.
Full house
From about three requests in flight, vLLM NVFP4 serves more.
Each build ran alone on the card, at four slots or sequences, with 1, 2, 4 and 8 requests in flight on the long prompt, 100 requests at each level. A single request decodes fastest on LM Studio. vLLM scales almost linearly to its sequence count and takes the lead at about three in flight.
| Output tokens a second, all requests | LM Studio Q4_K_S | vLLM NVFP4 | vLLM BF16 |
|---|---|---|---|
| 1 in flight | 361 | 249 | 176 |
| 2 in flight | 546 | 461 | 328 |
| 4 in flight | 773 | 873 | 491 |
| 8 in flight (the ceiling) | 773 | 874 | 579 |
| First token at 4 in flight, p50 / p95 | 130 / 215 ms | 41 / 46 ms | 28 / 45 ms |
At four in flight vLLM NVFP4 served 13% more tokens a second than LM Studio, and each request got its first token in 41 ms against 130. At eight, both queue: the first token waits 3.5 s on vLLM and 4.2 s on LM Studio. NVFP4 serves 1.4× BF16 at one and two in flight, and 1.51× at saturation. BF16's four-in-flight figure carries a looping request's long tail, so its ceiling is the eight-in-flight 579.
At 100 requests a level, a p99 is the second-slowest request, so this section gives the p95. The memory isn't comparable as run: vLLM took about 30.6 GB, because this phase ran it at its natural KV pool of 907,334 tokens. With the 32-block pool of the one-at-a-time runs it needed 7.5 GB. LM Studio held 5.2 GB.
Two slots, loops and reproducibility
- At two slots, LM Studio still leads at two in flight, 545 tokens a second against vLLM NVFP4's 459. Against one request in flight, two slots serve 1.5× the tokens on LM Studio and 1.85× on vLLM.
- Loops. Six of the 2,400 requests ran to the 8,192-token cap in a repetition loop, 3 on LM Studio, 1 on NVFP4 and 2 on BF16, all on this prompt at a repeat penalty of 1.0. They came from the model on every runtime and build, and they set every p99 they land in. None here came with one request in flight; the three under One at a time did.
- Reproducibility. With one request in flight, every configuration
repeated one answer 100 times in 100. LM Studio's answer held at 1, 2 and 4 slots;
vLLM's changed with
--max-num-seqs, 523 tokens at two sequences and 1,041 at four, from the same prompt, sampling and seed.
By the book
Every setting was matched, or its difference written down.
| Machine | AMD Ryzen 9 9950X3D, 64 GB, RTX 5090 (32,607 MiB, driver 617.14), Windows 11 Pro |
| LM Studio | 0.4.25, llama.cpp runtime 2.45.0, on Windows, at 127.0.0.1:1234 |
| vLLM | 0.29.0 (PyTorch 2.13, CUDA 13.0, FlashInfer 0.6.18) in WSL2 2.6.3 under Docker Desktop, NAT networking, at 127.0.0.1:8000; weights on the VM's own disk |
| Both | 16,384 context; one slot or sequence (two and four under load); flash attention; seed 42; temperature 0.6, top-p 0.95, top-k 20, min-p 0, repeat penalty 1.0, 8,192 max tokens, thinking on; the same chat template, byte for byte |
| Endpoint hop | GET /v1/models from the Windows client, 200 times: LM Studio 2.72 ms at p50, vLLM through WSL2 0.85 ms. The VM hop costs less than LM Studio's proxy to its llama-server |
- One more input token on LM Studio. The Unsloth GGUF prepends a
<s>that NVIDIA's own tokenizer doesn't add, so every LM Studio input is one token longer. Tristan's call: record it and move on. - A 16-bit KV cache on both, not the same one. f16 on LM Studio, bf16 on vLLM, because vLLM's FlashAttention can't pair an f16 cache with a bf16 model.
- Excluded, and kept. The script that queued the platform runs was started twice by mistake, so for about an hour two benchmark clients shared LM Studio's one slot. Those runs and the first alone control are marked excluded and left beside the clean reruns. Nothing else was affected.
How the NVFP4 build was made
Post-training quantisation with llm-compressor 0.14, the sequential
calibration pipeline, seed 42: 128 samples of open-perfectblend, up to
2,048 tokens each, in the model's chat template. The source is Unsloth's BF16 copy,
byte-identical to NVIDIA's.
| Precision | Layers | Parameters | Which |
|---|---|---|---|
| BF16 | 25 | 0.88B (25%) | The output head, the four attention layers' projections, and the Mamba projections of the layer before each: NVIDIA's own exclusions for its FP8 build |
| FP8 | 40 | 1.58B (44%) | The other 17 Mamba layers' projections, and the MLPs of layers 37, 39 and 41 |
| NVFP4 | 28 | 1.10B (31%) | The other 14 MLPs, in groups of 16 with FP8 scales |
# one request at a time: all three builds, interleaved, 200 runs each
dotnet run -c Release --project src/Apps/Continuum.Console -- benchmark latency \
--arm "lmstudio-q4ks|lmstudio|http://127.0.0.1:1234|nvidia-nemotron-3-nano-4b" \
--arm "vllm-nvfp4|vllm|http://127.0.0.1:8000|nvidia-nemotron-3-nano-4b-unsloth-nvfp4" \
--arm "vllm-bf16|vllm|http://127.0.0.1:8001|nvidia-nemotron-3-nano-4b-bf16" \
--runs 200 --warmup 10 --sampling "temperature=0.6,top_p=0.95,top_k=20,min_p=0,repeat_penalty=1,seed=42,max_output_tokens=8192" \
--prompt-file src/Core/Utilities/Continuum.Benchmarks/Models/Prompts/context-heavy-4k.md
# under load: 1, 2, 4 and 8 in flight, 100 requests a level
dotnet run -c Release --project src/Apps/Continuum.Console -- benchmark model-concurrency --capability inference \
--levels 1,2,4,8 --requests 100 --warmup 2 --endpoint http://127.0.0.1:8000 --runtime vllm