The quality numbers were measuring markdown.
The first durable-extraction pass rejected every fact in LM Studio's smoke test for evidence not found in the section. The evidence was there, behind bold, backticks and blockquote markers, and so was the evidence behind 588 of vLLM's 824 rejections.
Smoke and mirrors
LM Studio's smoke test accepted no facts at all.
On 26 September the durable-extraction benchmark ran Continuum's production
fact extractor on two builds of Nemotron 3.5 Lightning, NVIDIA's NVFP4 on vLLM and
Unsloth's UD-Q4_K_S on LM Studio, over 48 frozen sections of the engine's own
CONTEXT_LAWS.md and RULES.md. The extractor quotes evidence for
every fact, and FactValidator keeps a fact only if the quote is really in its
section. Tristan set the study's rule before it started: an egregious error means fix it,
purge the data and begin again.
vLLM went first, 288 extractions between 10:14 and 10:30. Its smoke test had already rejected 11 of 20 facts as "evidence does not appear in the section", and Claude filed them to be classified in the write-up. Then LM Studio's smoke rejected every fact it produced, seven across two sections, for the same reason. Read against the text, the rejected evidence was the section's own words.
evidence: Law Zero is not one of them — it governs where they are silent.
section: Law Zero is not one of
> them — it governs where they are silent.
The model quoted the words and left out the markup. The section wraps the sentence inside
a blockquote, and the > at the start of its second line was enough to
fail the match.
Quote, unquote
Most of the rejected quotes were verbatim.
Claude classified all 824 evidence rejections from vLLM's timed pass against the corpus.
| Why the evidence didn't match | Rejections |
|---|---|
**, __ or backticks the model left out | 577 |
> blockquote markers across a line wrap | 11 |
| a match only if case is ignored, so not verbatim | 19 |
| not in the text at all: paraphrase or invention | 215 |
| in the corpus, but not in its own section | 2 |
588 of the 824, 71%, were verbatim once the markup was ignored. RULES.md is
full of bold and inline code, and the model quoted the words without the asterisks and
backticks around them. The validator compared text with whitespace collapsed and nothing
else, so every dropped asterisk looked to it like an invented quote.
The engine's findings list already carried the defect, as "three rejections for mangled markdown in otherwise verbatim evidence". On this run it was 588. The share of facts rejected, and everything the judge would grade afterwards, stood on that check, so the study's quality numbers were measuring markdown as much as models. By Tristan's rule, that was egregious.
Strip search
The validator strips the markup from both sides and keeps the words exact.
// before: whitespace collapsed on both sides, and nothing else
private static bool ContainsNormalised(string haystack, string needle) =>
CollapseWhitespace(haystack).Contains(CollapseWhitespace(needle), StringComparison.Ordinal);// after: blockquote markers, **, __ and backticks come off both sides first
private static bool ContainsNormalised(string haystack, string needle) =>
Normalise(haystack).Contains(Normalise(needle), StringComparison.Ordinal);
public static string Normalise(string value) => CollapseWhitespace(StripMarkdown(value));StripMarkdown removes blockquote markers at line starts, and **,
__ and backticks, from the section and the quote alike, before the substring
match. Case and punctuation stay strict, since lowering one or dropping the other would
start accepting text the section doesn't contain.
Four unit tests pin the change: a quote across a wrapped blockquote passes, a quote without its bold and code markers passes, a change of case fails, and text that is absent once the markup is gone fails. All 233 model unit tests pass.
No free pass
The first pass was purged by exact file name.
Everything the old validator had scored went: vLLM's timed run, its second smoke, LM Studio's smoke, and the fact runs under them. The morning's first purge, for a model-registry bug, had guessed a file timestamp and deleted nothing, so this one named every file. The facts those runs recorded stay in the dev database, because the fact store never deletes, and new runs record under deterministic ids, so a fact extracted again lands on the same row.
The study began again with LM Studio, which was already loaded, then vLLM, then LM Studio once more to bound drift. LM Studio's new smoke accepted 4 facts and rejected 3 on each arm, where the old validator had rejected all 7. The three left are real: the model wrote "The Laws" where the text says "The laws".
Dead heat
With the check fixed, the judge found the two builds tied.
LM Studio UD-Q4_K_S | vLLM NVFP4 | |
|---|---|---|
| Facts accepted / rejected, first repeat | 306 / 51 | 294 / 40 |
| Judged clean | 66.0% of 306 | 69.0% of 287 |
| Unsupported / unfaithful | 3 / 10 | 2 / 7 |
| Assertions missed, 46 sections | 118 | 140 |
The judge, qwen3.8-27b loaded alone, saw each section and its facts and never
the runtime. vLLM's 3-point lead in clean facts sits inside a standard error of 3.8 points,
so quality is a tie. Neither build invents much, 2 or 3 unsupported facts in about 300,
because the verbatim check keeps invented evidence out before the judge sees it. That is
the job the validator was written for.
Check, please
The check is part of the measurement.
Every extraction figure passes through FactValidator before anything counts
it: the facts accepted, the facts rejected, and the judge's grades, since the judge only
sees the facts the validator kept. A fault in the check is therefore counted against the
model, and on the first pass most of the rejections were the check's own.
vLLM's smoke had shown the fault first, as 11 rejections filed to be classified later, and a full timed pass ran on top of them. What found it was reading the rejected quotes against their sections.
The re-run's figures, with the judge's verdicts and the setup, are on the durable extraction benchmark page.