The Jev Controversy

Chapter 2 took the contract apart and found it was six-sevenths packaging with one candidate distinction. This chapter asks the adversarial question: granted the contract is mostly packaging, did Jev’s implementation beat the alternatives anyway? There is exactly one independent benchmark that can answer that, and this chapter reads all of it — including the parts that embarrass both sides.

You should know the shape of the answer before the evidence: Jev won one task outright and lost the other by three points, while a 200-million-parameter classifier sat three-tenths of a point behind the winner at a sixth of the latency. “Did not dominate” is the accurate reading. “Lost” is wrong. So is “won”. The rest of this chapter is about why those three readings keep getting confused, what each of the book’s frozen hypotheses predicts from here, and the measurement rules that will keep later chapters from repeating the benchmark’s own mistakes — because the benchmark makes mistakes, and finding them is part of the job.

The benchmark, read completely

Geada, Misiura and Cyril, “Benchmarking AI decision models against traditional guardrails” (developers.redhat.com, 2 October 2026). Nine guardrail configurations across four paradigms, each built twice — once for prompt injection, once for content safety — inside NeMo Guardrails, and measured as round-trip judgments through a local server. The paradigms: pre-trained CPU classifiers (DeBERTa-v3 at 200M parameters for injection, Granite Guardian at 125M for safety — the “gold standard” that ships as defaults); zero-shot BART-large-mnli from 2019; LLM judges (Shieldstral 3B, Nemotron 4B with stock and custom policies, Qwen 35B); and Jev-style systems (Laya at 421M, DiffusionGemma 2.6B through an experimental vLLM endpoint, and Jev-1.13.0 itself over TypeSafe’s API, parameter count unknown).

The script below produces that table. It reads the Red Hat figures transcribed into examples/ch03-the-jev-controversy/claims.py (nothing here is measured by us; every number is transcribed and cited), so it runs in a fraction of a second and needs no model and no network. The point of the file is that the reading is computed, not quoted.

"""The controversy in eight lines: who won what, by how much.

    python examples/ch03-the-jev-controversy/demo_ch03.py

Reads the transcribed Red Hat tables and prints the accurate reading both sides
of the argument have to live with: Jev won content safety outright and lost
prompt injection by 2.96 points, while a 200M classifier sat 0.30 points behind
the winner at a sixth of the latency. Nothing is measured here; everything is
transcribed. The point of the file is that the reading is computed, not quoted.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent))

from claims import CLAIMS  # noqa: E402


def acc(subject: str, task: str) -> float:
    return next(c.value for c in CLAIMS
                if c.subject == subject and c.task == task and c.metric == "accuracy")


def lat(subject: str, task: str) -> float:
    return next(c.value for c in CLAIMS
                if c.subject == subject and c.task == task
                and c.metric == "median_latency_ms")


print(f"safety:            Jev {acc('Jev-1.13.0', 'content-safety'):.2f} "
      f"(1st of 9)")
print(f"injection:         Jev {acc('Jev-1.13.0', 'prompt-injection'):.2f} "
      f"(4th of 9, {acc('Qwen3.6-35B', 'prompt-injection') - acc('Jev-1.13.0', 'prompt-injection'):.2f} pp behind Qwen)")
print(f"injection gapTop:  Qwen {acc('Qwen3.6-35B', 'prompt-injection'):.2f} vs "
      f"DeBERTa {acc('deberta-v3-base-prompt-injection-v2', 'prompt-injection'):.2f} "
      f"= {acc('Qwen3.6-35B', 'prompt-injection') - acc('deberta-v3-base-prompt-injection-v2', 'prompt-injection'):.2f} pp, "
      f"{lat('Qwen3.6-35B', 'prompt-injection'):.1f} vs "
      f"{lat('deberta-v3-base-prompt-injection-v2', 'prompt-injection'):.1f} ms")
print("reading: did not dominate; did not lose either.")

python examples/ch03-the-jev-controversy/demo_ch03.py
safety:            Jev 86.20 (1st of 9)
injection:         Jev 86.35 (4th of 9, 2.96 pp behind Qwen)
injection gapTop:  Qwen 89.31 vs DeBERTa 89.01 = 0.30 pp, 312.5 vs 54.1 ms
reading: did not dominate; did not lose either.

Those four lines are computed from the transcribed tables, not quoted from the prose — a distinction that matters, as you are about to see. The full rankings, the median latencies, the precision/recall profiles and the prompt-tuning moves are transcribed number by number in examples/ch03-the-jev-controversy/claims.py, one row per figure in results/ch03.jsonl. The headlines: Qwen takes injection at 89.31% and 312.5 ms; Jev takes safety at 86.20% and 360.4 ms; the cheapest answers on each task are DeBERTa at 54.1 ms and Granite at 33.2 ms. Prompting alone moves Nemotron from 69.37% to 84.84% on injection and Laya from 57.87% to 75.20% on safety — while Jev loses 3.67 points on Laya’s tuned policy, which is the first hard evidence in this book that tuned policies do not transfer between providers.

The authors’ conclusion is careful and worth quoting once, because both camps misquote it: decision models “do not reliably outperform” judges, pre-trained models, or open decision models in speed or accuracy, while pre-trained predictive models “remain extremely competitive”. Note what that sentence does not say. It does not say Jev lost — Jev won a task outright. It does not say classifiers won either — the classifier won nothing outright and lost safety by six points. It says the field is crowded at the top and nobody dominates it. That is the accurate reading, and everything below defends it against its simplifications.

What the benchmark does not contain

The strongest thing in this chapter is an absence. Nowhere in the article — not in the method, not in the appendix — is there a confidence interval, a significance test, a seed, a stated N, or a warm/cold statement. Accuracy is quoted to two decimals throughout: 89.31 vs 89.01, a gap of three-tenths of a point, with no way to know whether three-tenths means anything at all. This is the evaluation-honesty problem that Hallucination From First Principles exists to name: a number without its uncertainty is not a measurement, it is a rumor with decimal places. The article is still the best independent evidence available. Both facts are true at once, and the protocol at the end of this chapter exists because of the second one.

The article also contradicts its own tables, four times. Its prose says DeBERTa is “only 0.20 percentage points behind first place”; its tables say 89.31 − 89.01 = 0.30. It says “the eight tested guardrails”; its tables list nine rows. Table 2 says BART safety is 68.67%; Table 5 says 0.6873. And Table 3 prints Laya’s tuned gain as +17.83 points while its own two numbers give 75.20 − 57.87 = 17.33 — that last one was caught by this chapter’s self-test asserting the printed figure, which is exactly what executable checks are for. None of these changes any ranking. All of them change how much you should trust a write-up over its tables, including this book’s. The transcription rule in claims.py is the lesson, stated as code: figures come from tables, never from prose about tables, and disagreements are recorded rather than silently resolved.

Wrong: The benchmark’s text is the result. Quote the gap the authors state.

Correct: The benchmark’s tables are the result. The text is one author’s reading of them, and this one misreads its own numbers four times. Compute the gap from the tables (0.30, not 0.20), count the rows (nine, not eight), and record every disagreement where the next reader can find it.

Three claims, and which one the critics attack

The controversy tangles three claims that need untangling before any experiment can separate them:

• C1: applications frequently need a decision, not a sentence. Nobody attacks this. The benchmark itself exists because guardrails need block/allow judgments; practitioners in the Hacker News thread describe decision steps in production harnesses. No chapter is asked to refute it.
• C2: a general decision model is a useful category. Attacked. For it: Jev won safety outright, and Jev-style systems reuse across tasks without retraining. Against it: LLM2Jev shows off-the-shelf LLMs already matching Jev-style models, and pre-trained classifiers remain extremely competitive where labels exist.
• C3: Jev’s implementation beats classifiers and LLM judges. Attacked. For it: the safety win and a competitive 86.35 on injection. Against it: the injection loss, the authors’ conclusion, Rao’s finding that Jev errs in the same places as the judges it would replace, and two unverified commenter claims (jev-sec-bench parity-with-small-models, a Banking77 gap) that Chapter 28 must test rather than cite.

    flowchart TD
    C1[C1: apps need decisions] --> UNCONTESTED[uncontested: no refutation asked]
    C2[C2: decision models are a useful category] --> H1H4[attacked: decided by H1–H4 in Ch 15, 28]
    C3[C3: Jev beats the alternatives] --> CH28[attacked: re-run in Ch 28]
  

Notice what the critics do not attack and what the defenders cannot claim. Nobody disputes that software needs decisions. Nobody has shown that Jev dominates. The fight is entirely over the middle claim — whether “decision model” names a useful category or is packaging around old mechanisms — and over whether the implementation wins loudly enough to settle it. It does not, in either direction. That is why the book needs hypotheses instead of opinions.

The judges are strong; the biases are shared

Two more papers set the terms of the fight, and both qualify somebody’s dismissive talking point.

First, the LLM-judge baseline is not a straw man. Zheng et al.’s MT-Bench study (arXiv:2306.05685) finds GPT-4 judges agreeing with humans at 85%, above the 81% humans manage with each other — with named biases (position first, verbosity, self-enhancement) that the paper documents rather than hides. Anyone who tells you judges are too biased to be a baseline has not read the baseline.

Second, the bias the judges carry, decision models carry too. Wang et al. (arXiv:2305.17926) show candidate order alone flipping two-thirds of pairwise verdicts, then measure the cures: evidence before rating plus aggregation across orders lifts a weak judge’s agreement with humans dramatically. Option order is candidate order. Every option-scoring provider this book builds — starting in Chapter 7 — rotates its options and records the rotation, because this paper already ran the ablation for us. Bucher’s fine-tuned small models (arXiv:2406.08660, re-read in full; the claim in the session brief that Chapter 1 had it “only partly” is corrected in evidence/notes-ch03.md) remain the critics’ strongest exhibit: where labels exist, bespoke classifiers win, and around 200 of them is where the winning starts. That is precisely what H1 leans on — and precisely what H4’s scenario excludes, since H4 is about the world where no labels exist.

Frozen predictions for scenario S1

The experiment in this chapter runs no models. It writes down, for the scenario where the label set changes at runtime and no labelled data exists, what each frozen hypothesis predicts and what would refute it. Seven rows, each naming its deciding chapter:

The hypotheses themselves are frozen in planning/hypotheses.md, unedited — the freeze is recorded in the preregistration commit, and selftest_ch03.py asserts their headings are still there. An executable check (check_predictions) rejects any row whose refutation names no chapter. The acceptance criterion is the prompt’s own: every hypothesis has at least one result that would refute it.

The protocol every later chapter measures by

planning/benchmark-protocol.md is this chapter’s durable artifact, and every rule in it traces to a Red Hat failure it prevents. Splits are fixed in code and the test split is touched once per chapter, because re-cutting is how baselines get flattered. Calibration and ranking are reported separately, never as proxies, because ranking is easier than calibration and conflating them hides it. Latency is round-trip to a usable value — parsing included — with local and remote never mixed and every number carrying its machine id, because the 56 ms floor taught us that geography is a confound. Prompting is a treatment with a stated budget per paradigm, tuned never on test, with per-provider prompt hashes in every row — because Nemotron gained fifteen points from prompts alone and Laya’s tuned policy cost Jev four. Wins require the Pareto front plus three seeds. Uncertainty is mandatory: intervals, or the explicit statement that N is too small for any. A chapter quoting two decimals with no N repeats the exact failure this protocol was written to prevent — and the protocol names the failure it was written from, so nobody has to guess what it costs.

Limitations

• No models ran. All predictions are written, none tested. The prediction table is a promise, not evidence.
• The Red Hat numbers are transcribed, not measured. Full text user-saved and read completely, but the article is HTTP 403 from this machine; nothing here re-runs it. Chapter 28 does.
• Rao is PARTIAL, LLM2Jev is PARTIAL. Rows resting on them cite only what was read (§§3, 5.3, 6–6.2 and §§1–3.3, 4.2 respectively).
• Commenter claims are mapped, never cited. The HN testability table sends jev-sec-bench and Banking77 to Chapter 28, calibration claims to Chapter 9, order-sensitivity to Chapter 7.
• Vendor pages were re-read with no contract changes observed, but pages change; later chapters re-read rather than inherit.
• The Decider name collision (resolved after drafting). AnyJev ships a class of that name; our line was provisionally called Decider when this chapter was written and was renamed arbiter / Arbiter on 2026-10-05.

What Chapter 3 leaves behind

• planning/hypotheses.md — frozen as written, unedited (freeze in commit 5741e1f).
• planning/benchmark-protocol.md — the rules Chapter 4’s harness implements.
• benchmarks/ skeleton — __init__.py and README; the harness fills it in Ch 4.
• examples/ch03-the-jev-controversy/ — claims.py (38 transcribed figures), predictions.py (7 rows), run_ch03.py, selftest_ch03.py, demo_ch03.py, requirements.txt, README.
• results/ch03.jsonl — 56 rows.
• evidence/notes-ch03.md, research/ch03-additions.md, evidence/ledger.md (claims 3.1–3.15).

Which single measurement would change your mind about each hypothesis? For H1: a decision model beating NLI plus first-token readout past noise on unseen labels. For H2: a provider swap that changes no caller code while removing a whole bug class. For H3: the diversity advantage appearing in Chapter 15. For H4: a decision family on the Pareto front where labels churn. For L1–L3: a construct that prevents a failure its library form allows, enforced handling that behaves differently, a compiler that ties the experts. Chapter 4 builds the harness that can run the first of those measurements — boring baselines first, sophistication only if it earns its place.