Appendix — Book Quality Lab: A Pi Agent with a Judgment Kernel
Run a read-only manuscript evaluator, inspect exact source evidence, and separate Pi analysis, typed judgment, policy and your editorial decision.
You can ask a language model whether a chapter is good. You then have an opinion, usually expressed with more certainty than its supporting evidence deserves. Book Quality Lab gives you something narrower and inspectable: a candidate concern, exact passages, a bounded question, a provider answer, a policy outcome and a place to record your own decision.
This appendix is an executable supplement. It preserves the verdict in Programming with Decisions. The offline implementation works, real-model inference did execute through the configured provider, and the comparative experiment returned a valid negative result: the typed judgment layer did not demonstrate superior accuracy. That finding is part of the result, not a reason to discount it.
The central conclusion of this capstone is:
A typed decision architecture does not automatically make an AI’s judgments better. Its demonstrated value here is in making decisions explicit, validated, traceable, recoverable and independently reviewable. Whether those decisions are more accurate remains an empirical question.
Run the finished offline path
From your Jev checkout, use the isolated examples package:
cd examples/book-quality-lab
npm ci --ignore-scripts
npm run setup
npm test
node cli.mjs evaluate fixtures/book --whole-book --provider rules --analysis pi-fixture
The setup command obtains the pinned μ source checkout; it does not download a model. With dependencies and source present, this pilot runs offline. The package README also shows how to select an existing μ checkout instead.
Pi executes its actual agent loop with a scripted provider. Its snapshot tool reads real fixture files through the host’s immutable inventory. Host rules prepare comparison candidates. μ’s actual decision engine validates and records the rule provider’s answers. This is observed software behavior, not evidence that an AI can edit or judge an unfamiliar book well.
The recorded acceptance suite passed 42 tests. It checks strict contract
validation, wrong spans, changed sources, unavailable judges, shadow behavior,
provider substitution, partial failures, human labels, read-only active mode,
checkpoint resume, invalid decision targets and the real-model contract
probes. A synthetic HTTP server also exercises the local adapter’s response,
truncation and redirect handling. Those transport tests are not model
observations. The historical measurement record is results/ch36.jsonl,
metric = software_tests; the 02B diagnostic snapshots live in separate
results/ch36-capstone02b-*.jsonl files and never replace it.
You can use the mechanical evaluator without μ or a model:
node cli.mjs evaluate C:/books/jev --chapter 01
node cli.mjs evaluate C:/books/jev --whole-book
node cli.mjs evaluate C:/books/pi --whole-book
You may instead supply the specific content/books/<book> directory. The
CLI discovers its repository root, chapter files, concepts companions and
chapter metadata. A whole-book run includes supporting Markdown pages, but
does not impose numbered-chapter ordering rules on those pages.
Follow one finding through the boundaries
The fixture manuscript says:
Acceptance means permission to publish.
Its concepts companion says:
Acceptance means not permission to publish.
Both quotations are exact source lines in the recorded fixture pilot. A concepts companion is another source to compare, not an authority that wins automatically. You need both locations before you can assess the discrepancy.
flowchart TD
A[Manuscript and companions] --> B[Host snapshot and mechanical checks]
B --> C[Pi reads and proposes candidates]
B --> D[Rule-based comparison candidates]
C --> E[Host verifies exact evidence]
D --> E
E --> F[Bounded typed decision in mu]
F --> G[Provider answer and original ledger]
G --> H[Shadow fallback or active categorization]
H --> I[Report and human review ledger]
The specification asks whether these passages use the concept compatibly. The provider in the offline pilot recognizes only a narrow same-scope definition-and-negation template. It is a rule, not a model. The runtime is μ’s decision engine. The evidence includes exact excerpts, line ranges, original-byte hashes and the returned answer. The action is to retain a review item. No layer receives permission to rewrite your manuscript.
The code is in examples/book-quality-lab: manuscript.mjs owns inventory,
parsing and evidence checks; analysis.mjs injects Pi’s snapshot tools;
kernel.mjs defines bounded questions; evaluate.mjs records results and
reports; cli.mjs exposes the workflow. The 02B contracts are
contracts-v2.mjs (strict versioned schemas), context-v2.mjs (exact source
contexts), structured.mjs (validated tool submission) and quality-v2.mjs
(bounded review plus quality.source_findings version 3, which explicitly
targets the original manuscript assertion, never the reviewer critique).
SCHEMA.md specifies the artifacts.
The execution pattern connects to Pi Agents: Pi Patch Desk: the agent performs bounded work, the host owns verification and the human owns acceptance. This capstone does not replace Pi Patch Desk or change the Pi book’s dependency pin.
Inspect the decision before trusting the report
The implementation imports defineDecision, DecisionEngine, Judge and
the ledger contract directly from the
pinned μ source.
The private package is not assumed to exist on the public npm registry.
The source pin and imported-file hashes are checked before runtime import.
No μ application fork is embedded in this book.
The recorded pilot ledger contains this projection of a decision:
{
"specId": "quality.definition_consistency",
"specVersion": 1,
"mode": "shadow",
"providerId": "rules:book-quality-v1",
"modelId": "rules-v1",
"outcome": {"status": "NEEDS_REVIEW"},
"source": "fallback",
"reason": "shadow",
"judged": {"verdict": "potential_conflict"},
"answers": {"verdict": {"type": "choice", "choice": "potential_conflict"}}
}
You can inspect the full record in
evidence/book-quality-lab/pilot-release/mu-ledger.jsonl. It includes the state,
question, exact passages, digest and original ledger identity. The wrapper
in judgments.jsonl links that identity to its candidate finding.
Notice the difference between judged and outcome. Shadow records the rule’s
conflict answer while retaining the baseline review policy. Active mode may
categorize a finding using the answer, but still has no edit operation and
never deletes a finding. An abstention or failure remains unresolved.
There is no reported probability in this answer. The wrapper records null, not an invented confidence. Ordinal clarity answers are positions on a rubric, not calibrated probabilities. The supplied-source claim statuses describe support, contradiction or unresolved verification; they never certify external truth.
The audit found two integration details worth preserving. μ’s current LLM
adapter supplies choice probabilities when a response omits them, so this
capstone instead uses a strict local completion adapter. μ’s Judge initializes
unreported usage counters to zero; the original ledger is preserved, while
the wrapper marks missing provider usage UNKNOWN for cost accounting. These
are source-inspected implementation behaviors, not Jev API guarantees. See
research/ch36-additions.md for the reuse map and license attribution.
Read the measured results at their actual scope
Policy correctness
The 02B experiment reproduced a real policy/code mismatch: the written refund
runbook permitted a request that predicate_decide declined. A 108-case
boundary grid showed 6 mismatches before the repair and 0 after
(results/ch36-capstone02b-policy.jsonl, metric = policy_contract_repair).
The owning Chapter 1 implementation was corrected; historical bytes
(tickets-before.py) and the original Chapter 1 measurements are preserved.
The regression is examples/ch01-generate-vs-decide/test_policy_contract.py
(3 tests, 108 subtests). Scope is the literal documented policy, not a claim
about correct real-world refund behavior.
Contracts and tests
Real-model outputs pass through strict versioned schemas. Invalid, truncated
or extra-field responses are rejected, never coerced; every attempt is
retained in attempts.jsonl with its status (VALID, INVALID_RESPONSE,
INVALID_TOOL_SUBMISSION, PROVIDER_FAILURE, MISSING_CAPABILITY kept
distinct). Abstention, invalid response, provider failure, missing capability
and deterministic fallback are never conflated. All 42 automated tests
pass, including real-provider development contract probes that gate any book
execution.
Benchmark (diagnostic, not confirmatory)
Sixteen opaque-ID diagnostic cases compared four approaches on identical source contexts (B and C receive byte-identical inputs; labels stay outside every model context):
| Approach | Valid result |
|---|---|
| A — structural only | Abstains on semantic families (by design) |
| B — direct fixed-question review | 16/16 |
| C — Pi candidate generation plus μ judgment | 14/14 valid (2 invalid decision targets excluded, see below) |
| D — narrow explicit-template rule | Mostly abstains; 2/4 on definition_consistency |
Two claim-status C cells were excluded as INVALID_DECISION_TARGET: the
typed answer supported a reviewer critique, which the scorer first mapped to
support of the manuscript assertion. The raw answers are preserved;
decision-target-audit.json records the exclusion, and
quality.source_findings version 3 now targets the original manuscript
assertion explicitly. The valid-cell figures (16/16 vs 14/14) use different
denominators and are not an accuracy claim. Reference labels are
agent-authored diagnostics, not independent human gold. No quality
improvement is demonstrated. The judgment path measured approximately
2–3× the latency of direct review in this comparison.
The read-only book scans below ran before this appendix was added. Their exact
source hashes are retained in the run manifests. Every count in this table
comes from results/ch36.jsonl, metric = manuscript_evaluation, with the
matching run name.
| Run | Documents scanned | Mechanical issues | Comparison candidates | Findings with exact source matches |
|---|---|---|---|---|
jev-final |
35 | 2 | 0 | 2 |
pi-final |
46 | 0 | 2 | 2 |
pilot-release |
2 | 1 | 2 | 3 |
chapter-final |
1 | 0 | 0 | 0 |
The Pi scan includes a supporting page alongside its numbered chapter files. The Jev issues are duplicate YAML mapping keys in companion metadata. The Pi candidates are exact repeated passages whose usefulness still needs an editorial decision. An exact match confirms the quotation; it does not confirm that repetition is needless. Both book scans recorded unchanged source bytes. Their semantic-model stage is explicitly NOT_RUN.
The protocol is EXPLORATORY. The experiment contract was written before execution, but the author’s instruction prohibited commits. It therefore does not meet the book’s committed-preregistration rule. No promotion of a book-level hypothesis follows from these results.
Coverage (partial by record)
Whole-book semantic evaluation is incomplete and reported as such:
| Book | Chapter pages completed (best run) | Cross-chapter pairs |
|---|---|---|
| Jev | 7/87 | 24 sampled (not exhaustive) |
| Pi | 0/83 | 24 sampled (not exhaustive) |
Structural checks ran (Jev 809, Pi 1093); metadata receives deterministic
checks only. Completed units mean receipt plus a valid bounded submission —
not proof of comprehension or absence of defects. Interrupted runs
(capstone02b-20261111, closure-v21) retain their failures; no partial run
is presented as complete.
Bring your own provider (bounded)
When you have a running, tool-capable OpenAI-compatible local server, select
its endpoint and model explicitly. The exact command and replacement arguments
are documented in the package README. --analysis pi-local --provider local
uses Pi to prepare candidates and μ to judge bounded questions;
--analysis direct-local bypasses Pi for direct review.
The 02B real-model study instead used the configured
opencode-go/deepseek-v4.1-flash Pi transport with frozen code/context
hashes before dispatch (EXPERIMENT02B.md, experiment02b.mjs). It is not
rerun here: the exposed diagnostic cases must not become new held-out tests,
and whole-book reruns would need fresh held-out labels plus a new freeze.
Only loopback HTTP is supported. The program refuses redirects and hosted endpoints, imposes request and reading budgets, and records partial failures. It does not discover credentials, install weights or silently select a different provider. Missing model revision and usage remain UNKNOWN. Served revision and billed cost were never observed; catalog estimates are not invoices.
Each family starts in shadow. A proposed promotion rule requires independent author labels, a committed comparison protocol and satisfactory per-family precision, recall, abstention and cost evidence. One successful family does not validate the others. That proposed criterion has not been met here.
Make your editorial decision durable
Open report.md, follow a finding’s evidence spans, and compare its raw judge
answer with the policy outcome. Then use the CLI’s review command to record
ACCEPT, REJECT, DEFER or INVESTIGATE against the printed finding ID. The package
README gives the commands; the acceptance suite demonstrates both acceptance
and rejection with synthetic review labels. Those test labels are not your
editorial judgments.
The review ledger preserves rejected findings. Each label belongs to a source
revision, rubric and task. If you change a chapter, verify marks affected
findings stale; a later evaluation creates new evidence. --resume continues
an identical run, while --previous can reuse unchanged chapter analysis in a
new run. Neither command overwrites the old source record.
What you have established is a useful software boundary: exact evidence, interchangeable bounded providers, explicit uncertainty and an inspectable policy before an editorial action. What you have not established is that a decision model improves manuscript judgment. The evidence supports publishing this as an exploratory case study with a negative benchmark result — and that is a result worth keeping.