Use with AI · Chapter Application

Audit a Learned Scorer Before You Trust Its Score

Apply Models From First Principles Step 01 to a real repository by reconstructing what a scalar quality model is actually learning, what evidence reaches it, and whether its score is meaningful.

Chapter Application Models From First Principles Learned scorers, rankers and quality models Intermediate
Read the source chapter → View the learning path →

How to use this

  1. Open a repository-aware AI assistant.
  2. Give it access to the repository or files you want reviewed.
  3. Copy the prompt below and run it unchanged first.
  4. Use the evidence it finds to decide what to inspect or change next.
PromptCopy and run against your own project
You are reviewing a learned scoring model in this repository.

Do not begin by proposing a larger architecture. First reconstruct the scoring contract the model currently implements.

Inspect the repository and identify:
1. every input that reaches the scorer;
2. how those inputs are encoded or embedded;
3. where representations are combined;
4. the scalar target being predicted;
5. how training labels or preferences are produced;
6. the loss function and weighting;
7. how the score is consumed downstream;
8. the baseline or non-learned alternative, if one exists.

Build a scorer contract with:
- semantic meaning of each input;
- tensor/representation boundary;
- target definition;
- valid score range;
- calibration assumptions;
- downstream decision threshold or ranking behavior.

Then test the following failure modes:
- target leakage;
- proxy labels that do not match the downstream decision;
- duplicate or redundant features;
- scale mismatch between embeddings;
- hidden dependence on length, frequency or other shortcuts;
- train/inference feature mismatch;
- score used as probability without calibration evidence;
- one scalar being asked to represent several incompatible notions of quality.

For each finding provide exact file/symbol evidence and classify it as confirmed defect, plausible risk, or unsupported hypothesis.

Do not recommend adding layers, attention, recurrence or another model unless the current scorer fails a measurable requirement that the added mechanism is intended to solve.

Output:
1. Scorer contract
2. Training-target reconstruction
3. Downstream score usage
4. Evidence-backed findings
5. Shortcut/leakage risks
6. Simplest viable baseline
7. Smallest experiments needed
8. Recommendation: Keep / Simplify / Change

A score is useful only if its learned target matches the decision the system later makes.