Part IV — Measuring the Representation

The leaderboard model that lost

A team picks the top model on a public embedding leaderboard. It scores well on 50-plus tasks. In their product — retrieval over dense technical documentation with heavy entity and version-number queries — it underperforms a smaller, older model. Nothing was misconfigured. The benchmark measured a population of tasks; the product is one task, and not one the benchmark weighted heavily.

What does an embedding evaluation number measure, and how far does it transfer to the problem in front of you?

The retrieval metrics, briefly

Given a ranked list of candidates with relevance labels:

• Precision@k — fraction of the top k that are relevant. Cares about the slot budget.
• Recall@k — fraction of all relevant items that made the top k. Cares about coverage.
• Reciprocal Rank / MRR — 1 / rank of first relevant. Cares only about getting one right answer high. Good proxy when the consumer is a model reading the first hit.
• nDCG@k — discounted cumulative gain, normalized. Handles graded relevance and position discounting. The default when relevance is not binary.
• MAP — mean average precision. Rewards packing all relevant items early.
• Rank correlation (Spearman/Kendall) — for semantic-similarity-score tasks (STS): does the model’s similarity ordering match human judgments?

Choose the metric that matches how the results are consumed. A RAG system that reads the top passage lives and dies by MRR / Recall@1. A research aggregator that needs everything relevant cares about Recall@k and MAP.

Representation quality vs. application quality

• Representation quality — how well the geometry captures a broad notion of semantic relatedness. Measured by large multi-task suites (STS, clustering, classification-by-embedding, generic retrieval). Portable, comparative, and not your task.
• Application quality — how well the model serves your queries, your corpus, your relevance definition, your consumer, your latency and cost budget.

The gap between them is created by:

• Domain shift. Legal, biomedical, code, and log data are underrepresented in generic benchmarks.
• Relevance definition. “Relevant” on a benchmark may be “same topic”; yours may be “answers this exact question” or “same entity, same year.”
• Query style. Benchmarks often use well-formed questions; your users type three keywords or paste a stack trace.
• Consumer. A human scanning 10 results tolerates noise a model reading 1 does not.
• Asymmetry. Some tasks need query≠document encoders (asymmetric); some benchmarks are symmetric.

Benchmark hygiene

• Contamination. Popular benchmarks leak into training data. A high score may be partial memorization. Prefer a private eval set drawn from your own data.
• Aggregate scores hide variance. A model averaging 62 across 56 tasks might be 40 on retrieval and 80 on classification. Read the per-task breakdown for the tasks like yours.
• Metric–consumer mismatch. A leaderboard sorted by average nDCG is not sorted for a Recall@1 consumer.
• Ceiling effects. On easy benchmarks, top models cluster within noise; the ranking is not meaningful.

Demonstration: build the eval that matters

MEASURED on RELATE, Wave 1 row 1.9 — artifacts experiments/embeddings-from-first-principles/wave1/artifacts/relevance-definition-sweep.json (v0.1) and .../v02/relevance-definition-sweep.json (v0.2 hard queries).

For the RELATE queries, define relevance three ways and evaluate three models (nDCG@10):

                                 all-MiniLM-L6-v2   all-mpnet-base-v2   bge-large-en-v1.5   winner
RELATE v0.1 (near-restatement queries)
  answers the query (grade 3)          0.943             0.949               0.933           mpnet
  substantially supports (>=2)         0.936             0.952               0.939           mpnet
  on-topic (>=1)                       0.936             0.952               0.937           mpnet
RELATE v0.2 (142 hard queries, real lexical gap)
  answers the query (grade 3)          0.838             0.847               0.852           bge-large
  substantially supports (>=2)         0.851             0.875               0.880           bge-large
  on-topic (>=1)                       0.851             0.875               0.880           bge-large

MEASURED. On RELATE v0.1 the ranking did not flip — all-mpnet-base-v2 wins under every definition, all three models within 0.02 nDCG@10, because v0.1’s queries are near-restatements of their answers and nothing separates the models. Rebuild the query set with a real lexical gap to the answer (RELATE v0.2, 142 hard queries) and nDCG@10 drops to 0.84–0.85 and the winner flips to bge-large-en-v1.5. So the query set, not the models, was the ceiling — and the winning model does change once the queries are hard enough to discriminate. (The winner still does not change with the relevance definition itself on RELATE, because the three definitions are highly correlated on this corpus; a corpus with more grade-1/grade-2 material would show that too.) The lesson is unchanged and sharpened: a single leaderboard number is the wrong basis for selection, and a private eval whose queries are as hard as your product’s is what actually reorders models.

What this chapter establishes and what it does not

Establishes: the standard retrieval and similarity metrics and which consumer each fits; the representation-quality / application-quality distinction and the five forces that open the gap; benchmark hygiene (contamination, aggregate variance, metric–consumer mismatch, ceilings).

Does not establish: that public benchmarks are useless (they are excellent for coarse filtering), or a single metric to rule them all. It establishes that model selection requires a private eval set with your relevance definition and your consumer’s metric.

Lab 13: your private eval set

PROPOSED, not executed.

Setup. 100–300 real queries from your logs. Label candidates with your relevance definition, written down explicitly. Pick the metric matching your consumer.

Task.

1. Evaluate 3–5 candidate models on this set with your metric plus one alternative.
2. Also record: p95 latency, cost per 1M tokens, embedding dimension, max sequence length.
3. Compare your ranking to the public leaderboard ranking.
4. Break the score down by query type (keyword / question / entity / long).

Success criterion. A selection with a one-paragraph justification that cites your relevance definition and your consumer — and a note on whether the leaderboard would have chosen the same model.

Companion component: the evaluation card

evaluation_card:
  eval_set:          {name, n_queries, source: "private / public", date}
  relevance_def:     <written definition, not "relevant">
  metric:            <primary + rationale tied to consumer>
  per_type_scores:   {keyword: .., question: .., entity: .., long: ..}
  operational:       {p95_latency, cost, dim, max_seq_len}
  contamination_risk: <low if private>

The Observatory stores an evaluation card per (model, task) and never ranks models by a metric that does not match the registered consumer.

Failure modes

• Picking by leaderboard average. It is a population statistic; you have one task.
• Undefined “relevance”. If it is not written down, different labelers and different models are being scored against different targets.
• Metric–consumer mismatch. Optimizing nDCG for a Recall@1 consumer.
• Ignoring operational cost. A 2-point quality gain at 5× latency and cost is often the wrong trade.
• Trusting scores on contaminated benchmarks. Use private data for the final decision.

What this chapter established

• Retrieval/similarity metrics and the consumer each fits (MRR/Recall@1 for a model reading one hit; Recall@k/MAP for exhaustive needs; Spearman for STS).
• Representation quality is portable and comparative; application quality is yours — the gap is opened by domain, relevance definition, query style, consumer, and asymmetry.
• Benchmark hygiene: contamination, hidden variance, metric–consumer mismatch, ceilings.
• Changing the query difficulty reorders models: RELATE v0.1’s near-restatement queries pin all three at 0.93–0.95 nDCG@10 with mpnet on top; RELATE v0.2’s hard queries drop them to 0.84–0.85 and put bge-large on top (row 1.9). Changing the relevance definition did not reorder them on RELATE (the definitions are too correlated) — but the broader lesson holds: an eval whose queries do not resemble your product’s cannot separate models for you.
• The evaluation card: private eval set, written relevance definition, consumer-matched metric, operational costs.

Next

Every evaluation so far reported a score. The next chapter attacks the score itself: a cosine of 0.81 means nothing until you know the distribution it came from, and the positive and negative distributions can overlap almost completely.