Hard Negatives

Mon, 07 Sep 2026 11:20:00 +0000

Part III — Retrieval Is an Experiment

Two benchmarks, same model, opposite verdicts

Benchmark E (easy negatives):
  positive: the correct answer
  negatives: 20 passages drawn at random from the corpus
  → model scores Recall@1 = 0.94

Benchmark H (hard negatives):
  positive: the correct answer
  negatives: the 20 passages most similar to the query that are NOT correct
  → same model scores Recall@1 = 0.55

Nothing changed but the negatives. The easy benchmark asked “can the model tell the answer from an unrelated paragraph?” The hard benchmark asked “can it tell the answer from the 20 things that look most like the answer?” Those are different questions, and only the second is the one production faces.

Retrieval Evaluation on Programmer.ie: Modern AI programming

Hard Negatives

Two benchmarks, same model, opposite verdicts