http://programmer.ie/tags/benchmark/ Recent content in Benchmark on Programmer.ie: Modern AI programming Hugo en-us Mon, 07 Sep 2026 12:00:00 +0000 http://programmer.ie/books/embeddings-from-first-principles/13-chapter/ Mon, 07 Sep 2026 12:00:00 +0000 http://programmer.ie/books/embeddings-from-first-principles/13-chapter/ <p><em>Part IV — Measuring the Representation</em></p> <h2 id="the-leaderboard-model-that-lost">The leaderboard model that lost</h2> <p>A team picks the top model on a public embedding leaderboard. It scores well on 50-plus tasks. In their product — retrieval over dense technical documentation with heavy entity and version-number queries — it underperforms a smaller, older model. Nothing was misconfigured. The benchmark measured a population of tasks; the product is one task, and not one the benchmark weighted heavily.</p>