http://programmer.ie/tags/thresholds/ Recent content in Thresholds on Programmer.ie: Modern AI programming Hugo en-us Mon, 07 Sep 2026 12:10:00 +0000 http://programmer.ie/books/embeddings-from-first-principles/14-chapter/ Mon, 07 Sep 2026 12:10:00 +0000 http://programmer.ie/books/embeddings-from-first-principles/14-chapter/ <p><em>Part IV — Measuring the Representation</em></p> <h2 id="what-is-081">What is 0.81?</h2> <p>A pipeline decides two documents are &ldquo;duplicates&rdquo; if their cosine exceeds 0.8. Someone picked 0.8 because it looked reasonable. Then:</p> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>distribution of cosine for KNOWN duplicate pairs: mean 0.79, sd 0.09 </span></span><span style="display:flex;"><span>distribution of cosine for KNOWN non-duplicate pairs: mean 0.71, sd 0.11 </span></span></code></pre></div><p>The threshold sits inside both distributions. A true duplicate at 0.78 is rejected; an unrelated pair at 0.82 is accepted. The number 0.81 does not carry its own meaning — it is a point on two overlapping bell curves, and without those curves it is noise.</p>