http://programmer.ie/tags/leakage/ Recent content in Leakage on Programmer.ie Hugo en-us Fri, 28 Aug 2026 11:15:00 +0100 http://programmer.ie/books/dspy-from-first-principles/16-chapter/ Fri, 28 Aug 2026 11:15:00 +0100 http://programmer.ie/books/dspy-from-first-principles/16-chapter/ <h1 id="dont-let-the-optimizer-cheat">Don&rsquo;t Let the Optimizer Cheat</h1> <p>Chapter 15 expanded the program&rsquo;s access. It can search repositories, retrieve memory, use tools, and maybe explore large contexts programmatically.</p> <p>That power threatens the experiment.</p> <p>The central question:</p> <blockquote> <p>How can an optimization experiment appear valid while the program or optimizer has quietly gained access to the answer?</p></blockquote> <p>Cheating is often accidental. A field named <code>validation_result</code> slips into generation inputs. A memory tool returns the historical accepted patch. A retriever indexes a later repository revision. A GEPA feedback string includes the exact gold answer. A developer inspects holdout failures, changes instructions, and evaluates on the same holdout again.</p>