Observability on Programmer.ie: Modern AI programming

Advanced Agents From First Principles 23: Your Infrastructure Is Healthy. Why Is the Agent Getting Worse? Detect Behavioral Drift and Roll Back Safely

Sun, 09 Aug 2026 12:00:00 +0100

Your Infrastructure Is Healthy. Why Is the Agent Getting Worse?

Your dashboards are green.

The model endpoint is responding.

The browser workers are alive.

The database is healthy.

The queue is draining.

The verifier service is up.

Latency has not exploded.

There are no obvious exceptions.

And yet the agent is getting worse.

It fixes fewer bugs.

It retrieves weaker evidence.

It escalates to expensive models more often.

It chooses the wrong tools more frequently.

Advanced Agents From First Principles 13: How Do You Debug an Agent That Made the Wrong Decision? Add Trajectory Observability

Sun, 09 Aug 2026 10:18:00 +0100

An advanced agent fails.

You look at the final answer.

It is wrong.

So you inspect the prompt.

The prompt looks reasonable.

You inspect the model response.

That also looks reasonable.

But somewhere between the original request and the final result the system:

chose the wrong specialist,
pruned the branch that contained the right solution,
trusted a critic that was wrong,
escalated to an expensive model unnecessarily,
failed to escalate when it should have,
retrieved stale memory,
spent most of its budget exploring duplicates,
accepted a weak verifier signal,
retried the same strategy under a different name,
or transformed a local success into a global failure.

The final answer does not tell you which one happened.

Advanced Agents From First Principles 25: Can You Reproduce an Agent Run Months Later? Add Deterministic Replay and Provenance

Sun, 09 Aug 2026 00:00:00 +0000

A production agent fails on Tuesday.

You inspect it on Wednesday and cannot reproduce the failure.

Three weeks later someone asks a harder question:

What exactly did this agent see, which release was running, which tools were called, which evidence was used, and why did the verifier accept the result?

If the answer is “we have some logs”, you do not yet have reproducibility.

You have fragments.

Advanced agents accumulate hidden variability quickly:

Advanced Agents From First Principles 26: Why Did the Agent Fail? Build an Incident Forensics Pipeline

Sun, 09 Aug 2026 00:00:00 +0000

A practical incident-forensics workflow for advanced agents: reconstruct the run, find the earliest divergence, distinguish root cause from downstream symptoms, measure blast radius, and prove that a remediation would have prevented the incident.

Advanced Agents From First Principles 27: How Reliable Does an Agent Need to Be? Define SLOs and Error Budgets

Sun, 09 Aug 2026 00:00:00 +0000

A practical reliability framework for advanced agents: define verified-success SLOs, false-success ceilings, UNKNOWN budgets, latency and cost targets, then use error-budget burn to decide when to ship capability and when to stop and harden the system.