Distributed-Systems on Programmer.ie: Modern AI programming

Build a Production AI Agent From First Principles: The Complete Reference Architecture

Sun, 09 Aug 2026 16:09:00 +0100

Build a Production AI Agent From First Principles: The Complete Reference Architecture

We have spent this series adding mechanisms only when a specific failure demanded them.

We started with a model call.

Then we added candidate generation, critique, planning, tool use, memory, search and verification.

Then the system stopped looking like a clever prompt.

It started looking like software.

Then distributed systems problems arrived:

duplicate work,
retries,
leases,
fencing,
backpressure,
dependency failure,
behavioral drift,
release compatibility,
replay,
incident forensics,
SLOs,
authority,
competence,
capability acquisition,
placement,
handoff,
stale state,
stale intent,
commitments,
durable workflows,
transaction recovery,
trust boundaries,
multi-agent coordination,
and finally an explicit control plane.

At this point the architecture is complete enough that adding another isolated mechanism would make the series worse rather than better.

Advanced Agents From First Principles 22: What Happens When One Dependency Starts Failing? Add Circuit Breakers, Bulkheads and Graceful Degradation

Sun, 09 Aug 2026 11:26:00 +0100

What Happens When One Dependency Starts Failing?

An advanced agent platform can be working perfectly and still collapse.

Not because the planner became confused.

Not because the router chose the wrong model.

Not because MCTS explored the wrong branch.

Because one dependency became unhealthy.

Maybe:

the frontier model starts returning 503s,
the browser pool starts timing out,
the embedding service slows from 80 ms to 8 seconds,
the verifier database starts dropping connections,
the search API starts rate limiting,
a sandbox cluster stops accepting jobs,
or an external provider is technically up but so slow that every request consumes a worker for minutes.

Then the retries begin.

Advanced Agents From First Principles 21: What Happens When Too Many Agents Compete for the Same Resources? Add Admission Control, Quotas and Backpressure

Sun, 09 Aug 2026 11:21:00 +0100

What Happens When Too Many Agents Compete for the Same Resources?

A single agent can look healthy in isolation.

It gets a request.

It calls a model.

It launches a few search branches.

It opens a browser.

It runs tests.

It asks a verifier to check the result.

Everything works.

Then production traffic arrives.

Ten agents start at once.

Then fifty.

Then five hundred.

Now every agent still has a perfectly reasonable local plan.

Advanced Agents From First Principles 20: Can Your Agent Coordinate Across Machines Without Duplicating Work? Use Leases, Idempotency and Fencing

Sun, 09 Aug 2026 11:13:00 +0100

Can Your Agent Coordinate Across Machines Without Duplicating Work?

A single-process agent can already be complicated.

It can plan.

It can search.

It can launch speculative branches.

It can cancel losing work.

It can verify outcomes.

Then you move that work onto multiple workers.

Now a new class of failure appears.

Two workers both believe they own the same task.

One worker pauses for thirty seconds.

Another worker assumes it died and takes over.

Advanced Agents From First Principles 26: Why Did the Agent Fail? Build an Incident Forensics Pipeline