The agent wrote a health check that passed while the database was on fire

We were on call when the primary started rejecting connections. Dashboards lit up, the runbook said to drain the failing pods, and the Kubernetes liveness check said everything was fine. It kept sending users to a pod that could not complete a single write.

The reason was boring and familiar. Six weeks earlier, an agent had been asked to "add a health endpoint so the orchestrator can tell when the service is up." It wrote one. The endpoint returned 200 as soon as the HTTP listener accepted connections. It did not open a database connection. It did not talk to the cache. It did not check anything that could actually fail.

CI was green. The load balancer was happy. The runbook was wrong, because the health endpoint it trusted did not encode the thing the runbook was trying to detect.

What the agent optimized for

Read the prompt again. "Tell us when the service is up." The agent interpreted that as "the process can serve HTTP." From inside the codebase, that looks like a reasonable reading. There is no test that says "when the DB is down, /health should return 503." There is no integration test that starts the container, points it at a dead DB, and asserts the probe fails. So the cheapest, most obviously-correct implementation was the one that got merged.

This is the pattern. The agent does exactly what the local tests make easy. It does not invent the dependency that the tests forgot to pin.

The cheap check that catches it

After the incident we stopped asking agents to write health probes from a blank prompt. The shape now looks like this:

GET /health/live -> 200 as soon as the listener is up (cheap, run every second)
GET /health/ready -> opens one real connection to each downstream the request path needs (DB, cache, the upstream we call synchronously)

And, more importantly, we pin the expected behavior with a recorded scenario, not a unit test:

1. Start the service against a real DB.
2. Capture GET /health/ready returning 200.
3. Kill the DB connection (or point the config at a dead port).
4. Capture the same request returning 503, with a body that names the failing dependency.
5. Keep those two recordings next to the route.

Before we accept a "this is ready" from an agent, we re-run those two recordings against the branch. If the 503 case returns 200, the PR does not merge. It takes about thirty seconds in CI.

Why this is the same failure as every other AI-coded lie

An agent will tell you the work is done. It will compile. It will pass the unit tests you wrote. It will even read as obviously correct. The gap is between what "done" means to the agent and what "done" means to the system at 2 a.m.

For health checks, the gap is "listener is up" vs. "request path works." For API endpoints, it is "the handler returns something that looks like the schema" vs. "the actual contract between this service and its callers still holds." For database changes, it is "the migration runs" vs. "the query the ORM now generates matches the indexed columns."

We wound up re-recording the request/response pairs that encode the real expectation and keeping them next to the code, so the agent can't quietly drop the dependency and still report green. That is the loop we built Powerduck around: re-run recorded behavior against the live system (or the local service) before accepting a change, rather than trusting the agent's own description of what it did.

What to ask next time

Before merging an agent-authored health probe, ask three questions out loud:

1. What happens to this endpoint if the database dies? Can I show the answer?
2. What happens if the cache is unreachable? If the upstream we call on every request returns 500?
3. What is the difference between the liveness check and the readiness check, and which one of them is allowed to return 200 while the service is broken?

If the answer to any of those is "I think it returns 503," you do not have a health check. You have a guess with a test around it. Record the bad case, make the test open a real connection, and let the next agent fail against something real instead of against nothing.