The Retry After a Timeout Created Two Records Because the First POST Had Already Landed

The agent was asked to make a flaky outbound POST reliable, so it added an automatic retry on timeout: wait, re-send, stop after three attempts. The test suite mocked the server as always failing on the first call, passed the retry on the second, and reported green. In production the first POST had actually committed while the client was still waiting on a slow response, and the retry wrote a second record.

The core mistake is treating a timeout as proof that nothing happened. A timeout only proves the client did not hear a response back within its window. The server may have parsed, validated, stored, and responded just after the cutoff.

Three outcomes the client cannot see at the cutoff

At the timeout	Server state likely	Blind retry does
Request still queued, nothing committed	Safe to retry	Correct
Request committed, response in transit	Unknown	Likely duplicates
Request rejected after validation	Safe to retry	Correct

A synthetic retry fixture collapses all three into one: first call errors before any state, second succeeds. That shape matches a connection refused or a dropped packet, but not a committed-but-unanswered POST, which is the expensive case.

Why POST is the wrong place for plain retry

A retry that re-sends the same body works cleanly when the call can be repeated without effect: a GET, or a PUT/PATCH keyed to a fixed record. A POST that creates a new record by definition changes state each time it lands. Sending it twice asks for two records unless the receiver can tell the two attempts apart.

Agents tend to reach for retry as a one-size reliability fix because it reads as discipline: attempt, back off, try again. The missing half is asking whether a second attempt is even safe, and that depends on the method and the endpoint, not on the error message alone.

Make the second attempt indistinguishable from the first

The fixes that actually hold up in traffic:

• Attach an idempotency key generated once per logical request, not per attempt. The receiver stores the first result keyed to it and returns that same result on a repeat instead of creating a second record.
• Retry only on failures that are clearly pre-commit: connection refused, unreachable host, explicit 503 retry-after, or a hard network error before the request was sent.
• On a timeout after bytes were transmitted, do not resend blind. Query the resource or read back state to see whether the first call landed, then continue or compensate.
• Back off with jitter and cap attempts, but treat the count as a safety ceiling, not permission to repeat a mutating call.
• Move duplicate protection server-side where the truth lives; the client cannot know what committed.

For endpoints that cannot accept an idempotency key today, the honest interim step is to stop auto-retrying the POST on timeout and surface the ambiguous case for a check instead of guessing.

Write the fixture that already committed

Review retry code by asking what happens when attempt one succeeded but went silent. A test that only simulates pre-send failure will keep endorsing the duplicate path. Have the agent add a case where the first call stores state and the response is dropped, then assert the second attempt returns the existing record rather than creating a new one.

Reliability does not mean every failed-looking call goes out again. It means repeating only what repeating cannot harm, and resolving the ambiguous middle before resending a mutation.

Where does your system currently retry a mutating call on timeout, and is there any key that tells the receiver the attempts are the same request?