The agent retried the whole task after one failed tool call and duplicated the side effects

A recurring job fetched an unpaid order, charged the card through the gateway, then sent a receipt email. On one run the payment call timed out at the network edge. The wrapper caught the exception, waited a moment, and retried the job from the first line. This time the charge went through, the receipt sent, and the run reported success. The customer's card, however, had already been charged once, and the inbox held two receipts.

The first attempt had gotten far enough for the gateway to accept the charge before the response was lost. The retry replayed every completed step instead of resuming after the failed one, so a transient timeout that never reached the server became two real side effects.

Why a full-task retry looks like resilience

A loop that retries on exception reads safe at a glance: it has a backoff, a bounded attempt count, and a final success status. The run log shows the job completing on attempt two, and nothing in that log marks which steps had already landed on attempt one. The retry wrapper itself has no view into side effects that happened before the failure, so replays look indistinguishable from a clean start.

Tests reinforce the illusion. They run the job against a mock that resets between attempts, so the second pass finds no prior charge, no prior row, and no prior email. A mock world that starts fresh makes replaying completed steps harmless; the real world does not reset.

What ships when retry wraps the whole task

• Duplicated side effects. A charge, an email, a row insert, or a webhook sent on attempt one repeats on attempt two; the caller pays or receives twice.
• Unknown commit state after a lost response. A timeout proves the reply did not arrive, not that the server rejected the call. Replaying blindly can land a request that already succeeded.
• The success status hides partial completion. Because the run eventually finishes, monitoring marks it healthy, and the duplicate surfaces only through a customer report.
• Bound retries do not bound harm. Three attempts against a non-idempotent external call mean up to three side effects, not one.
• Progress is lost on restart. Without a checkpoint, recovery has to decide from scratch what already ran, which is exactly what the wrapper fails to track.

The tell is a retry that wraps the entire workflow rather than the single failing call. If the task contains any step with a durable side effect, restarting from the top is not retries; it is replay.

Retry the failed step, make the call idempotent

Backoff and attempt limits are necessary but not sufficient. Separate the flow so recovery resumes where it stopped:

• Split steps and retry only the failed one. Fetch, charge, and notify are distinct phases; on a timeout in charging, re-attempt the charge rather than re-fetching and re-notifying the whole job.
• Give each side-effect call an idempotency key. A client-generated key sent on the charge or create request lets the gateway collapse a repeated submission into one action, so a lost response and a retry cannot double-write.
• Checkpoint progress instead of replaying. Record which step completed before the failure; on recovery, read the checkpoint and continue from the next uncommitted step.
• Distinguish never-reached from already-committed. On a timeout, query the upstream for the request outcome before assuming it failed; a call that landed should be observed and skipped, not retried blind.
• Keep the documented request shape honest. When the agent drives an outbound call, the idempotency header and payload shape should match what the live endpoint accepts; we keep that check local in Powerduck, re-running the documented request against the live endpoint instead of trusting a handwritten example that drifted.

Resilience means resume, not restart

A retry loop over a whole task only protects steps with no consequences. For anything that writes, charges, or sends, the safe boundary sits around the individual call, with an idempotency key and a checkpoint behind it. The goal is to repeat the failed action exactly once on an already-committed world, not to rerun the journey from the beginning.