Retries Are Cheap. Side Effects Are Not.
Reliability · June 3, 2026 · 8 min read
The first reliability feature anyone adds to a background job is a retry. It costs almost nothing to write: catch the error, wait a moment, try again. And it works, most of the time, because most of the time the failure really was a timeout, a connection reset, or a dependency restarting behind a load balancer.
The second feature, usually added several months later and usually the week after an incident review, is the machinery that stops the retry from doing damage. That machinery is the interesting part, and it is rarely designed. It gets bolted on once somebody has been double-charged.
The ambiguity at the centre of it
Here is the whole problem in one sentence: a request that times out and a request that succeeded but whose response was lost look exactly the same from where you are standing.
Your call returned an error. That error tells you nothing about what happened on the other side. The remote system may have rejected the request outright. It may have written the row, dispatched the email, moved the pallet, and then failed to get its acknowledgement back through a network that dropped it. From the caller, these are indistinguishable, and no amount of error-message parsing resolves it.
So a retry is a gamble on which of the two occurred. If it was the first, the retry heals. If it was the second, the retry duplicates. The entire discipline of idempotency exists to make that gamble unnecessary — to make both outcomes converge on the same end state.
Idempotency is a property of the effect
It is common to hear a function described as idempotent. Usually what is meant is that the function does not mutate anything, which is a different and much weaker claim. The property that matters is about the world, not the code: if this step runs twice, does the world end up in the same state as if it had run once?
That question separates steps into three groups, and they want different treatment.
Some steps are naturally idempotent. Setting a field to a fixed value. Writing an object to a known key in a store. Assigning a job to a technician who is already assigned to it. Repeating these is genuinely harmless, and the only thing needed is to make sure nobody quietly changes them into something that increments.
Some steps can be made idempotent with a key. Creating a record, taking a payment, sending a message, issuing a work order. The operation itself is not repeatable, but the decision to perform it can be recorded under a unique identifier, and any subsequent attempt carrying that identifier returns the original outcome rather than performing the operation again.
And some steps cannot be made idempotent at all, because the effect leaves your control entirely. A physical door unlocks. A machine cuts. An email lands in somebody’s inbox. Here the only defence is to reduce the window in which a duplicate is possible and to know that the window still exists.
The key must come from the caller
The single most common mistake in idempotency design is generating the key on the receiving side. If the server invents the identifier, then two attempts carrying the same intent arrive with two different identifiers, and the server has no way to recognise them as the same intent. The key has to travel with the request, and it has to be stable across retries of that request.
Which means it has to be derived from something that does not change when you retry. Not a timestamp. Not a random value generated at the moment of sending — that changes on every attempt, which defeats the point entirely. It should come from the work item itself: the workflow run identifier plus the step name, or the source record plus the operation. Anything that will reconstruct itself identically if the process crashes and the step is picked up again by a different worker.
This is where hosted workflow engines earn their keep. A durable execution system gives each step in each run a stable identity for free, which makes correct key derivation the default rather than a thing each developer has to remember.
Dedupe windows expire, and that is a design choice
Storing every idempotency key forever is possible and rarely done. Systems keep them for a window — hours, or days — and after that the memory is gone. A request arriving with an expired key is treated as new.
That window is a real parameter with real consequences, and it should be chosen against the longest plausible retry chain rather than against storage cost. If a message can sit in a dead-letter queue for two days before an operator replays it, and the dedupe window is twelve hours, then the replay will duplicate. The protection expired before the retry did.
The other half of this is what a duplicate detection should return. Returning an error is tempting and usually wrong: the caller will treat it as a failure and may retry harder. Returning the original result — the same identifier, the same status — lets the caller proceed as if its first attempt had worked, which is exactly the fiction you want to maintain.
Budgets beat backoff curves
Most writing on retries concentrates on the shape of the delay: exponential, with jitter, capped at some ceiling. That matters, and jitter in particular matters, because synchronised retries from many clients turn a brief blip into a sustained overload precisely when the dependency is trying to recover.
But the more useful control is a budget. A retry budget caps retries as a proportion of overall traffic rather than per request. Per-request limits still allow every client to retry three times simultaneously, which triples load on a struggling system. A budget notices that retries have risen above their normal share and starts refusing to spend more of them, which converts a retry storm into a set of honest failures.
The related discipline is knowing which errors are worth retrying at all. A validation error will fail identically on every attempt; retrying it consumes capacity to reproduce a known result. Retries belong to failures that are plausibly transient, and treating every non-success as retryable is how a broken input becomes a permanent load source.
Where to start
If you inherit a system with retries and no idempotency, the useful first move is not to build a framework. It is to write down every step that touches something outside your own database, and mark each one as naturally repeatable, key-protected, or unprotected.
The list is usually shorter than expected and the unprotected column usually contains one or two entries that would be genuinely expensive to duplicate. Those are the ones to fix. Everything else can wait, and a general-purpose idempotency layer built before that list exists tends to protect the steps that never needed it while missing the one that does.