Idempotency is the whole job

It will run twice

Not might. Will. A deploy lands mid-run. A provider sends the same webhook twice because your acknowledgement timed out. An operator clicks the button again because the spinner looked stuck. A queue delivers at-least-once, which is what every queue you can actually buy delivers.

So the design question is never how to prevent the second run. It is what the second run does. If the answer is 'charges the customer again', the system is not finished.

Idempotency keys, done properly

An idempotency key is a deterministic identifier for an intended effect - not a random UUID generated at call time, which produces a fresh key on every retry and defeats the entire mechanism.

Derive it from the business fact: invoice identifier plus attempt number, or order identifier plus the step name. The same intent must produce the same key on every retry, forever.

• Derive keys from business identifiers, never from a timestamp or a random value
• Persist the key with its result before returning, so a crash after the side effect still records it
• Store the response, not just the fact of the call - a retry should return the original result, not an error
• Set a retention window longer than your longest possible retry window

Durable execution beats cron

A cron job that processes a batch has no memory of where it was when it died. It either re-processes everything, which needs idempotency anyway, or it tracks progress in a table that is itself a small unreliable workflow engine you now maintain.

Durable execution inverts this. The workflow's state is persisted at every step by the runtime. A crash, a deploy, or a provider outage suspends the run; it resumes at the exact step it stopped, with the same inputs, and the history of every prior step is queryable.

For Heirview's payment recovery this was the difference between retries that lose their place during a Stripe incident and retries that resume cleanly afterwards.

Retry timing carries information

Fixed-interval retries treat every failure identically, which throws away the most useful signal available. A card reported stolen and a card that hit a daily spending limit are not the same event and should not be retried the same way.

Hard declines should stop immediately - retrying is both futile and, at scale, a signal to the network that you are behaving badly. Soft declines should be retried when they are likely to succeed, which for limit-related declines often means after a billing cycle boundary rather than in four hours.

This is ordinary business logic derived from issuer decline codes. No model required, and none wanted: the rules are knowable and must be right every time.

The dead-letter queue is a product surface

Every automation eventually produces work it cannot complete. The common failure is to log that and move on, which means nobody discovers it until a customer complains.

Failed work belongs in a queue a human can actually work: full context attached, the error in plain language, and a one-click retry or override. Alert on the queue growing, not on individual failures - an operator who receives an alert per event stops reading alerts.