The previous Note left a request in an uncomfortable state:
CLIENT
sent CreateOrder
received no response
The server might have:
- never received the request;
- received but rejected it;
- started processing it;
- committed the order;
- sent a response that was lost.
The caller knows that the interaction did not complete from its perspective.
It does not necessarily know the business outcome.
The tempting response is:
Try again.
Sometimes that is exactly right.
A connection may have failed before the request reached the server. A service instance may have restarted. A database transaction may have been selected as the victim of a deadlock. A short-lived network problem may have disappeared.
Another attempt may succeed.
But “try again” is not a complete recovery strategy.
The repeated operation might:
- create a duplicate;
- increase load on an overloaded service;
- repeat an external side effect;
- encounter the same permanent error;
- succeed while earlier damage remains;
- hide an unresolved uncertain outcome.
This gives me a useful distinction:
A retry is another attempt to perform an operation. Recovery is the process of returning the system and its business state to an acceptable condition.
A retry can be one recovery action.
It is not the same thing as recovery.
1. A retry is only another attempt
Suppose a catalog request fails because one service instance restarts.
attempt 1
│
└── connection reset
attempt 2
│
└── product returned
The second attempt replaced the failed call with a new call.
The application did not continue the first network exchange. It began another one.
LOGICAL OPERATION
GetProduct chair-42
│
├── network attempt 1
└── network attempt 2
That distinction matters because the server may observe both attempts.
Even if the client thinks:
I am still trying to perform one operation,
the receiver may see two separate requests.
A retry policy therefore needs to understand the relationship between one logical operation and several execution attempts. Without that relationship, another attempt may become another effect.
Recovery has a larger target
Suppose a payment request times out.
The application retries using the same idempotency key and receives the original successful result. That may recover the individual operation:
payment outcome:
authorised
But imagine that while payment was uncertain:
- the order remained stuck in
payment_pending; - inventory stayed reserved;
- a confirmation message was never published;
- the customer submitted another order;
- an operator opened an incident.
Returning the payment result does not automatically resolve those consequences.
Recovery might still require:
- updating the order state;
- releasing or confirming inventory;
- publishing missing events;
- reconciling duplicate orders;
- notifying the customer;
- verifying that downstream consumers caught up.
So recovery asks a wider question:
What must be true before this system can be considered healthy and the operation can be considered resolved?
That answer may include much more than obtaining one successful response.
2. When retries help and when they hurt
Retries are most naturally useful when the failure is transient and the operation is safe to repeat.
A transient fault is a condition expected to disappear without changing the request itself. Examples include:
- one service instance restarting;
- a brief loss of network connectivity;
- temporary leader election;
- a short-lived lock conflict;
- a momentary capacity limit;
- a database deadlock victim.
The operation may be valid while the environment is temporarily unable to complete it.
VALID OPERATION
│
▼
TEMPORARY CONDITION
│
▼
ATTEMPT FAILS
│
▼
CONDITION CLEARS
│
▼
LATER ATTEMPT SUCCEEDS
Even here, I need to ask:
- Is the failure genuinely transient?
- Is the operation safe to repeat?
- How long should I wait?
- How many attempts are justified?
- Is the dependency recovering or becoming more overloaded?
A transient-looking status is evidence for a retry decision. It is not permission to loop forever.
Permanent failures need a changed condition
Suppose a request is rejected because:
- the customer ID is invalid;
- the caller lacks permission;
- the product does not exist;
- the request violates a business rule;
- the schema is unsupported.
Repeating the identical request does not alter the cause.
attempt 1:
permission denied
attempt 2:
permission denied
attempt 3:
permission denied
The system has not recovered. It has performed the same unsuccessful work several times.
Recovery requires changing something:
- correct the request;
- use valid credentials;
- wait for an explicit business-state change;
- repair configuration;
- deploy compatible code;
- ask a human to resolve the conflict.
| Condition | Can the same logical operation later succeed? | Likely response |
|---|---|---|
| Brief network disruption | Often | Bounded retry |
| Deadlock victim | Often | Retry the complete transaction |
| Invalid request | No, not unchanged | Correct the request |
| Permission denial | Usually not | Change identity or policy |
| Outcome unknown | It may already have succeeded | Inspect, deduplicate or reconcile |
| Overload | Perhaps later | Reduce pressure before retrying |
Classifying failures correctly is more important than having an elegant retry loop.
Overload is not an invitation to send more work
Suppose a service can process 1,000 requests per second but receives 1,100.
The extra requests begin failing. If every failed request is immediately retried, the service receives even more traffic.
ORIGINAL LOAD
1,100 requests
│
▼
FAILED REQUESTS
│
▼
IMMEDIATE RETRIES
│
▼
HIGHER LOAD
│
▼
MORE FAILURES
The retry policy has become a positive feedback loop.
The additional work consumes CPU, memory, connections, threads, file descriptors and database capacity. The mechanism intended to improve reliability can prevent the service from recovering.
A retry can amplify one failure
Suppose one user request passes through three layers:
CLIENT
│
▼
SERVICE A
│
▼
SERVICE B
│
▼
DATABASE
Each layer makes an initial attempt plus three retries.
One client request can produce:
4 attempts from client to A
× 4 attempts from A to B
× 4 attempts from B to database
= 64 database attempts
The user performed one action.
The database received up to 64 attempts.
This is why retry policy should be coordinated. Retrying independently at every layer can multiply load precisely when the lowest layer has the least capacity to absorb it.
3. Designing a bounded retry policy
One layer should usually own the retry decision for a given operation.
A low-level library may understand connection failures and protocol statuses. It often does not understand whether ChargePayment is safe to repeat.
A high-level workflow understands the business operation. It may not know whether the request reached the server application.
The retry owner needs enough information to answer:
- Which layer understands the operation?
- Which layer knows its idempotency rules?
- Which layer has the remaining deadline?
- Which layer can avoid multiplying attempts?
- Which layer can observe the final result?
Sometimes Service B should retry one small idempotent database read locally. Sometimes Service A should rerun the whole business operation. The decision should be deliberate rather than emerging from three libraries that each retry by default.
Backoff creates room for recovery
An immediate retry assumes the failing condition has already disappeared.
That may be true for a tiny connection race. It is often false for overload, a service restart, leader election, network disruption or rate limiting.
A backoff policy waits between attempts.
attempt 1
│
└── wait 100 ms
attempt 2
│
└── wait 200 ms
attempt 3
│
└── wait 400 ms
The increasing delay reduces pressure on a dependency that may need time to stabilise.
Backoff is not recovery by itself. It prevents the retry mechanism from fighting recovery quite so enthusiastically.
Jitter prevents synchronised retries
Suppose 10,000 clients experience the same short outage.
If they all retry after exactly one second, the service recovers and immediately receives another large spike.
Adding random variation spreads attempts across a window:
- client A retries after 820 ms;
- client B retries after 1,070 ms;
- client C retries after 940 ms;
- client D retries after 1,160 ms.
Jitter is not decorative randomness.
It prevents independent clients from marching back towards the dependency as one enormous accidental regiment.
Attempts and time both need limits
Exponential delay can grow beyond the point where the result remains useful.
A policy normally needs:
- an initial delay;
- a multiplier;
- a maximum delay;
- a maximum number of attempts;
- an overall deadline.
These bounds answer different questions.
| Limit | Question |
|---|---|
| Attempt limit | How many times may this operation be tried? |
| Maximum delay | How long may one pause become? |
| Overall deadline | How long is the complete result still useful? |
| Retry budget | How much retry traffic may the service generate globally? |
The overall deadline should include connection setup, request transmission, server processing, response reception, backoff and all attempts.
A downstream service should normally receive a smaller remaining deadline so that its caller retains time to process the response and return its own result.
A retry budget protects the wider service
Per-request limits do not always prevent global overload.
Suppose every request may retry twice, but a temporary failure affects millions of requests. The total retry volume can still be enormous.
A service can maintain a retry budget. One simple policy might permit no more than 10 retry attempts for every 100 successful first attempts during a rolling interval.
Once that allowance is exhausted, new calls fail without retrying.
This sacrifices selected requests to protect the service from a retry-driven collapse.
The principle is:
Retries should remain a small correction to normal traffic, not become the dominant workload.
Useful measurements include:
- attempts per logical operation;
- retry rate;
- retry success rate;
- retries by status;
- budget exhaustion;
- extra dependency traffic caused by retries.
A retry hidden inside a library is still production traffic.
Server guidance informs when, not whether
A server may know that immediate retry is inappropriate.
For HTTP, it can return:
HTTP/1.1 503 Service Unavailable
Retry-After: 120
That guidance can help the client decide when another attempt might be appropriate.
It does not decide whether the operation is safe to repeat. The client must combine server guidance with the operation contract, remaining deadline, attempt limit and retry budget.
The server can suggest when.
The operation semantics determine whether.
4. Making repeated attempts safe
Suppose the client sends CreateOrder.
The server commits the order. The response is lost. The client retries.
Without an idempotency mechanism:
attempt 1
│
└── creates order 5001
attempt 2
│
└── creates order 5002
The network fault has become a duplicate business effect.
With a stable operation identifier such as:
checkout-7f82
the server can recognise both attempts as one logical operation.
attempt 1:
checkout-7f82
│
└── creates order 5001
attempt 2:
checkout-7f82
│
└── returns order 5001
The retry still happened.
The repeated effect did not.
Idempotency changes the receiver’s handling of repetition. It does not guarantee that every surrounding workflow has recovered.
Concurrent attempts need one winner
Two attempts with the same idempotency key may arrive at nearly the same time.
A naive implementation can do this:
ATTEMPT A ATTEMPT B
check key check key
not found not found
create order 5001 create order 5002
The server needs a one-winner rule.
Possible mechanisms include:
- a unique constraint scoped by tenant or client and idempotency key;
- an atomic insertion of an
in_progressclaim; - row locking or conditional writes;
- serializable execution where appropriate;
- a defined response when a duplicate arrives while the first attempt is still running.
The business effect, request fingerprint and idempotency result should normally share one protected state transition.
BEGIN
│
├── claim idempotency key
├── validate request fingerprint
├── create order
└── store result
COMMIT
The unique claim prevents two concurrent attempts from both winning.
Reusing the same key for different work should be rejected as a conflict.
A successful retry does not erase the failed attempt
Imagine a payment call.
Attempt one reaches the provider and succeeds, but the response is lost. Attempt two uses a different identity and creates another payment.
The second attempt returns successfully.
application result:
success
provider state:
two charges
From the caller’s immediate perspective, the retry worked.
From the customer’s perspective, recovery has not occurred.
The system now needs duplicate detection, a refund or reversal, order correction, customer communication and an audit trail.
A green response from the latest attempt can coexist with damage left by the earlier one.
Database retries depend on what is known
A database may abort a transaction because of a deadlock or serialization conflict.
Those failures normally give the application a conclusive result: the attempted transaction did not commit.
The application can retry the complete transaction logic.
Suppose the operation is:
- read current stock;
- decide whether reservation is possible;
- write the reservation;
- reduce availability.
If the transaction aborts, the next attempt should read and decide again:
ATTEMPT 1
begin
read current state
make decision
write changes
commit fails conclusively
ATTEMPT 2
begin again
read current state again
make decision again
write changes
commit
Retrying only the final SQL statement would reuse a decision made from an abandoned snapshot.
A connection failure is different.
It may happen:
- before the transaction reaches the database;
- while the transaction is running;
- after commit succeeds but before the client receives confirmation.
The last case leaves the transaction outcome unknown.
Blindly repeating the business operation could duplicate the effect. Recovery may require an idempotency record, a unique business key, an operation-status query or reconciliation.
A conclusive abort supports a complete retry.
An unknown commit outcome requires investigation or duplicate protection.
Retriable transaction bodies should not perform uncontrolled side effects
Suppose a database transaction does this:
read state
send email
update record
commit
The database aborts and the transaction is retried.
transaction attempt 1
│
├── email sent
└── database rollback
transaction attempt 2
│
├── email sent again
└── database commit
The database can roll back its own writes.
It cannot unsend the first email.
A safer design records the intended side effect inside the transaction:
BEGIN
│
├── change business state
└── record notification to send
COMMIT
Another component sends the notification later with its own idempotency and recovery policy.
Message redelivery is a retry too
Suppose a consumer receives GenerateInvoice.
It creates the invoice, then loses its connection before acknowledging the message. The broker does not know that processing completed, so the message becomes available again.
deliver message
│
▼
consumer creates invoice
│
▼
acknowledgement lost
│
▼
message redelivered
Redelivery creates another processing attempt.
The consumer needs a stable message or operation identity and an authoritative record of whether the effect occurred.
The deduplication check and the business effect should form one protected transition where possible.
A dangerous sequence is:
check message not processed
apply effect
crash before recording processed
A unique constraint, inbox record or conditional state-machine transition can make duplicate handling enforceable.
Acknowledgement policies control broker delivery state.
They do not automatically make the business effect exactly once.
5. Containment is not recovery
Some mechanisms protect the healthy path without resolving the failed work.
That is useful.
It should not be mistaken for completion.
Poison messages need quarantine
A message may fail temporarily because a database is unavailable.
Another may be permanently unprocessable because of a missing field, unsupported schema, invalid state or deterministic bug.
Immediate requeueing can create a hot loop:
receive
│
▼
fail
│
▼
requeue
│
▼
receive
│
▼
fail
After a bounded number of attempts, the system may move the message to a failed-work or dead-letter queue.
That protects the main consumer flow.
The business operation remains incomplete.
A useful failed-message process needs answers to:
- Who owns the failed item?
- How will the cause be diagnosed?
- Can the payload be corrected?
- Can it be replayed safely?
- Will replay repeat an existing effect?
- How long may it remain unresolved?
- When should somebody be alerted?
A dead-letter queue without an operating procedure is a cupboard where unfinished promises become historical artefacts.
Recovery may begin by stopping retries
Suppose a database is overloaded and application retries add 40 per cent more traffic.
The immediate recovery action may be to disable or reduce retries.
The system may also need to:
- reject low-priority requests;
- shed load;
- reduce concurrency;
- pause batch work;
- allow queues to drain;
- restore database capacity.
Only after the dependency stabilises should normal traffic return gradually.
Retries optimise the chance of individual request success.
Recovery protects the health of the whole system.
Those goals can temporarily conflict.
Circuit breakers should respond to dependency failure
A circuit-breaker-style policy stops sending ordinary requests after relevant failures cross a threshold.
CLOSED
calls flow normally
│
▼
dependency-failure threshold reached
│
▼
OPEN
calls fail quickly
dependency receives relief
│
▼
after recovery interval
│
▼
LIMITED PROBE
The breaker should normally count signals of dependency unavailability, overload or transport failure.
Expected application outcomes such as validation errors, permission denials, normal conflicts or legitimate not-found responses should not generally make the dependency appear unhealthy.
The open state contains failure.
It does not repair the dependency.
Failover needs capacity, state and fencing
Suppose traffic moves from an unhealthy instance to another one.
instance A unavailable
│
▼
traffic sent to instance B
Requests may begin succeeding again, but recovery still needs to ask:
- Does B have enough capacity?
- Does it have current state?
- Were in-flight operations on A completed?
- Can A safely return to service?
- Did failed requests produce partial effects?
- Can the old primary still accept authoritative writes?
That final question matters.
If the former primary can resume writing after failover, the system may create conflicting state. Fencing, leases, epochs or another ownership mechanism may be needed to ensure that only the current authority can act.
Changing destination is an action.
A stable service is the outcome.
Restarting can replay work
A restart can clear stuck memory, deadlocked threads, leaked resources and bad connections.
It can also erase useful evidence and replay unfinished work.
process restarts
│
├── in-memory queue lost
├── unacknowledged messages redelivered
├── cache emptied
└── clients reconnect
The process may become healthy while the system experiences duplicate processing, cold-cache load, a connection storm or backlog growth.
“Have you tried turning it off and on again?” is a procedure only when somebody also knows what turning it on will replay.
6. Repairing business state
Recovery often requires a different action from the one that originally failed.
Reconciliation resolves uncertainty
Suppose the order service records:
payment_status = pending
while the payment provider records:
payment = authorised
The response was lost. Blindly charging again is unsafe.
A reconciliation job can query the provider using the stable payment reference.
ORDER PAYMENT RECORD
│
▼
compare with provider
│
├── provider says authorised
│ └── update local state
│
├── provider says declined
│ └── record rejection
│
└── no conclusive result
└── escalate or check later
Reconciliation converts uncertainty into a known state.
That is recovery even though it does not repeat the original charge request.
Replay repeats the smallest incomplete step
Suppose an order commits but its OrderCreated event is not delivered.
The order exists. Downstream services did not receive the fact.
Recovery can replay an outbox or event-log record:
authoritative order state
│
▼
undelivered outbox record
│
▼
publisher retries delivery
│
▼
consumers receive event
This is not another attempt to create the order.
It is another attempt to deliver the durable fact that the order already exists.
A good recovery design identifies the smallest incomplete step that can be repeated safely.
Compensation moves forward into a repaired state
Some completed effects cannot be rolled back transactionally.
Suppose payment is authorised but inventory reservation fails.
The system may need to void the authorisation or issue a refund.
ORIGINAL EFFECT
payment captured
COMPENSATING EFFECT
refund issued
Compensation is not time travel.
It creates another business event whose purpose is to offset an earlier effect.
The compensating action is also a distributed operation. It can time out, be repeated, be rejected or leave an uncertain result. It therefore needs its own identity, idempotency rules, observable state and reconciliation process.
Recovery sometimes means moving forwards into a repaired state rather than pretending the earlier operation never occurred.
Restoration repairs lost or corrupted data
Retries cannot repair deleted records, corrupted storage, a destructive migration, lost database files or incorrect changes already committed.
The system may need:
- backup restoration;
- point-in-time recovery;
- replica promotion;
- data reconstruction.
After restoration, it may also need to replay events or reconcile with external systems for the period after the restored point.
restore database to 14:00
│
▼
identify work after 14:00
│
▼
replay or reconcile
│
▼
verify final state
Retrying the request that discovered corruption merely rediscovers the same damage.
Deterministic defects require changed software or input
Suppose every request containing a particular product option crashes the service.
Retrying on another instance reproduces the crash.
request
│
▼
instance A crashes
│
▼
retry on B
│
▼
instance B crashes
Recovery may require blocking the problematic request, rolling back a release, deploying a fix, repairing malformed data or reprocessing affected work.
Retry policies cannot reliably distinguish every deterministic defect from a transient infrastructure failure.
That is one reason attempt limits matter.
7. Recovery exists at several layers
One successful retry addresses only the layer it actually repaired.
| Recovery layer | Question |
|---|---|
| Request | Did this caller receive a usable result? |
| Operation | Did the logical business operation reach one known outcome? |
| Data | Are authoritative records complete and internally valid? |
| Workflow | Did every required downstream step finish or enter a managed state? |
| Service | Can the system sustainably handle normal traffic again? |
| Customer | Has the user received a truthful outcome and any necessary correction? |
A service can be available while several business operations remain unresolved.
A workflow can be complete while the customer still needs a refund or explanation.
These are different recovery claims.
A complete checkout example
Suppose checkout coordinates:
- order creation;
- inventory reservation;
- payment authorisation;
- confirmation publication.
The initial attempt proceeds:
1. order created
2. inventory reserved
3. payment authorised
4. connection fails before response
5. confirmation event not published
The client sees a timeout.
A careless recovery policy submits a completely new checkout request.
Possible result:
- a second order;
- a second inventory reservation;
- a second payment attempt.
A more deliberate design uses one checkout operation ID:
checkout-7f82
The client queries the operation state.
checkout-7f82
│
├── order: 5001
├── inventory: reserved
├── payment: authorised
└── confirmation: pending
The system does not recreate completed steps.
It retries only the incomplete publication:
publish OrderConfirmed
A consumer later sends the customer notification.
Recovery concludes when:
- the order has one known state;
- inventory and payment agree with it;
- the missing event is delivered;
- duplicate attempts are resolved;
- the customer receives one truthful result.
The recovery included a retry.
It was not simply another checkout.
Recovery needs an authoritative record
After failure, the system needs somewhere to ask:
- What was this operation?
- Which steps completed?
- Which steps remain pending?
- Which result has already been returned?
- Which attempts belong to it?
That record might be an idempotency table, operation resource, order state machine, durable workflow record, outbox or message log.
Without durable operation identity, recovery relies on timestamps, similar payloads, customer reports and scattered logs.
That can work.
It is a particularly Victorian way to run a distributed system: several witnesses, a missing telegram and one suspicious ledger entry.
8. Recovery needs verification and rehearsal
A recovery action can appear successful while leaving the system unstable.
Suppose operators add instances and error rates fall, but retries remain elevated, queues continue growing and the database is saturated.
The service is not fully recovered.
Verification might include checking that:
- request success rates returned to normal;
- latency distributions stabilised;
- retry traffic fell;
- backlogs are draining;
- dependencies have spare capacity;
- failed operations were reconciled;
- no duplicate effects remain;
- monitoring and alerts are healthy;
- customer corrections were completed.
Recovery is an outcome supported by evidence.
It is not the moment somebody ran a command without receiving an error.
Recovery should be rehearsed
A design may claim that failed messages can be replayed.
That claim should be tested.
A service may claim that a database can be restored.
The restore should be exercised.
A workflow may claim that duplicate payment requests are idempotent.
The duplicate should be sent in a controlled test.
Useful exercises include:
- losing a response after commit;
- delivering two concurrent requests with one idempotency key;
- redelivering a processed message;
- restarting a consumer before acknowledgement;
- aborting a database transaction conclusively;
- losing a database connection around commit;
- making a dependency return overload errors;
- replaying an outbox record;
- running a compensating operation twice;
- restoring data into an isolated environment;
- failing over while verifying that the old primary is fenced.
Recovery mechanisms that exist only in diagrams have not yet recovered anything.
9. A practical decision framework
| Failure condition | Is another attempt useful? | What else may recovery require? |
|---|---|---|
| Brief connection failure before request processing | Often | Backoff and bounded retry |
| Explicit validation error | No, not unchanged | Correct the request |
| Permission denied | Usually not | Correct identity, policy or authorisation |
| Deadlock victim | Often | Retry the complete transaction |
| Serialization conflict | Often | Re-read state and retry the complete transaction |
| Database connection lost around commit | Not blindly | Determine outcome through identity, query or reconciliation |
| Service overload | Perhaps later | Backoff, jitter, load shedding and capacity recovery |
| Response lost after business commit | Only with duplicate protection | Idempotency lookup or operation-status query |
| Message processed before acknowledgement loss | Redelivery is expected | Atomic deduplication and effect handling |
| Permanently malformed message | No | Quarantine, repair or deliberate cancellation |
| Missed downstream event | Retry publication | Replay the durable event or outbox record |
| Conflicting external-system state | Not blindly | Reconciliation |
| Completed unwanted external effect | No simple retry | Idempotent compensation and verification |
| Corrupted or deleted data | No | Restore, replay and repair |
| Deterministic software bug | Repetition may worsen failure | Roll back, fix or block bad input |
| Cascading overload | Retrying may be harmful | Reduce traffic and stabilise dependencies |
The useful question is not only:
Can this be retried?
It is:
What state will exist after the retry, and will that state satisfy the recovery objective?
The mental model I am keeping
My old model was:
operation fails
│
▼
retry
│
▼
recovered
The new model is:
FAILURE OBSERVED
│
▼
CLASSIFY THE CONDITION
│
┌──────────────────┼──────────────────┐
│ │ │
▼ ▼ ▼
transient permanent uncertain
│ │ │
▼ ▼ ▼
bounded retry change condition inspect state
│ │ │
└──────────────────┼──────────────────┘
▼
RESOLVE BUSINESS STATE
│
┌───────────────────┼───────────────────┐
│ │ │
▼ ▼ ▼
reconcile replay compensate
│ │ │
└───────────────────┼───────────────────┘
▼
RESTORE USEFUL SERVICE
│
▼
VERIFY
A retry asks:
Should I attempt this operation again?
Recovery asks:
- What was damaged or left uncertain?
- Which state is authoritative?
- Which work is incomplete?
- Which effects must not be repeated?
- Which effects need to be reversed?
- Can the system handle normal traffic safely?
- How will I prove the incident is resolved?
This gives retries a smaller and more useful role.
They are excellent for selected transient faults when:
- the operation is safe to repeat;
- the failure classification supports retry;
- the attempts are bounded;
- backoff and jitter protect the dependency;
- the total deadline is respected;
- retry traffic is observable.
They are dangerous when they:
- repeat non-idempotent effects;
- amplify overload;
- occur independently at every layer;
- hide permanent defects;
- replace reconciliation;
- continue without a deadline.
And they are insufficient when recovery requires repairing state, replaying missing work, compensating completed effects, restoring data, changing capacity, fixing software or communicating with a customer.
A retry repeats an action.
Recovery restores confidence.
Sometimes the shortest route to recovery is another attempt.
Sometimes it is to stop attempting, inspect what happened and repair the world left behind.
References and further reading
HTTP retry semantics
RFC 9110: HTTP Semantics
Defines safe and idempotent HTTP methods, restrictions on automatically retrying non-idempotent requests, and the Retry-After field.
RPC retries
gRPC: Retry
Explains that a retry creates a new RPC attempt and documents attempt limits, retryable status codes, exponential backoff, jitter and throttling.
gRPC: Status Codes
Distinguishes statuses such as UNAVAILABLE, ABORTED and FAILED_PRECONDITION, which imply different retry levels and preconditions.
gRPC: Client-Side Retry Design
Defines gRPC’s retry architecture, service configuration, retry throttling and commitment rules.
Backoff, jitter and retry amplification
Amazon Builders’ Library: Timeouts, retries, and backoff with jitter
Explains why retries can become selfish, and discusses timeouts, idempotency, backoff, jitter, attempt limits and retry control.
AWS Architecture Blog: Exponential Backoff and Jitter
Demonstrates how synchronised retry behaviour increases contention and how jitter reduces total work.
Google SRE Book: Addressing Cascading Failures
Explains retry amplification, retry budgets, load shedding, degraded operation and containment of cascading failure.
Message acknowledgement and redelivery
RabbitMQ Reliability Guide
Explains publisher confirms, consumer acknowledgements, connection failure and retransmission when delivery outcomes remain uncertain.
RabbitMQ: Consumer Acknowledgements and Publisher Confirms
Documents acknowledgement lifecycles, negative acknowledgements and redelivery behaviour.
Database transaction retries
PostgreSQL: Serialization Failure Handling
Explains why serialization failures require retrying the complete transaction logic and discusses deadlock retries.
PostgreSQL: Error Codes
Defines conditions including serialization_failure, deadlock_detected and transaction_resolution_unknown.