A consumer receives a message.
Processing fails.
The message is delivered again.
Processing fails again.
After several attempts, the message is moved away from the healthy path.
MAIN QUEUE
│
▼
CONSUMER
│
├── success ──► acknowledge
│
└── repeated failure
│
▼
QUARANTINE PATH
That can sound final.
The message no longer blocks ordinary consumption. The consumer is no longer trapped in a rapid cycle around the same broken input.
But if the message represented:
ReserveStockSendOrderConfirmationUpdateSearchIndexPaymentAuthorisedOrderConfirmed
then the corresponding business work may still be incomplete.
Moving the message may have protected the healthy processing path. It has not necessarily reserved the stock, sent the confirmation, updated the index, applied the payment result or advanced the order workflow.
That gives me a more useful definition:
A dead-letter queue is a quarantine for unresolved messages, not a cemetery for work the system no longer wishes to remember.
Its job is to preserve the failed item, its context and the opportunity to recover it safely.
A DLQ isolates failure; it does not complete work
Suppose a consumer handles OrderConfirmed by creating a notification job.
OrderConfirmed
│
▼
create confirmation email job
The notification database becomes temporarily unavailable.
A later attempt may succeed:
| Attempt | Result |
|---|---|
| 1 | Database unavailable |
| 2 | Database unavailable |
| 3 | Database recovered; job created |
This is a transient failure. Retrying is useful because the condition can change without changing the message.
Now suppose the consumer cannot decode the event.
event schema:
order.events.v9
consumer supports:
order.events.v1
Immediate redelivery produces the same result. The message becomes a poison message for that consumer.
It can consume processing capacity, broker traffic, logs, alert volume and the attention of every instance that receives it. If the consumer preserves queue or partition order, one poison message may also delay everything behind it.
A dead-letter path allows the consumer to say:
I cannot complete this message under the current conditions, and repeating it on the healthy path is no longer useful.
That is containment.
Recovery comes later.
The message still represents unresolved work. If notification dead-letters OrderConfirmed, the system may now contain two different truths:
| Capability | State |
|---|---|
| Order | Confirmed |
| Notification | Confirmation not created |
Likewise, if search cannot apply ProductPublished, catalog remains authoritative while the search projection is incomplete.
This is why DLQ monitoring should expose business impact, not only message counts.
unresolved messages:
1
affected capability:
customer order confirmation
oldest unresolved message:
42 minutes
One failed analytical update may tolerate several hours. One failed payment-result event may leave an order in a dangerous uncertain state.
A message count does not communicate that difference.
Dead-lettering is a policy, and the handoff can fail
Different platforms provide different dead-letter mechanisms.
RabbitMQ can dead-letter messages after rejection, expiry, queue-length limits or delivery-limit conditions. The configured dead-letter exchange then routes the message onwards. The exchange is not itself necessarily the queue where operators inspect failures.
Amazon SQS uses a redrive policy. After a message has been received without successful deletion more than the configured maxReceiveCount, SQS can move it from the source queue to a configured dead-letter queue.
Kafka brokers do not interpret arbitrary consumer exceptions as dead letters. An application or framework decides whether to publish a failed record to another topic. Kafka Connect sink connectors support a configurable dead-letter topic and can add error context to record headers. Current Kafka Streams releases also provide built-in DLQ configuration for supported exception paths when the default exception handlers are used.
The shared architectural shape is:
NORMAL PROCESSING PATH
│
▼
attempts exhausted
or failure classified as deterministic
│
▼
ISOLATED FAILURE PATH
The exact contract still needs answers:
- What caused the move?
- Who performed it?
- Was the source message acknowledged or its offset advanced?
- Which metadata was preserved?
- Can the item be replayed?
- Does moving it affect ordering?
- What happens when the quarantine destination is unavailable?
The last question is easy to miss.
Quarantine is another distributed state transition
The handoff to a DLQ, failure topic or recovery queue can itself be lost, duplicated, rejected or delayed.
For a custom Kafka consumer, the unsafe sequence is:
commit source offset
│
▼
publish failure record
If the process crashes between those steps, the source record has been skipped and no quarantine record exists.
Reversing the steps changes the failure mode:
publish failure record
│
▼
commit source offset
If publication succeeds but the offset commit fails, recovery may publish the failure record again.
Where the source offset and the quarantine record both live in Kafka, a Kafka transaction can commit the output record and consumed offset atomically within Kafka. That does not include an external database, email provider, payment service or arbitrary API.
The handoff invariant should be explicit:
The failed record and the diagnostic context required for recovery are durable before source progress advances.
That invariant should be tested when:
- the failure topic or queue is unavailable;
- the consumer lacks permission to publish;
- building the failure envelope itself fails;
- source acknowledgement succeeds while quarantine publication fails;
- the consumer restarts after publishing but before advancing progress.
RabbitMQ also has more than one dead-lettering guarantee. Quorum queues use at-most-once dead-lettering by default. They can be configured for safer at-least-once dead-lettering under specific queue, overflow and dead-letter settings, retaining the source message until the target confirms receipt. Missing exchanges, unavailable targets or unroutable messages still need monitoring and policy.
“Use a DLQ” is the beginning of these questions, not the answer.
Preserve evidence and expose business impact
A failed payload on its own may not provide enough evidence.
Suppose the quarantine queue contains:
{
"order_id": "order-5001"
}
An operator still needs to know:
- which queue or topic it came from;
- which logical consumer failed;
- which consumer build was running;
- which schema or content type was used;
- how many attempts were made;
- what error occurred;
- when the first and last failures occurred;
- which Kafka topic, partition and offset held the original record;
- which correlation and causation IDs belong to the workflow;
- whether any local or external side effect may already have completed.
A useful failure record might preserve:
{
"failed_message": {
"message_id": "event-44",
"message_type": "OrderConfirmed",
"payload": {
"order_id": "order-5001"
}
},
"failure": {
"consumer": "bfstore-notification-dispatch",
"consumer_version": "1.7.2",
"reason": "unsupported schema version",
"attempts": 5,
"first_failed_at": "2022-10-30T14:31:05Z",
"last_failed_at": "2022-10-30T14:37:42Z"
},
"source": {
"topic": "bfstore.order.events.v1",
"partition": 2,
"offset": 941
}
}
The exact representation varies.
The principle is that the failure path preserves enough context to diagnose the cause and verify the eventual recovery.
Preserve application identity, not only transport identity
A broker-assigned message identifier may change during redrive.
For example, SQS gives a redriven message a new message ID. The business envelope should therefore retain a stable application identity such as:
event_id:
event-44
operation_id:
checkout-7f82
Transport identity answers:
Which broker message is this delivery?
Application identity answers:
Which logical event or operation is this another representation of?
Replay safety normally depends on the second.
Broker metadata is useful but not complete
RabbitMQ records dead-letter information such as source queue, reason, count and time in x-death or x-opt-deaths metadata. That history is aggregated by queue and reason. It is not a complete attempt-by-attempt application trace.
Kafka Connect can add source and exception context to dead-letter headers.
Those features help, but a custom envelope may still need:
- consumer version;
- handler name;
- correlation and causation;
- first and last application failures;
- repair status;
- uncertainty about external effects.
Privacy still applies
Failure records may expose data to operators, support tooling, logging systems and long-lived recovery stores.
Diagnostic usefulness must be balanced against:
- personal data;
- payment information;
- credentials and tokens;
- commercially sensitive values;
- retention requirements;
- access granted to operators.
A dead-letter queue should not become the one place where every privacy rule quietly takes the afternoon off.
Every quarantine path needs an owner
A DLQ without an owner is a storage location for abandoned responsibility.
Ownership should answer:
- Which team receives the alert?
- Who understands the source and consumer contracts?
- Who can inspect the data safely?
- Who decides whether replay is permitted?
- Who fixes a producer or consumer defect?
- Who verifies the final business state?
- Who closes the recovery task?
The infrastructure team may operate Kafka, RabbitMQ or SQS. It may not know whether replaying PaymentCaptured would corrupt an order.
Recovery can involve several parties:
| Role | Responsibility |
|---|---|
| Platform owner | Keeps the quarantine channel available and observable |
| Consumer owner | Diagnoses why processing failed |
| Producer owner | Corrects invalid or incompatible messages |
| Workflow owner | Verifies the final business outcome |
Alerting should cover arrival, age and growth:
- new quarantined messages;
- total unresolved count;
- oldest unresolved age;
- arrival rate;
- repeated failures by message type or consumer version;
- failed attempts to publish to the quarantine destination;
- records approaching retention expiry;
- failed redrive or repair attempts.
The most dangerous state is often not a large DLQ.
It is a small one that everybody assumes belongs to somebody else.
Classify the failure before acting
A useful triage separates several kinds of failure.
Transient infrastructure failure
Examples include a temporarily unavailable database, a restarting dependency, a short network interruption or a rate limit with explicit recovery guidance.
Bounded retry and delay may be enough.
The message may not need quarantine unless the condition lasts beyond the normal retry policy.
Deterministic failure under the current consumer
Examples include:
- malformed or corrupted payload;
- required field missing;
- contract violation;
- unsupported schema or enum;
- current consumer missing a required capability.
Repeating the same payload against the same consumer will not help.
The long-term resolution may still differ:
- correct the producer;
- transform the message;
- upgrade the consumer;
- reject the work deliberately;
- restore a missing schema or configuration.
“Unsupported today” does not always mean “invalid forever”.
Consumer defect
The message is valid, but a bug in the handler causes failure.
Recovery may require:
- deploying a fixed consumer;
- testing the quarantined message against the fix;
- replaying affected records at a controlled rate;
- verifying that partial effects did not remain.
Unexpected business state
The message may be valid, but the consumer’s state does not permit it.
For example:
incoming:
OrderConfirmed
local projection:
order does not exist
Possible causes include:
- an earlier event is delayed;
- an earlier event failed;
- events arrived through different partitions or topics;
- local state was lost;
- the consumer began from the wrong offset;
- the workflow is genuinely inconsistent.
Blind replay may reproduce the failure. The consumer may need to restore or reconcile prerequisite state first.
External side effect with an uncertain outcome
The handler may have called an email, payment or shipping provider before failing.
The system must determine whether that call succeeded before replaying.
Otherwise recovery can create duplicate emails, shipments, captures or refunds.
The quarantine record tells me the handler did not complete normally.
It does not necessarily tell me which effects already happened.
Obsolete or superseded work
Some quarantined work no longer needs execution.
A later event may supersede it. The associated operation may have been cancelled. Policy may require deletion. A current-state rebuild may make replay unnecessary.
That can justify deliberate discard, but the decision needs an owner, evidence and a reason.
“Deleted because the queue looked untidy” is not a recovery policy.
Recovery begins after the cause changes
A redrive button can make recovery look mechanical:
DEAD-LETTER QUEUE
│
▼
REDRIVE
│
▼
SOURCE QUEUE
But replaying an unchanged message into an unchanged consumer usually creates an unchanged failure.
Before replay, I should be able to state:
This message failed because X. We changed Y. Reprocessing should now produce Z.
For example:
| Before | Change | Expected result |
|---|---|---|
| Consumer cannot understand schema v2 | Consumer 1.8 adds v2 support | Message can be decoded |
| Customer projection missing | Projection restored from source | Dependent event can be applied |
| Payment outcome uncertain | Provider status reconciled | Recovery avoids a duplicate capture |
Without that sentence, redrive is only another retry with a larger button.
Replay must protect the real effect
Suppose the original handler performed:
1. create notification job
2. update local order projection
3. acknowledge event
It failed during step two.
The notification job may already exist.
A replay-safe handler can use:
unique key:
order-5001 + order-confirmation
The database enforces one matching job. On replay, job creation becomes a harmless duplicate while the unfinished projection update can still complete.
Other protections include:
- processed-message records committed with the local effect;
- stable event or operation IDs;
- entity versions;
- unique business constraints;
- authoritative status checks;
- provider-side idempotency.
The mechanism must protect the actual side effect.
Marking an event ID as processed too early can suppress unfinished work. Comparing payloads alone can confuse two genuinely different operations.
Recovery has several routes
Returning a message to the original source is only one option.
| Route | Suitable when |
|---|---|
| Original queue or topic | The ordinary handler is fixed, ordering is understood and replay volume is safe |
| Dedicated recovery topic | Extra validation, lower concurrency, richer evidence or operator approval is required |
| Targeted repair | Replaying the original message would be unsafe or unnecessary |
| Authoritative rebuild | A derived projection can be reconstructed more reliably from current or retained source data |
| Corrective event | A newer fact should replace or explain an earlier failed path |
| Deliberate discard | Work is obsolete, invalid or prohibited, with an owner and recorded reason |
Native SQS redrive moves messages without filtering or modifying them. A transformed repair therefore needs a custom recovery consumer or another targeted path.
Redriven SQS messages can also interleave with newly produced messages at the destination. Returning a record to a FIFO source does not restore its former position in the sequence.
Rate control matters
A repaired consumer can still be overwhelmed by a sudden replay.
Recovery should define:
- maximum redrive rate;
- concurrency;
- dependency capacity;
- pause and resume controls;
- monitoring during replay;
- a stop condition when the original failure returns.
The goal is to restore capability without converting yesterday’s incident into today’s load test.
Resolution is a state, not a deletion
A quarantine record often passes through several states:
| State | Meaning |
|---|---|
| Quarantined | Removed from normal processing |
| Investigating | Cause and possible effects are being established |
| Repair ready | Required change has been verified |
| Reprocessing | Controlled recovery is active |
| Resolved | Final business state has been verified |
| Discarded | Deliberate decision with owner and reason |
Without resolution state, one DLQ count can mix active incidents, repaired messages and deliberately obsolete records.
Ordering, retention and replay constrain recovery
Suppose one ordered stream contains:
offset 40 OrderCreated
offset 41 OrderConfirmed
offset 42 OrderCancelled
The consumer cannot process offset 41.
If it stops, the partition remains blocked.
If it skips 41 and continues with 42, it creates a gap in its local history.
| Policy | Benefit | Cost |
|---|---|---|
| Stop on failure | Preserves sequential processing | Blocks later records |
| Skip to quarantine | Restores throughput | Later records may be meaningless without the missing transition |
The consumer may need:
- entity versions;
- state-machine validation;
- buffering of later records;
- reconciliation with the authoritative owner;
- per-key isolation;
- a repair process that fills or supersedes the missing transition.
A DLQ protects availability by removing an obstruction.
The design must decide whether later work remains valid without it.
Retention is a recovery deadline
Quarantined messages cannot be retained forever without cost, security and policy consequences.
Retention should reflect:
- maximum investigation time;
- business and legal requirements;
- expected repair time;
- message sensitivity;
- ability to reconstruct the work elsewhere;
- cost of retaining payload and evidence.
The system should alert before unresolved records expire.
message failed
│
▼
nobody investigates
│
▼
retention expires
│
▼
evidence and recovery option lost
SQS makes this especially concrete.
For a standard queue, expiry after movement to the DLQ is still based on the original enqueue timestamp. The DLQ retention period should therefore be longer than the source queue’s retention period.
For a FIFO queue, the enqueue timestamp resets when the message moves to the DLQ.
The broker’s age metric may also describe time since movement rather than the message’s complete age. Application metadata should preserve the times needed for investigation.
A retention period is not merely broker housekeeping.
It is the time the organisation has agreed to preserve that recovery option.
A growing DLQ is a symptom
If failures arrive faster than they are resolved, quarantine becomes another growing queue.
new failures:
1,000 per hour
resolved:
100 per hour
growth:
900 per hour
At that point, the system does not have an occasional poison message.
It has a broken producer, consumer, contract or dependency.
Scaling the dead-letter queue only gives the failure more floor space.
The response may require stopping the offending producer, rolling back a release, pausing consumption, restoring a dependency, repairing data or changing the main processing policy.
The DLQ provides containment and evidence while that work happens.
It should not normalise a permanently failing architecture.
A possible bfstore recovery flow
Suppose bfstore’s notification group consumes:
topic:
bfstore.order.events.v1
group:
bfstore-notification-dispatch
It receives:
OrderConfirmed
event_id:
event-44
order_id:
order-5001
The handler cannot decode a newly added DeliveryAddress representation.
After three delayed attempts, the consumer prepares a failure record for:
bfstore.notification.failures.v1
The record contains:
application message ID:
event-44
source:
topic bfstore.order.events.v1
partition 2
offset 941
consumer:
bfstore-notification-dispatch
consumer version:
1.7.2
reason:
unsupported address representation
attempts:
3
The consumer publishes the failure record and advances source progress through a Kafka transaction, so the quarantine handoff and consumed offset succeed or fail together within Kafka.
An alert opens for the notification owner.
The investigation finds:
producer:
emitted the compatible Protobuf field correctly
consumer:
used an older generated schema
The team deploys consumer 1.8.0.
Before redrive, it tests event-44 against the new decoder.
The record moves to repair ready.
A recovery worker republishes the event at a controlled rate.
The handler creates a notification job using a unique constraint:
order_id + notification_type
If an earlier attempt created the job before failing, replay does not create another.
Recovery is complete only when:
event-44is processed;- one notification job exists;
- customer communication is scheduled;
- the quarantine record is marked resolved;
- the alert closes with evidence of the final state.
The message did not go somewhere to die.
It went somewhere the system could stop, inspect the wound and decide how to continue.
A dead-letter queue checklist
| Concern | Question |
|---|---|
| Entry | Which failures leave normal processing? |
| Retry | How many attempts occur first, with what delay and backoff? |
| Handoff | Is the failure record durable before source progress advances? |
| Classification | Is the failure transient, deterministic, defective, state-dependent, uncertain or obsolete? |
| Evidence | Can the original record, source position and failure context be reconstructed safely? |
| Identity | Does application identity survive movement and redrive? |
| Privacy | Can operators investigate without exposing unnecessary sensitive data? |
| Ownership | Which team owns every quarantined item? |
| Alerting | Are arrival, age, growth and retention expiry monitored? |
| Impact | Which capability or workflow remains incomplete? |
| Ordering | Can later records be applied safely without this one? |
| Safety | Can recovery repeat an effect that already completed? |
| Change | What is different since the original failure? |
| Route | Replay, reconcile, transform, rebuild or discard? |
| Rate | Can recovery overload the repaired path or its dependencies? |
| Verification | How will the final application state be established? |
| Resolution | How is the item marked resolved, obsolete or deliberately discarded? |
| Retention | When does the recovery option expire? |
A dead-letter queue without these answers is not a recovery system.
It is only a second queue.
The mental model I am keeping
My old model was:
message fails
│
▼
move to DLQ
│
▼
problem removed
The stronger model is:
MESSAGE PROCESSING FAILS
│
▼
CLASSIFY THE FAILURE
│
┌───────────────┴───────────────┐
│ │
▼ ▼
transient unresolved
│ │
▼ ▼
bounded retry quarantine safely
│
▼
preserve evidence
│
▼
assign an owner
│
▼
diagnose and repair
│
┌─────────────────┼─────────────────┐
│ │ │
▼ ▼ ▼
replay reconcile discard
│ │ deliberately
└─────────────────┼─────────────────┘
▼
verify final state
The main path exists to process healthy work efficiently.
The quarantine path exists to keep unhealthy work visible without allowing it to consume the healthy path forever.
It provides:
- containment;
- evidence;
- time to investigate;
- a controlled recovery boundary.
It does not automatically provide:
- a durable quarantine handoff;
- a corrected message;
- a fixed consumer;
- idempotent replay;
- restored ordering;
- completed business work;
- an accountable owner.
Those must be designed around it.
So a message in a dead-letter queue is not dead.
It is unresolved.
The queue is not its grave.
It is the waiting room where the system must finally decide whether to repair the message, repair itself or admit that the requested work can no longer be completed.
References and further reading
RabbitMQ
RabbitMQ: Dead Letter Exchanges
Documents dead-letter triggers, exchange routing and x-death or x-opt-deaths metadata.
RabbitMQ: Quorum Queues
Documents quorum-queue dead-letter strategies, including the requirements and trade-offs of at-least-once dead-lettering.
Amazon SQS
Amazon SQS: Using dead-letter queues
Documents redrive policies, maxReceiveCount, retention behaviour and FIFO ordering considerations.
Amazon SQS: Configuring dead-letter queue redrive
Documents redrive rate control, new message identifiers and enqueue times, and the limitation that native redrive cannot filter or modify messages.
Apache Kafka
Apache Kafka 4.2: Kafka Connect configuration
Documents dead-letter topics and optional error-context headers for supported Kafka Connect sink failures.
Apache Kafka 4.2: Kafka Streams configuration
Documents the current Kafka Streams DLQ configuration and its exception-handler boundaries.
Apache Kafka: Design
Explains Kafka transactions and the boundary for atomically committing Kafka output records with consumed offsets.