The transactional outbox solves a very particular problem

Suppose the order service confirms an order.

Two things now need to happen:

1. The confirmed order must be committed to the order database.
2. An OrderConfirmed event must be published for notification, shipping and analytics.

The obvious implementation is:

if err := repository.ConfirmOrder(ctx, order); err != nil {
	return err
}

if err := publisher.PublishOrderConfirmed(ctx, event); err != nil {
	return err
}

The sequence looks reasonable.

The database is updated first.

The event is published afterwards.

But there is a failure gap between those two operations.

ORDER DATABASE
      │
      │ commit succeeds
      ▼
PROCESS CRASHES
      │
      ▼
EVENT NEVER PUBLISHED

The order is confirmed.

Notification, shipping and analytics never learn about it.

The service now contains two conflicting stories:

ORDER DATABASE

order-5001 is confirmed


EVENT STREAM

OrderConfirmed was never recorded

The database and broker were updated separately.

They were not part of one shared transaction.

This is the particular problem the transactional outbox addresses.

The transactional outbox allows a service to commit its local business change and a durable record of the message caused by that change in one database transaction.

The message can then be published asynchronously.

The outbox does not make the database and broker one transaction.

It avoids needing them to be one.

1. The dual write creates one particular failure gap

Suppose the service updates its database and publishes an event as separate operations.

There are two possible orders.

Database first

1. Commit business state.
2. Publish the event.

Failure can occur between the steps:

DATABASE

order confirmed


BROKER

event missing

The business fact exists, but consumers do not learn it.

Broker first

1. Publish the event.
2. Commit business state.

Failure can still occur between the steps:

BROKER

OrderConfirmed published


DATABASE

order remains pending

Consumers receive an event describing a fact the authoritative service never committed.

Changing the order merely changes which inconsistency becomes possible.

The difficulty is not the sequence chosen.

The difficulty is that two independently failing systems must both change.

A local transaction cannot normally include Kafka

A local database transaction can make several database writes atomic.

BEGIN
  │
  ├── update orders
  ├── insert order history
  └── insert audit record
COMMIT

Either the transaction commits all of those changes or it commits none of them.

But publishing to Kafka is not an ordinary write inside that MySQL or PostgreSQL transaction.

The database cannot normally roll back a Kafka record because its own commit failed.

Kafka cannot normally roll back a relational transaction because publication failed.

A distributed transaction protocol could attempt to coordinate several participants. That introduces another set of complexity, availability and operational trade-offs.

The outbox takes a narrower approach:

Require atomicity only inside the database that already owns the business state.

The broker update happens later.

That narrowness is the point.

2. The outbox turns publication into local durable state

Instead of publishing directly during the order transaction, the service inserts an outbox record.

BEGIN
  │
  ├── update order to CONFIRMED
  └── insert OrderConfirmed outbox record
COMMIT

The database now contains both:

These changes share one transaction.

If the transaction commits, the order confirmation and the durable publication record both exist.

If the transaction rolls back, neither exists.

The dangerous middle state disappears:

confirmed order

without any durable record
that an event must be published

The outbox record is not yet the event in Kafka.

It is the service’s durable obligation to publish that event.

Create the event inside the business transaction

Suppose OrderConfirmed contains:

• order ID;
• customer ID;
• confirmed lines;
• accepted total;
• confirmation time;
• order version.

Those values should come from the state accepted by the transaction.

BEGIN
  │
  ├── validate order
  ├── set status to CONFIRMED
  ├── determine accepted total
  ├── increment order version
  └── create OrderConfirmed outbox event
COMMIT

The event should not be reconstructed much later from mutable current state without care.

By then:

• the order may have changed;
• customer details may have changed;
• product descriptions may have changed;
• another order transition may have occurred.

An event is the owner’s authoritative assertion about one completed transition.

The outbox should preserve the accepted payload, or enough immutable facts to construct that payload without accidentally describing a later state.

Give the event a stable identity

A useful outbox record may contain:

• event ID;
• event type;
• aggregate type;
• aggregate ID;
• aggregate version;
• payload;
• schema or content version;
• headers and tracing metadata;
• creation time;
• routing or destination information where required.

The event ID should remain stable across publication attempts.

It can:

• identify the outbox record;
• identify the published event;
• support consumer deduplication;
• connect logs and traces;
• support investigation and controlled republishing.

Creating a new event ID for every relay attempt would turn one logical event into several apparently unrelated events.

Preserve payload longevity

A pending outbox record may survive several application deployments.

The service must still be able to publish it after code, schemas and libraries change.

A durable record therefore needs enough information to answer:

• Which event contract does this payload use?
• Which serializer can read it?
• Which destination should receive it?
• Which aggregate and version does it describe?
• Can the current relay publish rows created by the previous release?

A deployment should be tested against older pending rows before an incompatible schema or serializer change reaches production.

Otherwise the database may preserve the obligation perfectly while the current relay can no longer discharge it.

3. Polling and change-data capture are different relay lifecycles

A relay moves durable outbox records to the broker.

OUTBOX
   │
   ▼
RELAY
   │
   ▼
BROKER

The two common relay models are polling and change-data capture.

They solve the same broad publication problem, but their row lifecycles and progress mechanisms are different.

These models should not be blended into one schema by accident.

Polling publisher

A polling relay queries for records that have not been completed.

A polling-specific table might look like:

CREATE TABLE outbox_events (
    event_id VARCHAR(128) PRIMARY KEY,
    event_type VARCHAR(255) NOT NULL,
    aggregate_type VARCHAR(128) NOT NULL,
    aggregate_id VARCHAR(128) NOT NULL,
    aggregate_version BIGINT NOT NULL,
    schema_version INT NOT NULL,
    payload BLOB NOT NULL,
    created_at TIMESTAMP NOT NULL,
    published_at TIMESTAMP NULL,
    attempt_count INT NOT NULL DEFAULT 0
);

A simple query might begin with:

SELECT
    event_id,
    event_type,
    aggregate_id,
    aggregate_version,
    payload
FROM outbox_events
WHERE published_at IS NULL
ORDER BY created_at, event_id
LIMIT 100;

That query is only the beginning of the design.

Several workers need a claiming strategy so that they do not all treat one row as unrelated work. Common options include row locks, SKIP LOCKED, leases, or an explicit processing state.

Polling has useful advantages:

• a simple mental model;
• ordinary database queries;
• no need to read transaction logs;
• straightforward implementation in the service’s normal stack.

It also introduces costs:

• polling load;
• delay between polls;
• worker coordination;
• claim expiry and crash recovery;
• cleanup of completed rows;
• ordering concerns across workers.

Change-data capture

A CDC relay reads committed inserts from the database transaction log.

DATABASE TRANSACTION LOG
          │
          ▼
CDC CONNECTOR
          │
          ▼
BROKER

Debezium’s Outbox Event Router is designed around deliberate outbox rows. The application inserts an event record in the same transaction as the business change. The connector captures that insert and routes the event.

In the standard model:

• the outbox row is treated as an inserted event;
• updates to the event row are not the normal publication marker;
• the event ID can be propagated as metadata;
• the aggregate ID can become the Kafka key;
• cleanup deletes are filtered from the emitted business-event stream.

A CDC-compatible outbox table might therefore omit mutable publication state:

CREATE TABLE outbox_events (
    event_id VARCHAR(128) PRIMARY KEY,
    event_type VARCHAR(255) NOT NULL,
    aggregate_type VARCHAR(128) NOT NULL,
    aggregate_id VARCHAR(128) NOT NULL,
    aggregate_version BIGINT NOT NULL,
    schema_version INT NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMP NOT NULL
);

CDC can provide low publication latency and avoid application polling. It also introduces:

• connector infrastructure;
• transaction-log access;
• connector offset storage;
• routing and schema conventions;
• another operational pipeline that can stop or fall behind.

Debezium normally provides at-least-once delivery, so duplicate change events can still appear around some failures. Stronger exactly-once source delivery can be configured for supported environments, but the stable application event ID remains valuable for consumer deduplication, investigation and manual recovery.

Change-data capture does not remove the outbox concept.

The outbox is still the locally committed message record.

CDC is one way to relay it.

Raw table changes are not a domain event contract

A CDC tool can expose:

orders.status changed from PENDING to CONFIRMED

That is a technical persistence fact.

A deliberate outbox event can say:

OrderConfirmed

and carry:

• the order ID;
• the accepted total;
• the order version;
• correlation and causation metadata;
• the event’s stable identity.

The application chooses the event contract inside the transaction.

The connector transports that contract.

It should not be expected to invent a stable domain event from arbitrary table mutations.

4. Reliable publication can still produce duplicates

Consider a polling relay:

1. Publish event-44 to Kafka.
2. Mark event-44 as published.

The publication succeeds.

Before the row is updated, the relay crashes.

KAFKA

event-44 exists


OUTBOX

event-44 still appears pending

After restart, the relay may publish the same event again.

The outbox has prevented loss.

It has not automatically prevented duplicate publication.

Marking progress first can lose the event

The relay might reverse the order:

1. Mark event-44 as published.
2. Publish event-44.

A crash between those steps produces:

OUTBOX

event-44 marked published


KAFKA

event-44 missing

The relay will not retry because the row appears complete.

This is the same dual-write problem in a smaller coat.

The usual preference is to publish first and record progress afterwards. Duplicate publication can be handled through identity and idempotency. A missing event may be much harder for consumers to discover.

Separate the reliability boundaries

Several mechanisms may participate in the path:

Kafka producer idempotence protects Kafka writes under its producer protocol.

It does not atomically coordinate:

• the relational outbox row;
• a polling relay’s published_at marker;
• a consumer database;
• an email provider;
• a payment processor.

A relay can restart under a new producer session and publish the same application event again. A transactional Kafka producer can strengthen Kafka-side publication across sessions when configured with a stable transactional identity, but the database row still sits outside that Kafka transaction.

The event’s stable application identity remains necessary.

Kafka-to-Kafka transactions have a useful but bounded role

Kafka transactions can atomically publish Kafka records and commit consumed Kafka offsets in a consume-transform-produce pipeline.

That can provide exactly-once processing within the Kafka transaction boundary.

It does not automatically include:

• MySQL or PostgreSQL;
• a search index;
• an email provider;
• a payment provider;
• an arbitrary RPC.

The transactional outbox protects the database-state-to-message relationship.

Kafka transactions protect a different boundary.

Consumer idempotency protects another.

Stretching one guarantee across all three creates confidence that the system has not earned.

5. Ordering must survive the relay

Suppose one order transaction creates:

order-5001 version 6:
    OrderConfirmed

A later transaction creates:

order-5001 version 7:
    OrderCancelled

The outbox guarantees that both publication obligations exist durably.

It does not automatically decide which worker publishes first.

created_at is not enough

Ordering by created_at can be useful, but it is not a complete strict-ordering mechanism.

Problems include:

• identical timestamps;
• clock precision;
• transactions committing in another order;
• several workers claiming overlapping ranges;
• retries allowing a later event to overtake an earlier one.

A stronger design can use:

• a database-generated outbox sequence;
• aggregate ID;
• aggregate version;
• event ID as a deterministic tie-breaker;
• a unique constraint on (aggregate_id, aggregate_version).

For example:

CREATE TABLE outbox_events (
    outbox_sequence BIGINT GENERATED ALWAYS AS IDENTITY,
    event_id VARCHAR(128) PRIMARY KEY,
    aggregate_id VARCHAR(128) NOT NULL,
    aggregate_version BIGINT NOT NULL,
    event_type VARCHAR(255) NOT NULL,
    payload BLOB NOT NULL,
    created_at TIMESTAMP NOT NULL,
    UNIQUE (aggregate_id, aggregate_version)
);

The sequence describes one database ordering surface.

The aggregate version describes one entity’s business history.

They answer different questions.

Relay and Kafka key must agree

Suppose order events use:

Kafka key:
    order_id

All events for order-5001 reach one Kafka partition.

Kafka preserves the append order within that partition.

The relay must still append them meaningfully:

RELAY

version 6
then
version 7

Kafka cannot repair:

RELAY

version 7
then
version 6

The broker preserves the order it receives.

The producer and relay are responsible for producing a meaningful sequence.

Possible relay policies include:

• publish in deterministic outbox sequence;
• partition relay work by aggregate ID;
• allow only one in-flight publication per aggregate;
• use the same aggregate ID as the Kafka key;
• make consumers version-aware;
• quarantine later versions when a prior version cannot be published.

Define the version-gap response

Suppose a consumer currently holds order version 5 and receives version 7.

The version reveals that something is missing.

It does not decide what happens next.

Possible policies include:

• pause and wait for version 6;
• fetch an authoritative snapshot;
• replay from an earlier broker position;
• quarantine version 7;
• apply version 7 when the projection needs only latest state;
• rebuild the projection.

The event contract should define which policy is valid for that consumer.

Poison outbox rows need isolation

One outbox row may never publish successfully because of:

• an unknown event type;
• a payload serialization failure;
• an invalid destination;
• an oversized broker record;
• a missing partition key;
• revoked credentials;
• a schema that the current relay can no longer read.

A relay should not let one poison row block unrelated aggregates forever.

It needs a controlled state such as:

Strictly ordered aggregates need special treatment. Moving version 6 aside while publishing version 7 creates a gap. The system may need to block that aggregate while allowing unrelated aggregates to continue.

Quarantine is not silent deletion.

It is another unresolved obligation with an owner and a recovery path.

6. Operate the publication obligation

The outbox protects publication only while the relay continues to discharge its obligations.

A durable table that nobody is reading is still a failure.

Measure age, not only count

Useful measurements include:

• pending row count;
• oldest pending row age;
• time from business commit to broker publication;
• publication attempts and failures;
• quarantined row count;
• rows stuck in a claimed state;
• connector lag or polling delay;
• event type and destination;
• aggregate IDs affected by ordered gaps.

The most revealing signal is often the oldest unpublished event age.

Ten thousand pending rows may be healthy during a short traffic spike.

One record that has remained unpublished for six hours may indicate a poison row, broken route or stalled relay.

An outbox backlog is unfinished communication

Suppose order has committed 5,000 confirmed orders, but publication is twenty minutes behind.

Consumers may be missing:

• shipping work;
• customer notifications;
• analytical updates;
• recommendation signals.

The order API and database may look healthy.

The wider capability is degraded.

ORDER API

healthy


OUTBOX RELAY

behind


FULFILMENT

waiting for events

An outbox backlog is not merely housekeeping.

It is unfinished communication caused by already committed business state.

Cleanup is part of the design

Published rows cannot grow forever without consequence.

The system may:

• delete rows after a recovery and investigation period;
• archive limited metadata;
• partition the table by date;
• retain rows required for audit;
• delete CDC rows after connector progress and recovery needs are satisfied.

Immediate deletion reduces evidence.

Infinite retention increases indexes, backups, cost and privacy obligations.

The outbox is operational state, not an immortal scrapbook.

The outbox is not automatically the replay store

Consumer replay normally comes from:

• a retained broker stream;
• an event store;
• an authoritative snapshot;
• a projection rebuild process.

The outbox primarily protects publication.

Its retention should normally support:

• relay recovery;
• controlled republishing;
• incident investigation;
• publication evidence;
• audit where required.

It should not be retained forever merely because consumers might one day need history. That is a different storage responsibility.

Privacy and access still apply

Outbox rows may contain:

• customer identifiers;
• addresses;
• purchased products;
• payment references;
• operational metadata.

The table may be visible to application developers, database administrators, support tools and CDC infrastructure.

The event payload should contain only supported facts required by consumers. Retention and access should reflect the sensitivity of those facts.

The outbox should not become the one table where every privacy rule quietly takes the afternoon off.

Assign operational ownership

Someone must own:

• relay alerts;
• poison rows;
• connector failures;
• publication lag;
• schema compatibility;
• cleanup;
• controlled republishing;
• final verification.

The database team may keep the table available.

The platform team may operate Kafka Connect.

The service team still owns the meaning of the event and the obligation created by its business transaction.

7. What the pattern does not solve

The transactional outbox solves one relationship:

Local business state and the durable record of the message caused by that state change are committed together.

Several other responsibilities remain separate.

It does not guarantee immediate consistency

After the transaction commits:

Publication may take milliseconds or minutes during an incident.

The outbox improves reliable eventual publication.

It does not make asynchronous messaging immediate.

It does not make consumers correct

A consumer may:

• apply the event twice;
• reject a compatible schema;
• process versions out of order;
• create a partial local transaction;
• fail permanently;
• contain incorrect business logic.

The outbox worked if the event was durably recorded and published.

Every consumer still owns its own reliability boundary.

It does not finish a distributed workflow

Suppose OrderConfirmed reaches three groups:

The outbox can establish that order recorded and published its event.

It cannot declare the complete workflow successful.

A process requiring several mandatory outcomes may need explicit workflow state or orchestration.

The outbox is not a distributed workflow engine.

It does not coordinate several databases

An outbox in the order database can atomically record:

• order state;
• messages order must publish.

It cannot atomically commit inventory and payment state.

Each service may use an outbox around its own local transaction.

ORDER

local state + outbox


INVENTORY

local state + outbox


PAYMENT

local state + outbox

The wider process still depends on messages, retries, timeouts, compensation and reconciliation.

It does not replace idempotent commands

Suppose order receives the same command twice:

PlaceOrder

checkout_id:
    checkout-7f82

If the handler creates two orders, it can faithfully write two outbox events.

The outbox has preserved both duplicated business effects perfectly.

The command handler still needs a one-winner rule, such as a unique checkout_id.

The same transaction can protect both:

BEGIN
  │
  ├── enforce unique checkout_id
  ├── create order-5001
  └── insert OrderCreated outbox event
COMMIT

Command idempotency prevents duplicate business state.

The outbox ensures the accepted state change creates a durable publication obligation.

It does not create exactly-once external effects

The complete path may contain:

one database transaction

two event publications

two deliveries

one idempotent business effect

The useful result is still one business effect.

That comes from combining:

• stable event identity;
• consumer idempotency;
• business uniqueness constraints;
• idempotency keys;
• destination status checks;
• reconciliation where outcomes are uncertain.

The outbox provides reliable local message creation.

Exactly-once language should not be stretched across every boundary that follows.

It does not replace reconciliation

A destination may accept a request while its response is lost.

The outbox row may still appear unresolved.

The next attempt may repeat an external effect.

The system may need to query the destination and establish what actually happened.

The outbox tells us:

publication obligation remains unresolved

It does not always tell us:

the destination performed no work

8. A bfstore outbox and design checklist

Suppose bfstore confirms order-5001.

The order transaction performs:

BEGIN
  │
  ├── verify order is confirmable
  ├── set status = CONFIRMED
  ├── increment order_version to 7
  └── insert outbox event-44
COMMIT

The outbox record contains:

The payload contains order-owned facts:

message OrderConfirmed {
  EventMetadata metadata = 1;
  string order_id = 2;
  string customer_id = 3;
  Money total = 4;
  repeated ConfirmedOrderLine lines = 5;
  uint64 order_version = 6;
}

The relay publishes using:

If publication succeeds but a polling relay crashes before recording progress, event-44 may be published again.

Each consumer protects its own effect:

The outbox ensures that order confirmation and the obligation to publish confirmation cannot be separated by a process crash.

Every later component still owns its own reliability boundary.

Transactional outbox checklist

That final question matters more than it first appears.

The mental model I am keeping

My original model was:

update database
      │
      ▼
publish event
      │
      ▼
hope both happen

The stronger model is:

                     LOCAL DATABASE TRANSACTION
                               │
                 ┌─────────────┴─────────────┐
                 │                           │
                 ▼                           ▼
          BUSINESS STATE                OUTBOX RECORD
                 │                           │
                 └─────────────┬─────────────┘
                               ▼
                             COMMIT
                               │
                               ▼
                       MESSAGE RELAY
                               │
                               ▼
                            BROKER
                               │
                               ▼
                          CONSUMERS

The database transaction guarantees:

business state and publication obligation
are recorded together

The relay is responsible for repeatedly attempting publication until the obligation is completed, quarantined or deliberately resolved.

The stable event identity allows duplicate publication and delivery to be recognised.

Consumer idempotency protects downstream business effects.

Monitoring protects against an outbox that is durable but no longer moving.

The transactional outbox therefore solves one particular problem:

How can a service commit local state and durably record the message caused by that state change without requiring the database and broker to share one transaction?

It does not guarantee:

• immediate publication;
• single publication;
• single delivery;
• correct consumers;
• complete workflows;
• atomic changes across services;
• exactly-once external effects.

Those remain separate design problems.

This narrowness is not a weakness.

It is what makes the pattern understandable.

The outbox does not promise to conduct the entire distributed orchestra.

It makes sure one musician cannot finish the movement and forget to leave the next player their sheet music.

References and further reading

Chris Richardson: Transactional Outbox
Describes storing an outbound message in the same database transaction as the business-state update, followed by publication through a separate relay.

Chris Richardson: Polling Publisher
Describes a relay that periodically queries an outbox table and publishes pending messages.

Chris Richardson: Transaction Log Tailing
Describes publishing messages by reading committed database changes from the transaction log.

Debezium: Outbox Event Router
Documents insert-oriented outbox rows, event identity, aggregate routing and cleanup behaviour for CDC publication.

Debezium: Exactly-once delivery
Explains Debezium’s default at-least-once model and the supported exactly-once source-delivery configuration.

Apache Kafka: Producer configuration
Documents producer idempotence, acknowledgements, retries and transactional identities.

Apache Kafka: Design
Documents Kafka transactions and exactly-once processing boundaries.

Martin Kleppmann: Designing Data-Intensive Applications
Explores distributed transactions, logs, stream processing and the difficulty of coordinating writes across independently failing systems.