The phrase event-driven architecture can make a system sound as though it has acquired its own weather system.
Diagrams fill with event buses, event meshes, event backbones, real-time streams and reactive consumers.
The boxes become clouds.
The arrows become lightning.
The architecture begins to sound impressive before I can explain what one component actually does.
So I want to remove the grand language for a moment.
At its simplest, an event-driven interaction looks like this:
PRODUCER
│
│ publishes an assertion about a completed change
▼
EVENT CHANNEL
│
│ delivers or stores it
▼
CONSUMER
│
└── reacts
For example:
CATALOG
│
│ ProductPriceChanged
▼
MESSAGE BROKER
│
├──► SEARCH
├──► BASKET
└──► ANALYTICS
Catalog changes a product price.
Catalog publishes an event asserting that the change occurred.
Search updates its index.
Basket invalidates a cached product summary.
Analytics records the change.
Catalog does not call those consumers one by one.
The consumers do not need to perform their work inside catalog’s transaction.
That is the useful centre of event-driven architecture:
A component publishes authoritative assertions about changes it owns, and independently running consumers react according to their own responsibilities.
Everything else concerns how safely, quickly and understandably that interaction happens.
Event-driven architecture begins with an owned fact
The previous Note separated commands from events.
A command asks an owner to attempt an action:
ChangeProductPrice
An event reports an accepted result:
ProductPriceChanged
They may travel through the same broker, but they make different claims.
| Message | Claim |
|---|---|
| Command | Please attempt this work. |
| Event | The owner asserts that this change has occurred. |
That distinction matters because an event consumer is not being asked to approve the original decision.
Suppose catalog publishes:
ProductPriceChanged
product_id:
chair-42
price_before:
£229.00
price_after:
£249.00
Search cannot reverse the catalog decision merely because its index is unavailable.
Basket cannot declare that the old price remains authoritative because cache invalidation failed.
Catalog owns the current catalog price.
The event tells consumers what catalog says it committed.
Consumers own their reactions.
| Component | Owned state |
|---|---|
| Catalog | Product price |
| Search | Search projection |
| Basket | Basket representation |
| Analytics | Analytical record |
An event is an authoritative assertion, not an infallible object. Software defects, corrupt source data, faulty migration logic or compromised publishers can still produce incorrect events.
A consumer may reject or quarantine a message that is malformed, unauthorised, impossible under the contract or sent by an untrusted source. What the consumer does not normally do is remake the owner’s original business decision merely because its own processing failed.
The producer should own the assertion
Catalog may publish:
ProductCreatedProductPublishedProductPriceChangedProductDiscontinued
Inventory may publish:
StockReservedStockReleasedReservationExpired
Payment may publish:
PaymentAuthorisedPaymentDeclinedPaymentCapturedPaymentRefunded
Order should not publish PaymentCaptured merely because it observed a successful payment response.
Payment owns whether capture occurred.
Order might publish OrderMarkedAsPaid if that is a meaningful fact within the order lifecycle.
These are related assertions, but they are not the same:
| Owner | Assertion |
|---|---|
| Payment | Payment captured the money. |
| Order | Order accepted payment as complete. |
An event-driven system becomes confused when several services publish competing versions of facts they do not own.
The event name should point back to one authoritative source.
Corrections are new facts
An event can later turn out to be wrong or require correction.
The normal repair is not to silently rewrite the original event. It is to publish another event describing the correction:
PaymentAuthorised
│
▼
PaymentAuthorisationReversed
or:
ProductPriceChanged
│
▼
ProductPriceCorrectionRecorded
The correcting event does not pretend the earlier assertion was never made.
It records another occurrence that changes how consumers should interpret the history.
This distinction matters whenever events are retained, replayed, audited or used to explain past decisions.
The channel changes time and coupling
Producer time and consumer time separate
In a synchronous call, both participants must normally be available during the interaction.
CATALOG
│
│ invalidate request
▼
BASKET
If basket is unavailable, catalog’s call fails or waits.
With a durable asynchronous channel:
CATALOG
│
│ ProductPriceChanged
▼
BROKER
│
▼
BASKET
catalog can publish while basket is offline.
Basket processes the event later:
14:00
catalog publishes event
14:05
basket restarts
14:06
basket processes event
This is temporal decoupling.
The producer and consumer do not need to be running at precisely the same moment.
That can improve resilience.
It also means they may temporarily disagree.
CATALOG
price = £249.00
BASKET CACHE
price = £229.00
Until basket processes the event, its derived copy remains stale.
The channel has exchanged immediate consistency for asynchronous progress.
The producer need not know every consumer
Suppose catalog originally has two consumers:
ProductPublished
│
├──► search
└──► analytics
Later, recommendation needs the same assertion:
ProductPublished
│
├──► search
├──► analytics
└──► recommendation
Ideally, catalog does not need application code such as:
search.NotifyProductPublished(product)
analytics.RecordProductPublished(product)
recommendation.AddProduct(product)
Catalog publishes one supported event.
Each consumer subscribes independently.
Catalog knows the event contract.
Each consumer knows the event contract.
Catalog does not need to know which consumers currently exist.
This is one of event-driven architecture’s main attractions. New reactions can be added without turning the producer into a switchboard operator.
The coupling moved
The producer and consumer may not know each other’s network addresses.
They still share dependencies:
- the event schema;
- the meaning of the event;
- the broker or channel;
- delivery policy;
- ordering policy;
- retention policy;
- access policy.
If catalog changes ProductPriceChanged incompatibly, consumers can break.
If the broker is unavailable, publication may fail.
If retention is too short, an offline consumer may lose the opportunity to catch up.
If the event meaning is vague, each consumer may interpret it differently.
The system has reduced direct runtime coupling while introducing contract and infrastructure coupling.
| Removed | Added |
|---|---|
| Producer-to-consumer call dependency | Shared event contract |
| Immediate consumer availability | Shared delivery infrastructure |
| Producer knowledge of subscribers | Asynchronous consistency and lag |
The coupling moved.
It did not evaporate.
A broker is not the architecture
Installing Kafka or RabbitMQ does not automatically produce event-driven architecture.
SERVICE A
│
│ message
▼
BROKER
│
▼
SERVICE B
This proves only that a broker sits between two processes.
The message might be:
- a command;
- an event;
- a request awaiting a response;
- a file-processing job;
- a batch of database rows;
- an undifferentiated byte parcel.
The architecture comes from decisions around ownership, meaning, delivery, ordering, failure, recovery and consumer responsibility.
A broker is a transport and storage mechanism.
It cannot decide whether ProductUpdated is an authoritative business assertion, a command disguised in past tense or a database trigger wearing a name badge.
Different channels preserve different possibilities
Publish-subscribe and work distribution are different shapes
One event can be made available to several independent subscribers:
ProductPublished
│
├──► search subscription
├──► analytics subscription
└──► recommendation subscription
Each logical consumer receives the event for its own purpose.
Search processing the event does not remove analytics’ copy.
Several worker instances can also share one logical subscription or queue:
image jobs
│
▼
QUEUE
│ │ │
▼ ▼ ▼
W1 W2 W3
One worker handles each delivery within that consumer group.
The distinction is:
| Shape | Meaning |
|---|---|
| Multiple subscriptions | Each logical consumer needs the event. |
| Multiple instances in one subscription | Instances share the work of one logical consumer. |
An event-driven system often uses both:
ProductPublished
│
├──► search consumer group
│ ├── search instance 1
│ └── search instance 2
│
└──► analytics consumer group
├── analytics instance 1
└── analytics instance 2
Queues and streams emphasise different recovery models
A queue-shaped model primarily represents pending work:
QUEUE
message A
message B
message C
After successful acknowledgement, message A may no longer be available through the ordinary consumption path.
A retained event stream behaves more like an append-only log:
PARTITION
offset 41 ProductCreated
offset 42 ProductPublished
offset 43 ProductPriceChanged
offset 44 ProductDiscontinued
Consumers maintain positions:
search position:
44
analytics position:
42
Search has processed further than analytics.
Retention can allow:
- replay;
- consumer catch-up;
- new consumers reading earlier events;
- rebuilding derived state.
A queue asks:
Which work remains to be handled?
A retained stream can also answer:
Which ordered records remain available, and how far has each consumer read?
This is a useful model, not a rigid product taxonomy. Some queue systems retain or replay messages. Some streams compact or expire records. Actual behaviour depends on acknowledgement, retention, compaction, consumer-position management, redelivery and administrative replay support.
The required recovery and consumption behaviour should choose the channel shape.
Request-reply over a broker can still be synchronous
A sender can publish a request and wait for a reply:
REQUEST QUEUE
│
▼
CONSUMER
│
▼
REPLY QUEUE
│
▼
SENDER WAITS
The transport is asynchronous.
The interaction is synchronous from the caller’s perspective.
The caller still depends on consumer availability, response latency, deadlines, correlation and reply delivery.
This pattern can be useful.
It should not be described as decoupled event collaboration merely because queues carry the bytes.
The useful question is whether the sender can continue without the receiver’s immediate result.
Choose what the event carries
Event notification carries a small signal
One style announces that something changed without carrying the complete new state:
message ProductPriceChanged {
string event_id = 1;
string product_id = 2;
}
A consumer receives the event and asks catalog for current data:
ProductPriceChanged
│
▼
consumer calls catalog
│
▼
current product retrieved
Advantages include:
- a small payload;
- less duplicated product data;
- catalog remains the source of current detail.
Costs include:
- another network call;
- runtime dependency on catalog;
- the consumer may observe a newer version than the triggering change;
- many subscribers may call catalog at once.
One notification can become a callback storm:
one ProductPriceChanged
│
├── search calls catalog
├── basket calls catalog
├── analytics calls catalog
└── recommendation calls catalog
Suppose the price changes twice:
£229.00 → £249.00 → £259.00
A delayed consumer receives the first notification and calls catalog.
It may retrieve £259.00.
That can be correct when the consumer only needs the latest state.
It is not sufficient when the consumer needs to reconstruct each historical transition.
Event-carried state transfers useful data
Another event includes the state consumers need:
message ProductPriceChanged {
string event_id = 1;
string product_id = 2;
Money price_before = 3;
Money price_after = 4;
uint64 product_version = 5;
google.protobuf.Timestamp occurred_at = 6;
}
A consumer can update its local projection without calling catalog.
ProductPriceChanged
│
▼
update local product view
Advantages include:
- fewer synchronous callbacks;
- consumers can remain available while catalog is down;
- replay can rebuild a derived view.
Costs include:
- larger messages;
- duplicated data;
- more sensitive schemas;
- greater privacy exposure;
- more compatibility responsibility.
This is often called event-carried state transfer.
The consumer’s local data remains derived.
Catalog still owns the current product facts.
Event sourcing is a different decision
Event-driven architecture does not automatically mean event sourcing.
In an ordinary event-publishing service, a database may remain authoritative:
CATALOG DATABASE
│
└── current product state
EVENT STREAM
│
└── selected changes announced
In an event-sourced model, the event history itself is the authoritative record of state changes:
ProductCreated
ProductRenamed
ProductPriceChanged
ProductPublished
Current state is rebuilt by applying those events:
EVENT HISTORY
│
▼
replay
│
▼
CURRENT STATE
That is a much stronger commitment. It affects data modelling, storage, versioning, rebuilds, correction, privacy, query design and recovery.
A system can use events extensively without using event sourcing.
Calling every retained stream an event store muddles two different responsibilities.
Historical meaning is not permanent storage
An event describes a past occurrence.
The channel is not automatically an audit system or permanent historical record.
Retention may expire.
Compaction may remove earlier records.
Privacy controls may require deletion or restricted access.
Event semantics are historical. Event storage is only as durable and complete as its explicit retention and audit design.
Not everything must be asynchronous
Some interactions require an immediate authoritative answer.
Suppose basket needs to know whether a product is currently purchasable:
BASKET
│
│ GetPurchasableProduct
▼
CATALOG
A synchronous RPC may be appropriate because the caller needs a current answer to continue.
After basket accepts the item, it may publish BasketItemAdded for analytics or recommendation:
USER REQUEST
│
▼
synchronous validation
│
▼
local state change
│
▼
asynchronous event
Real systems often combine synchronous queries, synchronous commands, asynchronous commands, events and batch processing.
Replacing every direct call with a broker can turn a simple question into a complicated correspondence course.
Events create eventual consistency
Suppose catalog changes a price at 14:00.
Search processes the event at 14:00:01.
Basket processes it at 14:00:05.
Between those times, the system contains several versions of the product:
CATALOG
new state
SEARCH
perhaps new state
BASKET
perhaps old state
This does not necessarily mean the system is broken.
It means consistency occurs over time rather than through one distributed transaction.
The design needs to define:
- which copy is authoritative;
- how stale each derived view may become;
- how consumers catch up;
- what happens when they cannot catch up;
- which operations must bypass stale data.
The storefront may show a slightly stale browse price.
Checkout should validate the currently accepted price with the authoritative owner before committing the order.
Publish assertions reliably
Publication creates a dual-write problem
Suppose catalog updates its database and publishes an event as two separate operations:
1. update database
2. publish ProductPriceChanged
Several outcomes are possible.
If the database succeeds and publication fails:
CATALOG DATABASE
price = £249.00
EVENT STREAM
no ProductPriceChanged
Consumers remain stale.
If publication succeeds and the database fails:
EVENT STREAM
ProductPriceChanged to £249.00
CATALOG DATABASE
price remains £229.00
Consumers received an assertion that catalog never committed.
The database and broker do not normally participate in one local transaction.
This is the dual-write problem.
The transactional outbox records the assertion locally
One response is to place the business change and an outbound event record in the same database transaction:
BEGIN
│
├── update product price
└── insert outbox event
COMMIT
The database now contains:
PRODUCT
price = £249.00
OUTBOX
ProductPriceChanged pending publication
A separate relay reads the outbox and sends the event:
OUTBOX
│
▼
PUBLISHER
│
▼
BROKER
If publication fails, the relay retries.
The product price is not changed again.
The recorded assertion is delivered later.
The outbox does not guarantee that the broker receives the event exactly once.
It makes safe repeated publication possible.
Ordering must survive the relay
Where consumers rely on per-aggregate ordering, the outbox must record enough information to preserve it:
- aggregate or entity identity;
- aggregate version or sequence;
- event ID;
- event type;
- partition key;
- occurred time.
Several relay instances may publish concurrently.
An older outbox row may be retried after a newer one.
Two changes to the same aggregate may commit close together.
The owner should assign the event’s aggregate version and routing key when the authoritative transition is recorded, not invent them later in the relay.
The relay and broker configuration must preserve or allow consumers to reconstruct the promised order.
Event identity is part of the contract
The event contract should define the scope of event_id.
Is it unique:
- globally;
- within one producer;
- within one source and event type;
- within one aggregate?
Consumers also need a rule for conflicting reuse. The same event ID with different payload content should normally be treated as corruption or a contract violation, not as a harmless duplicate.
Infrastructure acknowledgements are not business completion
A broker may confirm that it accepted a publication.
A consumer may acknowledge that it processed a delivery.
These signals describe infrastructure progress:
| Signal | Meaning |
|---|---|
| Publisher confirm | The broker accepted the event under its configured durability policy. |
| Consumer acknowledgement | This consumer completed the work required before acknowledging. |
| Business result | The consumer’s own state changed correctly. |
The application must define what happens before acknowledgement.
Acknowledging before the consumer commits its own state can lose the reaction.
Committing before acknowledgement can cause redelivery after a crash.
That leads to duplicate-safe consumption.
Consumers must survive repetition and disorder
At-least-once delivery creates duplicates
A consumer may process an event successfully and then lose its connection before acknowledging:
broker delivers event
│
▼
consumer updates database
│
▼
acknowledgement lost
│
▼
broker redelivers event
The broker cannot know that the application’s database commit succeeded.
The consumer may see the same event again.
This is normal under at-least-once delivery.
Deduplication must be atomic with the reaction
Suppose notification consumes:
OrderConfirmed
event_id:
event-44
Without duplicate protection, two deliveries can create two confirmation jobs.
A safer shape is:
BEGIN
│
├── claim event-44
├── record notification work
└── commit
The event claim and local effect should form one protected transition where possible.
Otherwise this race remains:
check event not processed
apply business change
crash before recording event ID
An inbox table, unique constraint, conditional state transition or naturally unique business key can make the reaction duplicate-safe.
For example:
one order-confirmation notification
per order ID
can be protected by a unique constraint.
The correct identity depends on the effect being protected.
Exactly once needs a boundary
A broker may provide transactional or exactly-once features within a defined scope.
The complete business operation may still cross:
- broker;
- consumer database;
- email provider;
- payment provider;
- another broker.
The useful questions are:
Exactly once where?
For which records?
Inside which transaction boundary?
What happens when an external effect succeeds
and its acknowledgement is lost?
A duplicate event can still create one business effect through idempotency.
A singly delivered event can create two effects because of an application bug.
The business cares about effect count, not merely delivery count.
Ordering is usually scoped
Consumers may need events ordered for one product:
ProductCreated
ProductPublished
ProductPriceChanged
ProductDiscontinued
They may not need one total order across unrelated products.
A partitioned log can use product_id as the key so related events share one ordered partition.
The event contract should state the scope:
- all events for one product;
- only price events for one product;
- all catalog events;
- another owner-defined sequence.
Global ordering is expensive and often unnecessary.
Choose the smallest ordering scope the business requires.
Ordered delivery does not guarantee ordered effects
A broker can make records available in partition order.
Application processing can still reorder effects through:
- parallel handler execution;
- asynchronous database writes;
- retries;
- delayed external calls;
- consumer rebalances.
Where order matters, the consumer must preserve that order through its own processing and state commits.
Broker ordering is only one part of the guarantee.
Versions detect disorder but do not repair it
Suppose the consumer currently holds product version 19.
A delayed version 18 arrives.
The consumer can ignore it.
A harder case is:
current version:
18
incoming version:
20
Version 19 is missing.
The consumer needs a policy appropriate to its projection:
- buffer version 20 and wait;
- fetch an authoritative snapshot;
- apply version 20 because only the latest state matters;
- quarantine and alert;
- trigger reconciliation.
The version number detects a gap.
It does not decide how the gap is repaired.
Timestamps alone are usually weaker because clocks differ and several changes can occur close together.
Replay and retention shape recovery
Replay can rebuild derived state
A retained log allows a consumer to reset its position and process earlier events again.
This can support:
- rebuilding a search index;
- creating a new analytical projection;
- repairing data after a consumer bug;
- testing revised consumers against historical events.
EVENT LOG
│
├── ProductCreated
├── ProductPublished
├── ProductPriceChanged
└── ProductDiscontinued
│
▼
REBUILD SEARCH INDEX
Replay is powerful because consumers own their positions.
It is also capable of repeating every side effect.
A replay-safe design must distinguish rebuilding derived state from resending emails, recharging payments or reissuing refunds.
Not every consumer should replay through its ordinary production side-effect path.
Retention is part of recovery
Suppose a consumer is offline for three days but events are retained for one day.
When the consumer returns, its missing events are gone.
Recovery may require:
- loading a current snapshot;
- rebuilding from an authoritative database;
- restoring from backup;
- accepting lost historical detail.
Retention should reflect maximum expected outage, replay needs, audit obligations, storage cost and privacy requirements.
A durable stream is not infinitely durable by default.
Event schemas are long-lived contracts
Events may be consumed by old services, new services, replay jobs, future consumers and analytical pipelines.
With Protobuf, additive fields can support overlap:
message ProductPriceChanged {
string event_id = 1;
string product_id = 2;
Money price_after = 3;
Money price_before = 4;
}
Older consumers ignore fields they do not know.
New consumers tolerate absence in historical records.
Unsafe changes include:
- reusing field numbers;
- changing field meaning;
- changing incompatible field types;
- removing fields consumers still require.
An event is not private merely because only one consumer exists today.
Publish-subscribe makes future consumers unusually easy to add.
Privacy follows the copies
Events may be visible to consumers, broker administrators, logging systems, schema tools, support operators and development environments.
The payload should contain only data required by supported consumers.
Avoid publishing complete customer records, payment credentials, unnecessary addresses, private notes, secrets or authentication tokens merely because the producer has them.
An event can outlive the original request and be copied into several systems.
Privacy design therefore needs to consider retention, access, log redaction, encryption, erasure obligations and downstream copies.
Fan-out is useful.
It also increases the audience.
Operate the reaction graph
Choreography distributes workflow knowledge
In choreography, services react to events without one central workflow component directing every step:
OrderCreated
│
▼
inventory reacts
│
▼
StockReserved
│
▼
payment reacts
│
▼
PaymentAuthorised
Advantages include low direct coupling and the ability to add reactions.
Challenges include:
- workflow state is distributed;
- no single component may know overall progress;
- timeouts and compensation can be unclear;
- debugging requires reconstructing the chain.
Consumers own how they fulfil their responsibility, although the wider workflow may explicitly depend on particular consumers reacting.
Events reduce direct addressing.
They do not eliminate workflow coupling or obligations.
Orchestration makes workflow state explicit
An orchestrator can record the multistep process:
CHECKOUT ORCHESTRATOR
│
├── ReserveStock
├── AuthorisePayment
└── ConfirmOrder
It tracks which step is pending, which completed, which retry is safe and which compensation is required.
Advantages include visible workflow state, explicit timeout handling, retries and compensation.
Costs include central workflow coupling, additional state and availability requirements, and knowledge of participant contracts.
Orchestration is not the opposite of event-driven architecture.
An orchestrator can consume events and publish commands.
The difference is whether one component explicitly owns the multistep process.
Buffering absorbs spikes, not sustained imbalance
Suppose producers publish 1,000 events per second and consumers process 800.
The backlog grows by 200 events per second.
For a short spike, buffering is useful.
For sustained imbalance, the queue grows indefinitely.
The system may need consumer scaling, producer rate control, load shedding, batching, backpressure or more processing capacity.
A broker is a shock absorber, not an infinite warehouse.
Lag measures freshness
A consumer can be running with a low error rate while serving data that is hours old.
Useful measurements include:
- event-count lag;
- oldest unprocessed event age;
- incoming rate;
- processing rate;
- estimated time to catch up.
Count alone is not enough. Five hundred events might represent one second or five hours.
Architecture needs capability-level freshness, not merely green process health.
Failed events need owned states
A consumer may fail because a dependency is unavailable, the event is malformed, the schema is unsupported, domain state is unexpected or the application contains a bug.
Transient failures may justify delayed retry.
Permanent failures may require quarantine.
EVENT
│
▼
PROCESSING FAILURE
│
├── transient ──► retry later
│
└── permanent ──► failed-event state
A dead-letter queue can prevent one bad event from blocking the main flow.
Moving it aside is not recovery.
The system still needs to know:
- who owns it;
- how it is investigated;
- whether it can be corrected;
- whether replay is safe;
- whether later state has made it obsolete;
- when an operator should be alerted.
A dead-letter queue without an operating process is simply a well-organised attic.
Poison events can amplify failure
One malformed event can cycle through every consumer instance:
receive event
│
▼
consumer fails
│
▼
event redelivered
│
▼
consumer fails again
Useful controls include bounded attempts, delayed retries, validation, quarantine, dependency circuit breaking and alerts.
Retry policy belongs to event processing just as it belongs to synchronous RPC.
Events can isolate and spread failure
When notification is unavailable, an OrderConfirmed event can allow order to commit while notification catches up later.
The failure has not disappeared. It has become backlog, delayed communication and operational responsibility.
Events can also spread failure through hidden reactions. One schema change may break ten consumers. One publication burst may overload several downstream systems. One accidental loop can create:
event A triggers event B
event B triggers event C
event C triggers event A
An event catalogue and dependency map should answer:
- who publishes each event;
- who consumes it;
- which events each consumer may publish;
- which customer capabilities depend on the chain.
Observability replaces the missing call stack
An event-driven flow can unfold over minutes:
order commits
│
▼
event published
│
▼
inventory consumes
│
▼
another event published
│
▼
payment consumes
Useful telemetry includes:
- publication success and latency;
- event ID and aggregate version;
- producer and event type;
- topic, partition and offset;
- consumer group;
- delivery attempt;
- processing duration;
- lag and event age;
- acknowledgement result;
- failed-event count;
- correlation and causation IDs;
- trace context.
The goal is to answer:
Was the event recorded?
Was it delivered?
Which consumer processed it?
What state did that consumer commit?
Is another consumer still behind?
Can the workflow be replayed or reconciled?
A dashboard showing only broker CPU cannot answer those questions.
Test time, repetition and gaps
Tests should cover duplicate delivery, delayed delivery, out-of-order records, missing versions, consumer restarts, broker disconnection, replay, schema overlap, poison events, large backlogs and partial consumer outages.
For example:
1. deliver ProductPriceChanged version 18
2. deliver version 19
3. redeliver version 18
expected:
projection remains at version 19
Or:
1. process OrderConfirmed
2. crash before acknowledgement
3. redeliver OrderConfirmed
expected:
one notification job exists
The interesting bugs often live between attempts rather than inside one clean handler call.
Schema checks, producer contract tests, consumer tests, broker integration tests and a small set of end-to-end journeys provide different kinds of evidence.
Design framework and a bfstore example
When event-driven interaction fits
It is especially useful when:
- several consumers need the same assertion;
- consumers can react independently;
- producer and consumer need temporal decoupling;
- work can happen after the original request;
- derived views need asynchronous updates;
- traffic arrives in bursts;
- replay is valuable;
- the business naturally contains meaningful events.
Examples include updating search, sending notifications, feeding analytics, processing uploaded images, building recommendations and maintaining read projections.
When it may not fit
It may be a poor default when:
- the caller needs an immediate authoritative answer;
- the complete operation belongs in one local transaction;
- the workflow is simple and contained;
- the team cannot yet operate asynchronous failure;
- eventual consistency is unacceptable;
- replay provides little value;
- one producer and one consumer always change together.
The existence of a broker does not require every interaction to pass through it.
A possible bfstore flow
Suppose a customer completes checkout.
The request reaches order synchronously.
Order validates the request and records order-5001 as pending.
The workflow sends ReserveStock to inventory.
Inventory commits the reservation and records a StockReserved outbox event.
The event is published:
INVENTORY
│
│ StockReserved
▼
EVENT CHANNEL
│
├──► ORDER WORKFLOW
└──► ANALYTICS
The workflow sends AuthorisePayment.
Payment publishes PaymentAuthorised.
Order confirms the order and publishes OrderConfirmed:
OrderConfirmed
│
├──► NOTIFICATION
├──► SHIPPING
├──► ANALYTICS
└──► RECOMMENDATION
Each consumer owns a different reaction.
| Consumer | Reaction |
|---|---|
| Notification | Create one confirmation-delivery job |
| Shipping | Begin fulfilment planning |
| Analytics | Record confirmed-order metrics |
| Recommendation | Update customer preference signals |
Order does not call all four consumers inside its confirmation transaction.
The event connects them after the fact.
The event still needs a contract
syntax = "proto3";
package bfstore.order.events.v1;
import "google/protobuf/timestamp.proto";
message OrderConfirmed {
EventMetadata metadata = 1;
string order_id = 2;
string customer_id = 3;
Money total = 4;
repeated ConfirmedOrderLine lines = 5;
uint64 order_version = 6;
}
message EventMetadata {
string event_id = 1;
string correlation_id = 2;
string causation_id = 3;
string source = 4;
google.protobuf.Timestamp occurred_at = 5;
google.protobuf.Timestamp published_at = 6;
}
message ConfirmedOrderLine {
string product_id = 1;
string product_name_at_purchase = 2;
uint32 quantity = 3;
Money unit_price_at_purchase = 4;
}
message Money {
int64 amount_minor = 1;
string currency = 2;
}
Before publishing it, I should still ask:
- Does every consumer need
customer_id? - Does notification need full order lines?
- Should analytics receive a separate projection?
- How long will the event be retained?
- Which scope makes
event_idunique? - May a customer request erasure?
- Is
order_versionthe ordering identity for every order event? - What should a consumer do when a version is missing?
- Can old events without
order_versionbe replayed? - Does success mean the event is in the outbox or already accepted by the broker?
The schema compiles the structure.
The architecture must compile the consequences.
A practical design checklist
| Concern | Question |
|---|---|
| Fact | What completed occurrence does the event assert? |
| Authority | Which component owns that assertion? |
| Consumers | Which current reactions are expected, and may new consumers appear later? |
| Necessity | Would a local call or RPC be clearer? |
| Payload | Is this notification-only or event-carried state? |
| Publication | Can state commit while the event is lost? |
| Identity | What is the uniqueness scope of the event ID? |
| Ordering | What is ordered, and within which aggregate or partition? |
| Gaps | What happens when a version is missing? |
| Delivery | Can the event arrive more than once? |
| Idempotency | How does each consumer protect its local effect? |
| Retention | How long must consumers be able to catch up? |
| Replay | Can derived state be rebuilt without repeating external side effects? |
| Schema | Can old and new producers and consumers coexist? |
| Failure | Who owns failed events and reconciliation? |
| Backlog | How much lag and event age are acceptable? |
| Security | Who may publish, consume and inspect the event? |
| Privacy | Does the payload expose more than consumers need? |
| Evidence | Can one event be followed through every important consumer? |
If these answers are absent, the system is not yet event-driven architecture.
It is messaging infrastructure awaiting a plot.
The mental model I am keeping
My original model was:
EVENT-DRIVEN ARCHITECTURE
│
▼
put Kafka between services
The stronger model is:
AUTHORITATIVE CHANGE
│
▼
PRODUCER COMMITS STATE
│
▼
EVENT ASSERTION RECORDED
│
▼
EVENT CHANNEL
│
┌───────────────┼───────────────┐
│ │ │
▼ ▼ ▼
CONSUMER A CONSUMER B CONSUMER C
│ │ │
▼ ▼ ▼
local reaction local reaction local reaction
Around that flow sit several promises:
- the producer owns the assertion;
- the event follows committed state;
- incorrect assertions are corrected with new events;
- publication can recover from failure;
- delivery may be repeated;
- consumer effects are duplicate-safe;
- ordering has a defined scope;
- version gaps have a recovery policy;
- schemas evolve compatibly;
- lag is measured;
- failed events have owners;
- replay is deliberate;
- derived data remains derived.
The benefits are real:
- temporal decoupling;
- independent consumers;
- fan-out;
- buffering;
- replay;
- scalable processing;
- removal of optional work from synchronous request paths.
The costs are equally real:
- eventual consistency;
- duplicate delivery;
- out-of-order processing;
- schema evolution;
- broker operations;
- consumer lag;
- distributed debugging;
- failed-event recovery;
- less visible workflow control.
So event-driven architecture is not:
Send everything through a broker and hope the services become independent.
It is:
Let owners publish durable assertions about completed changes, then give consumers the contracts and operational machinery required to react independently without corrupting their own state.
The broker carries the event.
The event carries the assertion.
The architecture carries the responsibility for everything that happens after the assertion leaves home.
References and further reading
Event-driven architecture
Microsoft Azure Architecture Center: Event-driven architecture style
Describes event producers, consumers and channels, together with publish-subscribe and event-stream models, scalability, decoupling, eventual consistency and operational challenges.
Microsoft Azure Architecture Center: Asynchronous messaging options
Distinguishes commands from events and discusses queues, publish-subscribe messaging, temporal decoupling and competing consumers.
Different meanings of event-driven
Martin Fowler: What do you mean by “Event-Driven”?
Separates event notification, event-carried state transfer, event sourcing and CQRS, which are frequently blended under one architectural label.
Kafka event streams
Apache Kafka documentation
Documents topics, partitions, records, consumer groups, offsets, retention, ordering within a partition and event-stream processing.
RabbitMQ delivery
RabbitMQ tutorial: Publish/Subscribe
Introduces publishing one message to multiple independent consumers.
RabbitMQ tutorial: Work Queues
Demonstrates competing consumers, message acknowledgements, redelivery and fair dispatch.
RabbitMQ tutorial: Publisher Confirms
Explains how a publisher can learn whether messages reached the broker.
Common event metadata
CloudEvents specification
Defines common event attributes including event identity, source, type, subject, time and content type.
Reliable publication
Chris Richardson: Transactional Outbox
Describes storing an outbound message in the same local database transaction as the business-state change, then publishing it through a separate relay.
Protobuf event schemas
Protocol Buffers best practices
Provides guidance on stable field numbers, reserved identifiers, compatible evolution and long-lived schema design.