Consumer groups finally clicked

I had two apparently incompatible ideas about Kafka consumers.

The first was:

Every subscriber to a topic receives the events.

The second was:

Adding more consumers allows the work to be processed in parallel.

If every consumer receives every event, adding consumers duplicates the work. If the events are divided between consumers, then every consumer does not receive every event.

Both ideas appear in descriptions of Kafka. The missing concept was the consumer group.

TOPIC
  │
  ▼
CONSUMER GROUP
  │
  ├── consumer A
  ├── consumer B
  └── consumer C

The group is one logical subscriber. Its individual consumers are members cooperating to perform that subscriber’s work.

In a traditional Kafka consumer group, Kafka assigns each subscribed partition to one member at a time.

PARTITION 0 ──► consumer A
PARTITION 1 ──► consumer B
PARTITION 2 ──► consumer C

Together, the members process the group’s subscribed partitions from that group’s valid starting positions, subject to retention. Records may still be delivered again after failure or recovery.

That gives me the model that finally clicked:

A traditional consumer group is one logical subscriber. Its members divide the assigned partitions, while other groups consume the same topic independently for other purposes.

                         TOPIC
                           │
             ┌─────────────┴─────────────┐
             │                           │
             ▼                           ▼
       SEARCH GROUP                ANALYTICS GROUP
       │          │                │             │
       ▼          ▼                ▼             ▼
   search A   search B       analytics A   analytics B

Between groups, consumption is independent. Within one group, members divide that group’s partition work.

A consumer group is one logical subscriber

One group represents one application purpose

Suppose bfstore publishes:

ProductPublished

Search needs the event to update its index.

Analytics needs it to record publication activity.

Recommendations may need it to add the product to a candidate set.

These are three different reactions.

ProductPublished
      │
      ├──► search
      ├──► analytics
      └──► recommendation

Each reaction can have its own consumer group:

• search-indexer
• catalog-analytics
• product-recommendations

Kafka makes the event available independently to each group.

ProductPublished
      │
      ├──► search-indexer group
      ├──► catalog-analytics group
      └──► product-recommendations group

The group ID identifies the logical subscriber.

Consumers sharing:

group.id = search-indexer

cooperate on the search-indexing workload.

Consumers using:

group.id = catalog-analytics

form a separate subscriber.

The search group processing an event does not remove it from the analytics group.

Each group tracks its own progress.

Separate groups consume independently

This was the other half of the click.

Suppose a topic has:

partition 0
partition 1
partition 2

Search uses one consumer group:

group.id = search-indexer

Analytics uses another:

group.id = catalog-analytics

Kafka assigns all three partitions within each group.

SEARCH GROUP
│
├── partition 0 ──► search A
├── partition 1 ──► search B
└── partition 2 ──► search B


ANALYTICS GROUP
│
├── partition 0 ──► analytics A
├── partition 1 ──► analytics A
└── partition 2 ──► analytics B

Every record is available to both groups.

Within the search group, only one member processes a given partition.

Within the analytics group, only one analytics member processes that partition.

So the two rules are:

This is how Kafka supports both publish-subscribe and competing consumers.

Different groups represent different subscriptions.

Members of one group compete or cooperate to process that subscription.

Members share work within one group

Suppose a topic contains four partitions:

catalog-events
│
├── partition 0
├── partition 1
├── partition 2
└── partition 3

The search group initially has one consumer.

SEARCH GROUP

consumer A

Kafka can assign all four partitions to that member:

consumer A
│
├── partition 0
├── partition 1
├── partition 2
└── partition 3

Consumer A processes the complete search workload.

If a second consumer joins:

SEARCH GROUP

consumer A
consumer B

Kafka can redistribute the partitions:

consumer A
│
├── partition 0
└── partition 1


consumer B
│
├── partition 2
└── partition 3

The members now work in parallel.

They do not both receive every record.

Each processes the records from its assigned partitions.

Together, they still represent one logical search-indexing consumer.

Partitions are the unit of assignment

Kafka assigns partitions rather than alternating records

I had imagined Kafka handing records to consumers one by one:

record 1 ──► consumer A

record 2 ──► consumer B

record 3 ──► consumer A

The actual assignment boundary is the partition.

partition 0 ──► consumer A

partition 1 ──► consumer B

Consumer A continues reading records from partition 0.

Consumer B continues reading records from partition 1.

PARTITION 0                 PARTITION 1

offset 10 ──► A             offset 30 ──► B
offset 11 ──► A             offset 31 ──► B
offset 12 ──► A             offset 32 ──► B

This preserves each partition’s ordering relationship.

One consumer member owns a partition within the group at a time.

The group gains parallelism by processing several partitions concurrently.

Kafka does not ordinarily split one partition’s records between two active members of the same group.

One member owning a partition does not automatically make the application process that partition sequentially. A local worker pool can still begin or complete later offsets before earlier ones. Traditional assignment protects the read boundary; application concurrency still determines effect order.

Current note: This article describes traditional consumer groups. Newer Kafka share groups allow several members to cooperate on records from the same partitions for queue-like work distribution. That finer-grained sharing trades away traditional partition-order processing.

Partition count limits active parallelism

Suppose the topic has three partitions:

partition 0
partition 1
partition 2

The group has one member.

consumer A:
    partitions 0, 1 and 2

A second member joins:

consumer A:
    partitions 0 and 1

consumer B:
    partition 2

A third joins:

consumer A:
    partition 0

consumer B:
    partition 1

consumer C:
    partition 2

A fourth member joins:

consumer A:
    partition 0

consumer B:
    partition 1

consumer C:
    partition 2

consumer D:
    no assignment

There are only three partitions to assign.

The fourth consumer cannot create a fourth unit of partition-level work.

This gives a simple upper bound:

• maximum active consumers in one group
• for one topic
• ≈
• number of partitions

A consumer may subscribe to several topics, so real assignments can involve more partitions.

The central point remains:

Adding consumers beyond the available partitions does not increase ordinary consumer-group parallelism.

The extra members wait without assigned work. They add no steady-state partition throughput, although warm spare members may shorten recovery when an assigned member leaves.

For one topic, the partition count sets the upper limit on active traditional members. Across several subscribed topics, the total subscribed partitions determine potential parallelism, while the assignment strategy determines how evenly those partitions are distributed. A range-style assignment can leave work uneven even when the total count looks sufficient.

Fewer members than partitions is normal

A group does not need one consumer per partition.

Suppose there are twelve partitions and three consumers.

Kafka can assign four partitions to each member.

consumer A:
    partitions 0, 1, 2, 3

consumer B:
    partitions 4, 5, 6, 7

consumer C:
    partitions 8, 9, 10, 11

Each consumer can interleave reads from its assigned partitions.

It must still preserve the required processing order within each partition.

partition 0:
    A1 → A2 → A3

partition 1:
    B1 → B2 → B3

The consumer could process:

• A1
• B1
• A2
• B2
• A3
• B3

That does not violate partition ordering.

The relative order between partition 0 and partition 1 was never guaranteed.

The group ID defines application identity

Stable application identity

A group ID should identify the consuming application or processing purpose.

Examples might include:

• bfstore-search-indexer
• bfstore-order-projection
• bfstore-notification-dispatch
• bfstore-catalog-analytics

It should not normally identify one short-lived process instance:

search-consumer-pod-7f889

If every process starts with a different group ID, each process becomes a separate logical subscriber.

consumer A:
    group search-1

consumer B:
    group search-2

consumer C:
    group search-3

All three may receive the complete topic independently.

Instead of sharing search-indexing work, they may index every event three times.

To share the workload, the instances use the same stable group ID:

consumer A:
    group search-indexer

consumer B:
    group search-indexer

consumer C:
    group search-indexer

The process identity and group identity are different.

The members can come and go.

The group’s processing purpose remains.

A group does not broadcast within itself

Suppose three notification consumers share:

group.id = notification-dispatch

An OrderConfirmed event is assigned through one partition to one group member.

OrderConfirmed
      │
      ▼
notification consumer B

Consumers A and C do not also receive it as members of that same group.

If every notification instance needs the event independently, they should not be members sharing one logical subscription.

They would need separate group IDs or another broadcasting arrangement.

Normally, notification instances are interchangeable workers.

Only one needs to create the delivery job.

The group represents one logical reaction: send one confirmation notification. Several interchangeable worker instances share that reaction workload.

The group expresses that relationship.

Unrelated applications must not share a group

Suppose search and analytics mistakenly use:

group.id = bfstore-consumers

Kafka treats them as members of one logical subscriber.

Partitions may be divided between unlike applications.

partition 0 ──► search instance

partition 1 ──► analytics instance

partition 2 ──► search instance

Search sees only some product events.

Analytics sees another subset.

Both applications become incomplete.

The group ID should therefore identify one coherent processing purpose.

Group membership means:

These processes are interchangeable participants in one logical workload.

Processes performing different business reactions are not interchangeable.

Progress belongs to the group

Committed offsets record group progress

Kafka records consumer progress using offsets associated with a consumer group and topic-partition.

Conceptually:

group:
    search-indexer

topic:
    catalog-events

partition:
    2

processed through:
    144

committed offset:
    145

The analytics group may be at another position:

group:
    catalog-analytics

topic:
    catalog-events

partition:
    2

processed through:
    120

committed offset:
    121

Search has committed progress further through partition 2.

Analytics is behind.

PARTITION 2

offset 121        offset 145        latest 150
    │                 │                │
    ▼                 ▼                ▼
analytics         search           newest record

The topic contains one partition log.

Each group maintains its own position within it.

This is why one slow consumer group does not force another group to move backwards.

Search and analytics have independent progress.

Kafka’s convention is important: a committed offset identifies the next record to consume. If the application has durably processed offset 144, it normally commits 145. Treating the committed value as the last processed record creates an off-by-one recovery error.

Progress survives member replacement

Suppose search consumer A owns partition 2 and has processed records and committed the next position from which the group should resume.

The process crashes.

consumer A
    │
    └── crash

Another group member can receive the partition.

partition 2
    │
    ▼
consumer B

Consumer B continues from the group’s committed progress.

The offset belongs to:

search-indexer group

not permanently to consumer A.

This allows members to be replaceable.

consumer process disappears
        │
        ▼
partition reassigned
        │
        ▼
group continues

The application still needs to coordinate offset commits with its business effects.

Kafka can restore the group’s reading position.

It cannot know whether a consumer’s database transaction or external call completed.

Progress survives replacement of individual members, but it is not immortal. Committed group offsets can expire after a group remains inactive for the configured retention period. Deleting the group or topic also removes the corresponding relationship. A returning application should therefore define what happens when no valid committed offset remains.

Changing the group ID creates a new reader

Suppose search normally uses:

group.id = search-indexer

A deployment accidentally changes it to:

group.id = search-indexer-v2

Kafka sees a new group.

The new group has no existing committed offsets under the old identity.

Where it begins depends on its offset-reset configuration and which records remain available.

It may start:

from the earliest retained records

or:

from records arriving after the new group begins

This can cause:

• large replay
• missed historical records
• duplicate indexing
• unexpected broker and database load

Changing a group ID is not a cosmetic rename.

It creates a new logical subscriber.

If the intention is to rebuild a projection, that may be useful.

If the change is accidental, the consumer can quietly begin a very different relationship with the topic.

A versioned identity such as search-indexer-v2 can be intentional when a new projection must run alongside the old one. The migration should still document:

• why the new group exists;
• where it begins;
• whether it replays retained history;
• how duplicates across old and new projections are handled;
• when the earlier group can be retired.

New groups can replay retained history

The same property can be used deliberately.

Suppose bfstore creates a new recommendation service after the event stream has existed for six months.

It joins with:

group.id = product-recommendations

If the required events are still retained and the group begins from the earliest available offsets, it can process historical events.

EVENT LOG
│
├── ProductCreated
├── ProductPublished
├── ProductViewed
├── OrderConfirmed
└── ProductDiscontinued
        │
        ▼
NEW RECOMMENDATION GROUP

Existing search and analytics groups keep their own positions.

The new group reads independently.

This is one of the useful properties of a retained event log:

A new application can create its own view of existing history without resetting existing consumers.

Whether that history is complete depends on retention, schema compatibility and the events originally published.

The starting position is governed by existing committed offsets. When no valid offset exists, auto.offset.reset determines whether the group begins at the earliest retained records, the latest position, or fails under a stricter policy. No setting can restore records that retention has already removed.

Membership changes move partition ownership

Rebalances redistribute assignments

When a consumer:

• joins
• leaves
• fails
• stops polling for too long

the group may need to redistribute partitions.

This process is commonly called a rebalance.

Suppose the group begins as:

consumer A:
    partitions 0 and 1

consumer B:
    partitions 2 and 3

Consumer C joins.

The group may become:

consumer A:
    partition 0

consumer B:
    partition 2

consumer C:
    partitions 1 and 3

Partition ownership has changed.

The application needs to stop work safely on revoked partitions and begin work correctly on newly assigned ones.

A rebalance is not only a Kafka administration event.

It can affect:

in-progress records

offset commits

duplicate processing

temporary consumption pauses

connection and cache state

The consumer lifecycle should expect partitions to move.

Current Kafka deployments may use either the classic rebalance protocol or the newer consumer rebalance protocol. The newer protocol is incremental, so a membership change does not necessarily revoke every partition from every member. Handlers should respond to the actual assigned and revoked partitions rather than assume that all ownership disappears at once.

max.poll.interval.ms bounds how long a group-managed application may go between calls to poll(). If processing prevents polling beyond that interval, Kafka treats the member as failed and begins reassignment. Slow work therefore needs a processing model that keeps ownership and progress visible.

The logical group survives process replacement

Suppose one consumer instance is replaced during deployment.

old consumer B
      │
      ▼
new consumer D

The group ID remains:

search-indexer

The logical subscriber has not changed.

Its membership has.

SEARCH-INDEXER GROUP
│
├── consumer A
├── consumer C
└── consumer D

This is similar to replacing one worker in a team.

The team’s job remains the same.

The work is redistributed.

That is why consumer applications should avoid keeping essential partition state only in one process’s memory.

When ownership moves, the new member needs to reconstruct whatever it requires from:

• the partition records
• committed offsets
• durable local state
• external state stores

Ephemeral process identity should not become the only map to durable progress.

When a partition is revoked, its old owner should stop starting new work and safely finish, cancel, or record any in-flight work before the new owner proceeds. Delegated worker models often need pause and resume, partition-specific progress tracking, and commits only after the delegated work has completed.

Static membership, configured with a stable group.instance.id, can reduce unnecessary assignment churn during short restarts or rolling deployments. It does not remove crash recovery or offset coordination, and each active instance ID must remain unique.

Deployments and shutdown are part of handover

Suppose a deployment replaces four consumer instances one by one.

Each departure and arrival may affect membership and partition assignment.

consumer leaves
      │
      ▼
partitions reassigned
      │
      ▼
new consumer joins
      │
      ▼
partitions reassigned again

Frequent rebalances can interrupt steady consumption and increase duplicate-processing windows.

Deployment strategy and group configuration can reduce unnecessary disruption.

The exact controls depend on the Kafka client and assignment strategy.

The architectural point is stable:

Scaling and deployment change partition ownership, so they are part of the consumer’s processing lifecycle.

A consumer is not only a loop reading records.

It is a temporary custodian of assigned partitions.

Static membership can reduce avoidable rebalances during brief, predictable restarts. Cooperative or incremental assignment can also reduce how much work moves during a rebalance. Neither feature removes the need to make in-flight work, offset commits, and process termination safe.

When a consumer shuts down deliberately, it should stop accepting new work and complete or safely abandon work for its assigned partitions.

A graceful sequence might be:

• 1. stop starting new record processing
• 1. finish eligible in-progress work
• 1. commit safe progress
• 1. leave the group
• 1. allow partitions to move

An abrupt termination may leave:

• uncommitted completed work
• partially applied effects
• longer failure detection
• more redelivery

The new member must still be able to recover correctly.

Graceful shutdown reduces avoidable uncertainty.

It cannot guarantee that every termination will be graceful.

Consumers still need crash-safe processing.

Recovery can repeat or lose application work

Reassignment can repeat completed effects

Suppose consumer A processes offset 50:

• 1. update search index
• 1. crash before committing progress

The partition is reassigned to consumer B.

Consumer B begins from the group’s last committed position and receives offset 50 again.

consumer A:
    index updated

consumer B:
    offset 50 redelivered

The group has preserved at-least-once progress.

The search update may be repeated.

For a projection update such as:

set product chair-42 to version 19

repetition may be harmless if the operation is idempotent.

For a side effect such as:

send customer email

repetition may be visible.

Consumer groups distribute and recover work.

They do not remove the need for duplicate-safe handlers.

Committing progress too early can lose work

Consider the opposite order:

• 1. commit offset 50
• 1. update search index

The consumer crashes after step one.

Kafka records that the group has advanced.

The search index was not updated.

After reassignment, the new consumer may continue from a later offset.

KAFKA PROGRESS

offset 50 handled


SEARCH STATE

offset 50 effect missing

The event can be skipped from the application’s perspective.

This produces the basic processing trade:

Many applications prefer duplicate-safe processing because repeating known work can be managed more reliably than silently losing it.

The exact coordination depends on where the consumer writes its state and whether Kafka transactions are part of the design.

Assignment is not business exactly-once

Kafka can ensure that one partition is assigned to one member of a group at a time during ordinary operation.

That does not mean one business effect occurs exactly once.

A consumer may:

• receive once and write twice because of a bug
• receive twice after recovery
• write successfully but fail to commit progress
• commit progress but fail to write
• call an external system whose response is lost

The useful boundaries remain separate:

Consumer groups coordinate the first and help track the fourth.

Application design controls the business effect.

There is one important bounded case. In a consume-transform-produce pipeline that stays inside Kafka, a Kafka transaction can atomically commit the output records and the consumed offsets. That transaction does not automatically include a relational database, search index, email provider, payment provider, or arbitrary remote call.

Scaling is constrained by the partition layout

More consumers do not repair a hot partition

Suppose a topic has six partitions.

Most traffic is balanced, but one product key produces half of all events.

partition 0:
    moderate

partition 1:
    moderate

partition 2:
    extremely busy

partition 3:
    moderate

partition 4:
    moderate

partition 5:
    moderate

The group has six consumers.

One member receives partition 2.

consumer C:
    partition 2

That consumer becomes overloaded.

Adding a seventh consumer does not divide partition 2.

The seventh remains idle because all partitions already have owners.

The bottleneck is the partitioning strategy or the hot key, not the number of group members.

Possible responses include:

• change the business key scope
• split unusually large work behind the consumer
• reduce producer concentration
• improve processing efficiency
• repartition into a new topic

Each option may weaken or change ordering guarantees.

Consumer scaling cannot outrun a partition boundary.

One slow record can delay its lane

Suppose one partition contains:

offset 100  ordinary event

offset 101  event requiring 10 minutes

offset 102  ordinary event

offset 103  ordinary event

If processing must remain sequential, offsets 102 and 103 wait behind the slow record.

100 ──► complete

101 ──► still processing

102 ──► waiting

103 ──► waiting

Other partitions can continue on other consumers.

The slow record affects its own ordered lane.

This is sometimes desirable because later records depend on it.

It can also produce localised lag.

The consumer needs a policy for unusually long work:

• perform it inside the consumer
• hand it off to a durable job system
• pause the partition
• fail and retry later
• move permanently bad work aside

The choice must preserve whatever ordering and completion semantics the stream requires.

If the consumer hands the record to another durable job system, the completion boundary changes. The Kafka consumer may advance only after the job has been durably accepted, not after the eventual business action finishes. The job system then owns retry, deduplication, ordering, and final completion.

Handing work off is therefore not a free way to keep polling. It moves the processing contract to another durable boundary.

Lag belongs to a group

Because each group has independent offsets, lag is measured per group and partition.

Suppose partition 2 has latest offset 500.

search-indexer offset:
    490

catalog-analytics offset:
    300

Then:

search lag:
    10 records

analytics lag:
    200 records

The topic itself is not simply:

behind

One group is nearly current.

Another is far behind.

Useful monitoring identifies:

group ID

topic

partition

current position

latest position

oldest unprocessed event age

The age often matters more than the count.

Two hundred events could represent:

two seconds of traffic

or:

two days of traffic

A consumer group is healthy only when its processing freshness satisfies the application’s purpose.

Suppose the analytics group members are:

• connected
• polling
• producing no errors

But incoming traffic exceeds processing capacity.

incoming:
    1,000 events per second

processing:
    800 events per second

Lag grows by:

200 events per second

The processes are alive.

The application is falling behind.

Consumer-group monitoring therefore needs more than instance uptime.

It should include:

lag trend

oldest-event age

processing rate

incoming rate

rebalance frequency

failed-record rate

partition skew

The group is an application-level unit.

Its health should be measured at that level.

Different groups can have different freshness

Suppose OrderConfirmed is consumed by:

• notification group
• shipping group
• analytics group

Notification may process the event immediately.

Shipping may be five minutes behind.

Analytics may intentionally process in hourly batches.

OrderConfirmed
      │
      ├── notification: current
      ├── shipping: five minutes behind
      └── analytics: one-hour batch

The producer does not wait for every group to reach the same offset.

Each application owns its own freshness and recovery policy.

This is valuable decoupling.

It also means there is no single answer to:

Has this event been processed?

The more precise questions are:

• Has notification processed it?
• Has shipping processed it?
• Has analytics processed it?

An event can be complete for one consumer group and pending for another.

Design the group as an application boundary

A bfstore example

Suppose bfstore publishes order events to:

bfstore.order.events.v1

The topic has six partitions.

Records use:

key = order_id

Events for one order remain in one partition.

order-5001
│
├── OrderCreated
├── OrderConfirmed
└── OrderDispatchRequested

Notification group

group.id = bfstore-notification-dispatch

Three notification consumers share the six partitions.

notification A:
    partitions 0 and 1

notification B:
    partitions 2 and 3

notification C:
    partitions 4 and 5

Each OrderConfirmed event should create one notification job.

The handler uses a unique business key such as:

order_id + notification_type

to prevent duplicate jobs after redelivery.

Shipping group

group.id = bfstore-shipping-planner

Two shipping consumers independently read all six partitions as another group.

shipping A:
    partitions 0, 1 and 2

shipping B:
    partitions 3, 4 and 5

Shipping sees the same order events as notification.

It maintains separate offsets and lag.

Analytics group

group.id = bfstore-order-analytics

One analytics consumer owns all six partitions.

It processes more slowly but remains within its accepted freshness window.

These are three logical subscribers.

Within each subscriber, its members divide the partitions.

Consumer-group checklist

A consumer group is not simply a name in configuration. It defines one application’s consumption identity, progress, scaling boundary, and recovery behaviour.

The mental model I am keeping

My old model was:

TOPIC
  │
  ▼
CONSUMERS

somehow everybody receives events
and somehow the work is shared

The stronger model is:

                             TOPIC
                               │
               ┌───────────────┴───────────────┐
               │                               │
               ▼                               ▼
        CONSUMER GROUP A                CONSUMER GROUP B
        one logical reader              another logical reader
               │                               │
       ┌───────┴───────┐               ┌───────┴───────┐
       │               │               │               │
       ▼               ▼               ▼               ▼
   member A1       member A2       member B1       member B2
       │               │               │               │
       └── share A's partitions        └── share B's partitions

Between groups:

each group receives the stream independently

Within a group:

members divide the assigned partitions

The partition remains the unit of assignment.

• one partition
• → one member of the group at a time

The number of partitions limits active consumer parallelism.

The group’s committed offsets preserve its progress across member replacement.

Rebalances move partitions when membership changes.

Duplicate-safe processing protects effects around crashes and reassignment.

This finally separates three identities that I had been blending together:

The topic contains the shared history.

The group records one application’s journey through it.

The member is merely the worker currently carrying part of the luggage.

References and further reading

Kafka consumer groups and assignment

Kafka Consumer API
Explains traditional consumer groups, group identities, partition assignment, independent groups and committed offsets.

Apache Kafka consumer configuration
Documents group identity, assignment, static membership, polling limits, offset-reset behaviour and consumer lifecycle settings.

Consumer configuration and progress

Apache Kafka consumer configuration reference
Documents group.id, group.instance.id, group.protocol, auto.offset.reset, max.poll.interval.ms, assignment strategies and session settings.

Apache Kafka broker configuration
Documents group-offset retention and related broker-side group behaviour.

Rebalancing and newer group models

Apache Kafka: Consumer rebalance protocol
Describes the newer incremental consumer-group rebalance protocol and its relationship with the classic protocol.

Kafka Share Consumer API
Describes queue-like record sharing among several consumers and the resulting difference from traditional partition-order consumption.

Transactions

Apache Kafka design: transactions
Explains the Kafka transaction boundary for atomically committing produced Kafka records and consumed offsets.