How to scale message consumption

We discuss system design concepts that enable processing more messages and doing so faster.

This means we are addressing scalability and performance.

Batching and compression, covered in prior videos, improve performance.

Yet, even with these techniques, a single consumer will eventually reach its processing limit.

When this limit is reached, two options emerge:

• Scale consumers vertically
• Scale them horizontally

Vertical scaling means using a more powerful consumer machine.

Horizontal scaling means deploying multiple consumers competing for messages.

You already understand these concepts.

Benefits of a Single Consumer

The primary advantage is preserved message processing order.

A single consumer receives a message, processes it, then receives the next message from the queue.

Messages are processed sequentially.

Drawbacks of a Single Consumer

Availability is a key concern.

A single consumer will eventually fail.

When it fails, messages remain unprocessed until a new consumer is activated.

Common failure scenarios include network partitions and consumer application restarts.

How Multiple Consumers Improve Availability

If one consumer becomes unavailable, others continue processing messages.

This improves system resilience.

Loss of Processing Order

With multiple competing consumers, message processing order is no longer guaranteed.

Consumers operate independently and do not coordinate.

For example, in a push-based system like RabbitMQ:

• Messages A, B, and C exist in a queue.
• Two consumers are present: one and two.
• The broker distributes messages in a round-robin fashion.

Message A goes to consumer one. Message B goes to consumer two. Message C goes to consumer one.

If message processing order matters, message A must be processed before message B.

However, consumer one may process message A slowly or fail.

Consumer two successfully processes message B.

Thus, message B is processed before message A.

This can be critical in systems like booking platforms.

Message A represents a booking creation event. Message B represents a booking cancellation event.

If message B is processed first, and no booking exists, it is cancelled.

Then, when message A is processed, a booking is created.

The result: a booking is created and never cancelled.

This violates business logic.

Double Processing Risk

In pull-based systems like SQS:

• Messages A, B, and C exist in a queue.
• Two consumers pull messages.

Consumer one pulls message A.

The broker marks message A as invisible to all consumers.

Consumer two pulls message B.

Message B becomes invisible.

Assume both consumers process messages successfully and in correct order.

Message A is processed first.

But when consumer one attempts to delete message A from the queue, the call fails.

Message A remains in the queue.

After the visibility timeout expires, message A is pulled again.

This time, consumer two retrieves it.

This results in message A being processed twice.

This is a double-processing issue.

Both pull and push-based systems face these problems.

If message A is processed by the first consumer, it is marked as seen and stored in memory.

The second consumer, unaware of this, sees message A as new and processes it again.

This results in duplicate processing of the same message.

Using the booking analogy: the first consumer creates a booking, the second cancels it, then creates it again.

Competing consumers increase the likelihood of processing the same message multiple times.

One solution is to use a shared deduplication cache.

This cache is not local to individual consumers.

All consumers access the same cache to check if a message has been processed.

This prevents redundant processing across consumers.

We will explore distributed caches in later course sections.

In systems like Kafka and Kinesis, consumers use offsets to track message positions.

Each consumer maintains its own offset.

Messages are not immediately deleted from the queue.

Every consumer processes every message in the queue.

No matter how many consumers are added, all process every message.

This contradicts the idea that Kafka is not scalable.

Kafka is highly scalable.

Multiple consumers on the same queue enable parallel message processing.

This increases system availability.

However, message order is lost across the entire queue.

Duplicate messages may appear due to parallel processing.

Offset-based systems do not support competing consumers on a single queue.

An elegant solution is data partitioning or sharding.

Instead of multiple consumers on one queue, use multiple queues.

Each queue has one consumer.

Data is split into partitions.

Each partition is processed by a dedicated consumer.

This enables parallel message processing.

The setup appears as one logical queue with multiple physical partitions.

Partitioning removes global message ordering.

Messages in each partition are processed independently.

Order is preserved within each partition.

This is sufficient for most systems.

Many systems do not require total event ordering.

They only need ordering within a specific group or partition.

In the booking system, all events for the same user must be processed in order.

Partitioning ensures all events for a single account go to the same partition.

This allows parallel processing of different accounts.

Each account is processed by only one consumer.

This preserves processing order for each user.

Partitioning is a foundational concept in distributed systems.

It enables scalability, availability, and ordered processing within groups.

It remains central to modern system design.