Message delivery guarantees

In messaging systems, producers and brokers rely on acknowledgments to ensure message persistence and successful delivery.

When a broker pushes a message to a consumer, it waits for an acknowledgment.

This acknowledgment signals the broker to delete the message.

If the acknowledgment does not arrive or delivery fails, the broker may retry immediately or requeue the message for later delivery.

After multiple retries, the broker may apply a fallback: sending the message to a dead letter queue.

A dead letter queue isolates messages that cannot be processed correctly.

Such messages are often called poison pills.

In a pull-based system, the consumer initiates a request to fetch a message.

If the request fails, the consumer retries.

After processing, the consumer sends a confirmation to the broker.

This confirmation is an acknowledgment.

The broker uses this to delete the message.

Deleting a message is not the only option—this alternative will be discussed in the next video.

For now, assume deletion occurs upon successful acknowledgment.

When multiple competing consumers exist, each message must be delivered to only one consumer.

Competing consumers improve processing speed and system reliability through redundancy.

In push-based systems, the broker manages message deletion.

It selects one consumer at random from the group and sends the message.

If delivery fails, the broker attempts to push the message to the same consumer again.

After repeated failures, the broker fails over to a different consumer in the group.

In pull-based systems, all consumers poll for messages.

If one consumer is processing a message, others must not receive it.

To prevent duplication, the broker must mark the message as unavailable.

This state is called visibility timeout.

The broker sets a visibility timeout period.

During this time, no other consumer can access the message.

This period is expected to be sufficient for one consumer to process and delete the message.

AWS SQS implements this visibility timeout mechanism.

It ensures message visibility is maintained during processing.

Now, we introduce message delivery guarantees.

There are three types: at most once, at least once, and exactly once.

At most once means a message is delivered no more than once.

If delivery fails, the system does not retry.

This means the message might be lost.

This guarantee is often described as best-effort semantics.

At least once means the system retries until delivery succeeds.

The message is never lost, but may be processed multiple times.

Exactly once means every message is delivered and processed exactly once.

This is the ideal guarantee.

We always want exactly once delivery.

It ensures no duplication and no loss.

In an ideal system with no failures, each message is processed exactly once. But failures are inevitable.

The way failures are handled defines the delivery guarantee of the system.

A messaging system offers at most one delivery guarantee if the broker does not wait for acknowledgment when sending a message to a consumer.

As soon as message bytes are written to the TCP socket, the broker considers delivery complete.

It then removes the message from the queue.

During this process, the message may be lost in transit to the consumer.

The consumer might also fail during processing, resulting in message loss.

This guarantee trades higher throughput for reduced safety.

A system offers at least one delivery guarantee if the broker waits for acknowledgment.

Until it receives an acknowledgment, the broker keeps the message in the queue.

In a push-based system, unacknowledged messages are requeued and re-delivered.

In a pull-based system, unacknowledged messages become available when visibility timeouts expire.

In both cases, the consumer may receive the same message multiple times.

The consumer must send an acknowledgment only after successfully processing the message.

If acknowledgment is sent before processing, the broker removes the message from the queue.

If message processing fails, there is no mechanism to retry processing.

This behavior results in at most one delivery semantics, not at least one.

Exactly one delivery is achievable.

Examples include Kafka and SQL, which support such guarantees.

This can be achieved through idempotency or distributed transactions.

For idempotency:

• The broker must be idempotent to detect and eliminate duplicate messages during publication.

• The consumer must be idempotent to avoid processing the same message more than once.

Systems with exactly one delivery guarantee typically offer lower throughput.

They also introduce complex usage nuances.

Most messaging systems default to at least one delivery guarantee.

This configuration prevents message loss, maintains high throughput, and is easier to implement.

The topic of exactly one processing will be explored in greater detail on my YouTube channel.