Asynchronous messaging patterns

Asynchronous messaging patterns enable systems to decouple components, improve scalability, and ensure reliable processing of events.

Two primary models exist: Message Queue and Publish Subscribe.

In Message Queue, only one consumer receives each message.

In Publish Subscribe, the same message is delivered to all subscribers.

A video sharing platform example illustrates both models.

A user uploads a video to YouTube-like platform.

Before playback, the video must be transcoded into formats for smartphones, tablets, and others.

The architecture begins with user upload.

The uploader server handles the request.

It uploads the original video to object storage, such as AWS S3.

When persisted, the uploader creates a message with file location and sends it to a messaging system.

The video transcoder service receives the message.

It downloads the video from object storage and transcodes it.

Upon completion, it uploads the transcoded videos and creates a metadata message.

This message is sent to the messaging system.

The system broadcasts it to multiple receivers.

These include:

• A database to process metadata
• A cache to serve metadata quickly
• A notification service to alert users
• A content delivery network to serve video content

The first messaging system acts as a queue.

Each message is processed only once.

The second system uses publish-subscribe.

Each message is processed by multiple systems.

The diagram represents messaging systems as separate components.

Physically, this may be a single system.

Many messaging systems support both queue and publish-subscribe models.

This architecture includes several widely used patterns.

The video transcoder service has multiple instances.

Each is a queue consumer.

Instances constantly try to get messages from the queue.

The queue guarantees each message is received by one instance.

This is the competing consumers pattern.

Multiple independent consumers compete for messages.

This pattern improves scalability, availability, and performance.

Scalability improves by adding more consumers.

The queue balances load across all instances.

Availability improves due to redundancy.

If an instance fails, others process the messages.

Performance improves because more messages can be processed in parallel.

This pattern works well for independent, asynchronous tasks.

Such queues are often called task queues.

Request-response messaging allows confirmation of message processing.

For every message sent, the uploader wants a response from the transcoder.

To confirm successful processing.

If processing fails, the uploader may resend the message or notify the user.

A callback queue is introduced.

Transcoder instances post response messages to this queue.

Each message sent includes:

• A callback queue name
• A unique request identifier

One transcoder instance processes the message.

It sends a result message to the callback queue.

The result includes a correlation identifier.

The uploader matches the response to the original request.

Priority queue pattern prioritizes messages.

Larger videos get higher priority.

Some systems support this natively.

Internally, messages are reshuffled so higher-priority messages are processed first.

If not natively supported, separate queues per priority are created.

Each priority has its own consumer pool.

Higher-priority pools have more consumers.

These consumers may run on faster, more expensive hardware.

The acclaim check pattern handles large messages.

Examples include large images, videos, audio files, or logs.

Sending large files directly to messaging systems is discouraged.

Most systems have message size limits.

Even if limits allow, large messages cause memory and performance issues.

The acclaim check pattern stores the full payload in shared storage.

The message contains only a reference to the stored payload.

The consumer retrieves the payload using the reference.

This is how video payloads are handled.

Videos are stored in object storage.

Only metadata and a file reference are sent to the transcoder.

An alternative is to split large files into chunks on the producer side.

Chunks are merged on the consumer side.

---