Log

Keeping messages in memory allows both message puts and gets to be fast.

• Memory resources are limited, restricting the number of messages that can be stored.
• System performance degrades under these conditions:
  • Consumers become slow
  • Messages grow too large
  • Producers send significantly more messages than usual

In such cases, either messages must be discarded or stored externally to preserve data.

If messages are stored only in memory, a broker crash results in complete data loss.

Storing messages on disk helps mitigate both memory limitations and data loss risks.

• RabbitMQ and ActiveMQ: store messages both in memory and on disk.
• Kafka: stores all messages exclusively on disk, with no in-memory storage.

Kafka achieves high throughput and low latency despite not using memory.

These techniques enable high performance using disk storage.

They are not limited to message queues but apply to databases, object storages, and file systems.

Messages are stored using a queue data structure.

• A queue is essentially an array.
• New messages are added to the end.
• Messages are read from the beginning.

This creates a sequential sequence of messages.

• Instead of an in-memory array, use a file on disk.
• Messages are appended to the end of the file.
• Each message has a unique identifier.

• The identifier can be an auto-generated Unix string.
• For simplicity, think of it as a sequential number.

All messages are stored in chronological order.

• The sequence of messages is append-only and totally ordered.
• This structure is known as a LOPOC.
• Also referred to as a Datalog or Journal.

• Application logs are human-readable text files.
• Datalogs store messages as raw bytes for program consumption.

• Adding messages to a log file is simple.
• Messages are appended to the end of the file.

• Deleting a message from a large file is inefficient.
• Scanning and removing a single message takes significant time.

• Split the log file into chunks, called segments.
• Use predefined segment sizes (e.g., 1GB).

• Messages are written to the active segment.
• When the active segment reaches its size limit, it is rolled over to a new segment.
• Once all messages in a segment are processed, the segment is deleted.

• Messages are deleted in chunks, not individually.
• Segments are removed en masse when no longer needed.

This approach avoids costly individual deletions and enables efficient, scalable log management.

To retrieve a specific message efficiently, we need to locate it quickly in the log.

A naive approach involves iterating through log segments and messages within each segment until the target is found. This method is slow and inefficient.

A better approach requires knowing the byte position of each message's start within its log segment.

With the message's byte position known, we skip segment iteration entirely.

We simply identify the correct segment and seek directly to the message's position in the file to read its contents.

Thus, efficient message retrieval depends on maintaining a mapping from message identifier to message position in the log segment.

Every time a new message is appended to the active segment, a new record is added to this mapping.

Two key questions remain:

• Where to store this mapping?
• How to store this mapping?

Storing the mapping in memory is straightforward and enables fast access.

The storage structure of the mapping, however, presents multiple design options.

These options will be discussed in the next video.