Data encoding formats

A messaging system begins with a simple architecture: a single server, known as a broker. This broker receives messages from producers, stores them locally, and forwards them to consumers.

Every message has two components: metadata and a payload. Metadata includes attributes like content type, timestamp, and payload size. These are stored as key-value pairs, where keys are strings and values can be strings or byte arrays.

The payload is a sequence of bytes. From the messaging system’s perspective, it contains no structure or meaning. The system does not inspect or modify the payload.

Consumers must interpret the payload to extract meaningful data. To do this, both producers and consumers must agree on a payload encoding format.

Applications store data as objects in memory. When sending data over a network, these objects must be converted into a byte sequence. This process is called serialization or encoding.

The reverse process—reconstructing objects from bytes—is called deserialization or decoding.

Encoding formats fall into two broad categories: textual and binary.

Textual formats include JSON, XML, and CSV. Binary formats include Apache Thrift, Protocol Buffers, and Apache Avro.

Textual formats are human-readable and widely supported. Modern browsers natively support JSON. This makes debugging and testing easier.

JSON is a common default choice for web applications due to its simplicity and broad language support.

However, for high-scale systems with high throughput, binary formats offer performance advantages.

Textual formats are verbose. This results in larger message sizes and slower serialization and deserialization.

Binary formats use optimizations to reduce message size. For example, small integers are encoded in a single byte rather than multiple bytes.

The largest space savings comes from using a schema for the payload.

In JSON, field names consume significant space. In XML, tag names take up bytes. What if field names could be eliminated?

A schema defines field names and their data types. When a schema is defined, field names are no longer included in the payload.

This reduces payload size and improves efficiency.

Both producers and consumers must share the schema. Producers use it to serialize data; consumers use it to deserialize payloads.

Schema sharing introduces a small complexity but is manageable.

Several methods exist for schema sharing.

One method is to embed the schema in code. First, define a schema. Then, use a code generation tool to produce serialization and deserialization code for multiple programming languages.

This ensures both sides of the system use the same structure and types.

The schema becomes the contract between producer and consumer.

Without shared schema, messages may be misinterpreted or fail to parse.

Schema consistency is essential for reliable message processing.

The schema enables compact payloads and efficient processing.

It is the foundation of efficient binary encoding in messaging systems.

This step establishes the data layer upon which scalability and reliability are built.

Producers and consumers use generated code for messaging, coding, and decoding.

Thrift and Protocol Buffers are based on this principle.

Alternative schema sharing: store schema in a shared storage system called a schema registry.

Both producers and consumers can access the schema directly from this registry.

A database can serve as a schema registry.

When sending a message, the producer includes a schema identifier.

The consumer retrieves the schema from the registry using that identifier.

The consumer should cache the schema locally to reduce remote registry access costs.

This approach is typically used to manage every schema version.

Another method: embed the schema within the message payload as metadata.

This approach increases message size, raising transmission latency and storage costs.

It avoids the need to build and maintain a schema registry—a major advantage.

It enables easy reprocessing of historical messages.

For example, machine learning model training on past data.

Choose this method if message size is not a constraint.

Schemas may and will change over time.

New fields can be added or existing fields removed.

Thrift, Protocol Buffers, and Avro all support schema evolution.

This means generated code continues to work with both old and new schema versions.

This property is called compatibility.

Backward compatibility: new code can read old message versions.

Or: consumers using new schema can read messages produced with old schema.

Forward compatibility: old code can read new message versions.

Or: messages produced with new schema can be read by consumers using old schema.

Backward compatibility is generally easier to achieve.

Forward compatibility is more difficult to implement.

When deploying a new schema version, producers and consumers cannot be updated simultaneously.

For some period, messages with different schema versions coexist.

All messages—old and new—must be successfully processed.

Failure to handle this leads to data loss or system errors.

Compatibility ensures system resilience during schema transitions.