Large-scale push architectures

A persistent connection between server and client is cheap from the client's side.

But for a server, thousands of concurrent connections can strain resources.

This challenge is known as the C-10K problem.

Coined in 1999, it targets optimizing a single web server to handle 10,000 concurrent connections.

Modern servers now handle millions of concurrent connections.

A related problem, C-10M, aims to support 10 million concurrent connections on one machine—also solved.

Handling millions of concurrent connections differs from handling millions of concurrent requests.

Connection handling is about efficient scheduling.

Request handling is about processing speed.

In this section, we examine two real-world architectures that scale push messages to millions of clients.

MailRoo: From Polling to WebSockets

MailRoo is a large email service in Russia.

Originally, it used short polling.

At peak load, 50,000 HTTP requests per second were made.

60% of these returned empty results—no new emails.

This wasted server resources.

To reduce load and improve delivery speed, MailRoo switched to websockets.

In the polling model:

• Browser makes HTTP calls every few seconds.

• Server queries storage for new emails.

• If no new emails, response is empty.

In the push model:

• Browser establishes a WebSocket connection.

• When storage detects a new email, it sends a message to a messaging queue.

• Web servers subscribe to the queue.

• Upon message arrival, they send it directly to the browser.

This change enabled 3 million online connections per web server.

Netflix: Push Messaging via Zoo

Netflix uses a push notification service.

The architecture begins with Zoo, an API Gateway.

Zoo serves as the front door for all client requests.

It performs standard API Gateway functions:

• Authentication
• Authorization
• Logging
• Monitoring
• Request routing
• Load balancing
• Rate limiting
• Load shading

Zoo also supports push messaging.

It supports two protocols:

• WebSocket
• Server-Sent Events

Zoo operates as a cluster of multiple server machines.

When a user opens the Netflix app:

• A persistent connection is established.

• The connection links the user’s device to one of the Zoo servers.

After authentication, the user is registered in the push registry.

The push registry is a data store.

Options include Redis, Cassandra, DynamoDB.

It stores which user is connected to which Zoo server.

Why is this needed?

To deliver messages, the system must locate the correct server.

The push registry enables this lookup.

This allows messages to be routed directly to the connected client.

The registry is essential for efficient message delivery.

Without it, message routing would require scanning all servers.

This would be slow and inefficient at scale.

The architecture ensures low-latency, high-throughput message delivery.

It supports millions of concurrent clients through persistent connections and efficient routing.

The design separates message delivery from request handling.

This enables scalability without sacrificing performance.

Each component plays a distinct role in the overall system.

The system is built for real-time, high-volume message delivery.

It demonstrates how push architectures solve scalability challenges at scale.

Message producers—backend services—send messages to Kafka, a messaging system.

A message processor reads messages from Kafka and extracts a user ID.

The processor queries the push registry to locate the server holding the user’s connection.

If the server is found, the user is online and connected to the Zoo server.

The message processor connects directly to that server and pushes the message to the user.

If the user ID is not found in the push registry, the user is not connected or offline.

The message processor drops the message in this case.

The architecture is horizontally scalable.

A single Zoo server can handle tens of thousands of connections.

A cluster of hundreds of such servers can manage millions of connections in total.

The push registry uses a highly scalable, low-latency data store.

Writes occur only when a user first connects to a Zoo server.

Reads occur every time a message is sent.

This results in significantly more reads than writes.

Kafka is a highly scalable messaging system—no new elements here.

Message processors can run in parallel.

Photoscaling can be applied to message processors.

More instances can be launched as the message queue grows to increase throughput.

Instances can be stopped when load decreases to save cost.

Netflix runs all systems on public cloud.

Cost savings is a key design consideration.

Handling millions of long-lived connections at scale is challenging.

Two key problems arise:

• Blocking I/O creates a thread per connection.

• This model scales poorly under long-lived connections.

Non-blocking I/O is preferred.

It uses a single thread to manage multiple connections.

Zoo is built on Netty, a high-performance non-blocking I/O framework.

During server restarts—such as software upgrades—two approaches exist:

• Migrate connections to a new server without requiring client reconnection.

• Force clients to reconnect to a different server.

The "sounder-in-hearth" problem must be avoided.

When many clients attempt to reconnect simultaneously, it causes traffic spikes.

Clients should be slowly disconnected, not all at once.

Even if a single server can handle millions of connections,

it is not advisable in production.

Using multiple smaller servers is better than one large server.

When a server crashes, clients previously connected to it attempt reconnection.

It is easier to manage thousands of reconnections than millions.

Many distributed systems use a load balancer between clients and servers.

It distributes connections evenly across servers.

A WebSocket connection remains open after initial setup.

No requests are pending, yet the connection stays active.

Older load balancer versions lack native WebSocket support.

Connections are terminated after a period of inactivity.

Modern load balancers support WebSocket proxy functionality natively.

This eliminates the issue of connection drops.

An alternative approach is to operate the load balancer at layer 4 as a TCP load balancer.

Rather than functioning as an HTTP load balancer at layer 7.

The distinction between layer 4 and layer 7 load balancing is critical.

Layer 4 operates at the transport layer, handling raw TCP connections.

Layer 7 operates at the application layer, interpreting HTTP protocols.

Running a load balancer at layer 4 avoids protocol-level parsing.

This reduces the risk of connection drops due to inactivity.

Layer -7 load balancing may drop connections when no data is sent for a while.

Layer-4 TCP load balancing maintains persistent connections without requiring application-level signaling.

This makes it more reliable for real-time push architectures.

The choice between layer 4 and layer 7 depends on the specific use case.

Layer 4 is simpler and more resilient for long-lived connections.

Layer 7 offers richer features but introduces more drop risks.

We will explore load balancing in greater depth later in the course.

For now, understand that layer 4 TCP load balancing provides a more stable connection model.