Push and pull technologies in real-life systems

This lesson explores real-world system design choices between pull and push technologies.

AWS SQS is a pull-based system supporting both short and long polling.

When designing applications, long polling is preferred.

Long polling reduces empty responses and lowers operational cost.

Kafka uses a custom TCP protocol, not HTTP.

Short and long polling are not natively applicable.

However, an HTTP-based proxy can be introduced in front of a Kafka cluster.

Clients communicate with the proxy over HTTP.

The proxy connects to Kafka over TCP.

Polling over TCP can mimic short and long polling behaviors.

• Clients can request messages immediately
• Or wait for new messages to accumulate

This behavior mirrors short and long polling.

Kafka offers rich configuration properties to control polling.

Kinesis operates over HTTP.

The Kinesis Client Library uses short polling.

It retrieves a batch of messages, processes them, then makes a new request.

If no messages are available, the client sleeps for one second before retrying.

RabbitMQ and Apache ActiveMQ support web sockets.

Web sockets enable bidirectional communication.

Web sockets are a natural choice.

Clients send messages to the server.

The server pushes new messages to clients.

Long polling and Server-Sent Events (SSE) are also viable.

Both deliver messages from server to client.

Message sending from client to server is straightforward.

Clients can use regular HTTP POST requests.

Early Facebook Messenger used long polling.

Long polling supports all browser versions.

It works when web sockets or SSE are unavailable.

A basic chat app can be built with SSE in under one hour.

News feeds (e.g., Facebook, Twitter, LinkedIn) can use either pull or push.

Clients can pull data periodically.

Or the server can push updates when new content is published.

Feeds typically contain dozens to hundreds of messages.

Examples include stock price dashboards.

Prices require frequent updates.

This is high-frequency messaging with small payloads.

Short and long polling are suboptimal.

HTTP headers dominate message size.

Header overhead exceeds message payload.

In such cases, use either WebSockets or Server-Sent Events.

These technologies minimize overhead and enable efficient real-time delivery.

A browser game can use WebSocket for real-time, bidirectional communication between clients and servers.

Many collaborative code editors—commonly used in coding interviews—require real-time updates.

For such editors, WebSocket is a preferred option.

The use case resembles that of an online chat application, where real-time messaging is essential.

WebSocket looks like a solution to every problem in real-time systems.

This is logical because both polling and server-sent events can be implemented with WebSockets.

This assumes both client and server support the WebSocket protocol.

There is a key difference: what can be done versus what should be done.

If only one-way message pushing is needed—such as server-to-client—server-sent events may suffice.

WebSocket becomes overkill in such cases due to unnecessary complexity.

System design is difficult because many technologies exist, and choosing the right one is challenging.

Mentioning multiple options shows breadth of knowledge.

True value lies in selecting the right tool for the specific requirement.

Engineers must evaluate pros and cons, prioritize simplicity, and avoid unnecessary engineering.

Investigate technologies used by favorite websites.

Web browsers enable reverse engineering via developer tools.

In Chrome, access the network tab to inspect communication.

Gmail uses polling, not WebSocket, for updates.

YouTube does not use WebSocket to play videos in a web browser on a PC.

Try it yourself—some websites will surprise you.