How to choose a network protocol

When designing a distributed system, selecting the right network protocol is critical.

This decision applies to both client-facing communication and internal system messaging.

The choice depends heavily on system requirements and use cases.

Here are key guidelines to consider.

TCP ensures reliable, ordered data transfer.

Lost packets are retransmitted and packet order is preserved.

TCP is connection-oriented: both client and server must establish a connection before data transfer.

Connection establishment uses a three-way handshake.

Three-Way Handshake Steps

• Client sends a SYN message to the server.
  • Contains a random sequence number A.
• Server replies with a SYN-ACK message.
  • Contains its own random sequence number B.
  • Includes an acknowledgment number (A + 1).
• Client responds with an ACK message.
  • Contains the number (B + 1).

At this point, both parties acknowledge the connection.

The connection is established.

Data can now flow in both directions.

Each client may establish multiple TCP connections to a server.

Each TCP connection consumes memory, primarily via TCP buffers.

Modern servers can manage hundreds of thousands of simultaneous TCP connections.

This is useful in scenarios like IoT, where many devices connect periodically.

Note: Having many open connections does not mean the server can process hundreds of thousands of concurrent requests.

Request processing incurs additional overhead.

For example, in a thread-per-request model, each request requires a dedicated thread.

A single TCP connection typically uses kilobytes of memory.

A Java thread may use up to a megabyte of memory.

There is a significant difference between maintaining many open connections and processing data from each connection simultaneously.

Before sending an HTTP request, a client and server must establish a TCP connection.

HTTP 1.0 opens a separate TCP connection for each request.

This is inefficient due to repeated handshake rounds.

HTTP 1.1 introduced persistent connections.

With persistent connections, a single TCP connection can serve multiple HTTP requests.

This reuses the established TCP connection.

Persistent connections reduce the number of connection setups.

This reduces latency for subsequent requests.

The result is faster response times for repeated client interactions.

Persistent connections improve performance in high-frequency, low-latency scenarios.

This efficiency is especially valuable in consumer-facing services.

The choice of protocol must align with both performance and resource constraints.

Understanding these mechanisms enables better design decisions in real-world systems.

UDP provides no connection setup, no handshaking, no delivery guarantees, no packet ordering, no duplicate protection, and no error recovery.

It is fast but unreliable. Data may be lost or arrive out of order.

Use UDP when:

• Data must be transferred as soon as possible
• Occasional data loss is tolerable

The principle: the most recent data—though possibly incomplete—takes priority over older, more complete data.

TCP offers reliable, ordered delivery with error recovery.

Use TCP when:

• Speed is still important
• Data loss must be avoided

TCP treats communication as a stream of bytes.

It does not recognize discrete messages.

When a client sends multiple messages over TCP:

• The server receives them as a continuous byte stream
• Message boundaries are lost

To distinguish one message from another, we must define message boundaries.

Both client and server must agree on:

• How to set boundaries on the client side
• How to reassemble messages on the server side

This is the core function of a protocol.

HTTP defines how messages are framed on the client side and reassembled on the server side.

It establishes clear message boundaries in a byte stream.

Private networks may allow open TCP ports.

Public-facing systems—especially those accessible via browsers—face strict security rules.

Firewalls typically block all traffic except web traffic on port 80 (HTTP).

Direct TCP or UDP socket connections from browsers are blocked by default.

HTTP is universally supported:

• All browsers
• All web servers
• All client libraries

It has extensive development and debugging tools.

HTTP is slightly slower than TCP but remains reliable.

HTTP is ideal for systems that need to provide a public API.

Its design supports interoperability and broad accessibility.

HTTP does not support real-time communication.

It is not suitable for low-latency, bidirectional, real-time data.

These characteristics make HTTP a foundational protocol for public-facing distributed systems.

The simplicity and standardization of HTTP enable widespread adoption and integration.

Its rules are embedded in every web interaction.

This makes it a critical choice for public APIs and browser-based services.

The design of HTTP reflects the need for both reliability and broad compatibility.

These traits are essential in real-world distributed systems.

The protocol's structure ensures that message boundaries are preserved and interpreted correctly.

This enables consistent, predictable communication between diverse systems.

The combination of framing, reliability, and security makes HTTP a cornerstone of modern web infrastructure.