Blocking vs non-blocking IO

A socket is an endpoint defined by an IP address and a port number.

Each end of a connection has a socket. A connection is uniquely identified by a tuple of four values: source IP, source port, destination IP, destination port, and protocol.

On the server side, a socket is bound to a specific port—such as port 80. All clients know this port and use it to initiate connections.

When a client connects, it specifies the server's host name and port. The client identifies itself using its IP address and a randomly assigned port.

This random port is short-lived and is known as an ephemeral port.

Ephemeral ports allow a single client to establish multiple connections to the same server, each with a distinct port number.

Each connection thus has a unique tuple, ensuring no two connections are identical.

Sockets operate in two modes: blocking and non-blocking.

In blocking mode, when a read or write operation is called, the calling thread suspends execution until the operation completes.

For a server, this means it must wait for data to arrive before proceeding.

In non-blocking mode, the operation call returns immediately, regardless of whether data is available.

The server can then proceed with other tasks and read data from clients at a later time.

With blocking sockets, each connection requires a dedicated thread.

When one thread is blocked waiting for data, the server switches between threads to process other sockets.

This allows the server to maintain progress by continuously switching between active threads.

This model—where IO operations are waited for until completion—is known as blocking IO.

With non-blocking sockets, a single thread can manage multiple connections.

The server allocates only a few threads—typically one per CPU core.

Each thread handles multiple connections simultaneously.

During request processing, a thread may read only part of a message and then proceed to read from other clients.

It does not wait for the full message to arrive.

The operating system raises an event when a socket is ready for IO.

This event signals the thread that data is available.

This approach—where IO operations do not wait for completion—is known as non-blocking IO.

Non-blocking IO reduces the cost of handling connections.

A connection is inexpensive: it is merely a file descriptor and a listener.

A thread, however, consumes significant system resources.

Non-blocking IO enables servers to handle far more connections than blocking IO.

Blocking and non-blocking IO apply to both reading request data and writing response data.

Three common models exist for request processing:

• Thread per connection
• Thread per request with non-blocking IO
• Event loop

In the thread per connection model, each connection has a dedicated worker thread.

The business logic of the application runs inside this worker thread.

In most cases, the business logic itself is blocking.

Some code executions can block a worker thread. Examples include:

• Making a remote HTTP call
• Querying a database
• Reading data from disk

When such operations occur, the worker thread waits for completion. This is acceptable in a thread-per-connection model because each thread serves one connection. Blocking does not affect other threads.

In this model, a thread pool handles multiple connections. Each IO thread serves multiple sockets.

• A single IO thread reads bytes from multiple sockets
• Data is written into dedicated buffers
• When sufficient bytes are read to form an HTTP request, a worker thread from the pool processes it

Blocking worker threads is acceptable. However, blocking the IO thread is unacceptable because it halts all client IO operations.

This model uses one thread per CPU core that handles all IO operations and acts as a worker thread.

• The thread runs continuously
• It manages all IO operations and executes application-specific request processing
• It eliminates the need for a dedicated thread per request

Conceptually, the model functions like a task queue:

• The event loop acts as both producer and consumer
• After reading request bytes from a socket, it creates one or more tasks
• Tasks are submitted to the queue for asynchronous processing
• The event loop retrieves the next task and processes it

Blocking the event loop thread halts all operations. If a blocking call is required:

• Create a separate thread
• Execute the blocking operation in that thread

People often misunderstand the event loop due to its single-threaded nature.

The confusion arises from the intuition that more threads mean more work.

But on a single-CPU machine:

• Only one thread can execute at a time
• The CPU switches between threads to make it appear tasks run simultaneously
• This is concurrency, not parallelism

• Concurrency: Multiple tasks run in sequence on one CPU core
• Parallelism: Multiple threads run simultaneously on multiple CPU cores

On a single-CPU system, only one thread runs at any moment.

All work can be done by one thread if tasks do not involve blocking IO.

• Blocking IO causes thread idle time
• During idle time, the CPU could process other tasks
• Without blocking IO, a single thread can handle all requests efficiently

Thread creation overhead is avoided

Multiple threads are only needed to compensate for blocking IO operations

When tasks do not require blocking IO, a single thread suffices without performance loss

In such cases, a single thread can even outperform multiple threads due to reduced overhead

The thread per connection model was widely used in past server designs. Each incoming connection spawns a dedicated thread.

This model is simple to implement and debug. It allows direct observation of request progress via thread stack traces.

Exceptions surface clearly in the stack, making error handling straightforward.

Thread-local variables work naturally in this model.

However, it is less scalable and resource-intensive.

Threads consume significant memory and CPU.

Context switching—swapping threads in and out of execution—uses substantial CPU cycles.

Modern multi-core servers can manage hundreds of threads.

But under thousands of concurrent requests, resource exhaustion becomes likely.

Performance degrades rapidly as thread counts grow.

This model is vulnerable to traffic spikes.

When timeouts occur, clients retry, increasing request volume.

The system may need to spawn thousands of new threads.

This can overwhelm server resources.

Solutions like load sharing and rate limiting help mitigate these risks.

These mechanisms are discussed later in the course.

Debugging is easier in thread-per-request systems.

Stack traces are meaningful and traceable.

The model is intuitive for developers.

It remains popular due to ease of development and operation.

Used by popular servers such as Jetty, NGINX, and Tomcat.

Implemented in native frameworks, Node.js platforms, and Netflix’s Zoo API gateway.

Easy to implement, test, and debug.

However, asynchronous code is harder to read and analyze.

Stack traces are meaningless and do not reflect request flow.

Debugging becomes difficult without clear execution paths.

This model is efficient for I/O-bound workloads.

It scales well with thousands of open connections.

It handles sudden traffic spikes more resiliently.

When clients retry due to timeouts, the system absorbs the load without creating new threads.

It avoids resource exhaustion during spikes.

The event loop model is better suited for high-concurrency scenarios.

If request processing involves heavy I/O, the event loop model is preferable.

If workloads are CPU-bound, the thread request model is better.

Non-blocking IO provides little benefit in CPU-bound scenarios.

CPU-bound tasks do not benefit from asynchronous I/O.

Development and operation are simpler with thread request models.

Thus, both models have distinct use cases.

There is no universal winner.

Choice depends on workload characteristics.

I/O-bound workloads favor event loop models.

CPU-bound work as favor thread request models.