Synchronous and asynchronous clients

Systems that shape incoming traffic using load shading and rate limiting are known as admission control systems.

When a client's request is rejected by either a rate limiter or a load shader, such systems usually return back an exception.

Two common response status codes are 49, 2 money requests, and 503, service unavailable.

The client is expected to gracefully handle responses with these status codes.

In the previous video, we listed the options available.

The client should also remember to log the exception and emit a metric.

We will discuss logging and monitoring in detail later on the course.

One thing the client should avoid when it sees these codes is an immediate retry.

The service is already overloaded, and sending more requests to the service is counterproductive and only makes sense worse.

Retrying with exponential back-off and jitter is a much better option.

First, delays between retries will give the service time to scale out.

Second, jitter will help distribute requests more evenly.

But what if the service is not elastic and it won't scale quickly?

What if the service is overloaded and becomes slow or completely unavailable for an extended period of time?

As smart as the retries are, they won't help.

What we see here is the instance of a more general problem.

The problem with the failure of the service is not transient.

And the service cannot recover quickly.

The client may think that the problem is transient, and that retries will eventually succeed.

But instead, the client may queue up a large number of original requests along with retries, exhaust client-side resources, and crash.

Let's elaborate on this.

Remember when we discussed blocking and non-blocking IO?

Back then, we looked at things from the server's perspective.

But clients can also be built on either blocking or non-blocking IO.

Let's say we have a client application inside which one or more threads are running.

When the client application needs to send a request to the server, each thread either establishes a new connection,

or reuses an existing connection from the pool of persistent connections, and executes IO operations.

In case of the blocking IO model, the thread is blocked, waiting for IO to complete.

Simply put, the thread is blocked until the response is available.

If the client needs to send multiple concurrent requests to the server, we need to create multiple threads inside the client application.

And each thread will create and use its own connection.

To create and manage connections, we use various client libraries.

For example, Apache HTTP client.

Blocking IO clients are also called synchronous.

In case of a non-blocking IO client, application threads do not use connections directly.

Instead, threads send requests to the request queue.

A single IO thread dequeues requests from the queue, establishes a new connection,

or obtains one from the pool, and streams request data out using that connection.

When any of the responses comes back, the application is notified,

and one of its threads can then process the response.

As you can see, the non-blocking IO client is more complex, as it takes care of request queue, the IO thread, and connection management.

But a single thread now can handle all the IO.

And if the client application needs to make many concurrent requests, it no longer needs to allocate a separate thread for each request.

Non-blocking IO clients are often referred to as asynchronous clients.

One of the advantages of asynchronous clients is its simplicity.

It is easier to write, test, and debug applications that use synchronous clients.

And if a client makes a small number of concurrent requests, asynchronous client can have lower latency.

Asynchronous clients, on the other hand, guarantee higher throughput if the client has to make many concurrent calls.

This property also enables asynchronous clients to be more efficient in handling traffic spikes.

Asynchronous clients are more resilient to server outages and degraded performance.

When a server is unavailable or overloaded, a client may accumulate failed requests and retry them.

Piling up requests in a queue is significantly cheaper than piling up threads.

The key difference between blocking and non-blocking IO clients lies in how they manage concurrent requests from the client.

Non-blocking IO clients can handle many more concurrent requests than blocking clients.

For example, when watching a video, a browser acts as a client.

It retrieves video segments every few seconds from a CDF.

No concurrent requests are needed in this scenario.

The server that serves user requests acts as a client to downstream systems.

For instance, it calls a video metadata service.

When thousands of users access the server simultaneously, the server must forward thousands of concurrent requests to its dependencies.

A common architecture includes:

• A non-blocking IO server
• A pool of worker threads for request processing
• A blocking IO client to call downstream dependencies

An alternative version uses:

• A non-blocking IO server
• A pool of worker threads
• A non-blocking IO client to call dependencies

Both architectures are widely used today.

The industry is gradually shifting toward non-blocking IO across all layers.

If a downstream dependency becomes unavailable or slow and the issue is persistent:

• Synchronous clients block worker threads
• Asynchronous clients accumulate requests in a queue

Both models suffer under sustained overload.

Asynchronous clients handle failures better than synchronous clients.

However, both models remain vulnerable.

Even with an event loop model, problems persist.

The task queue will eventually fill up.

No matter the thread model, if incoming traffic exceeds outgoing traffic:

• A backlog of pending requests grows
• The backlog eventually overflows

In such cases, the client should stop sending requests for a short period.

This reduces client-side resource consumption.

It allows the service to recover faster during overload.

This behavior is the foundation of the circuit breaker pattern.

The circuit breaker pattern will be discussed in the next video.