Timeouts

In distributed systems, failures are inevitable. Some failures are immediate, others are silent and destructive.

When a request fails to return, both client and server may waste resources. This can lead to cascading failures.

Timeouts are essential to convert slow failures into fast ones.

Fast failures occur immediately.

Example: A service returns a 503 error.

Slow failures occur silently.

Example: No response from a service after a request is sent.

Both services hold onto resources.

These include memory, threads, and open connections.

As more requests pile up, resources may be exhausted.

Eventually, one service may crash.

There are two types of timeouts:

• Connection timeout
• Request timeout

Connection timeout defines how long a client waits to establish a connection.

Typical value: Tens of milliseconds.

This is because connection setup is fast and lightweight.

Request timeout defines how long a client waits for a response after the connection is established.

It applies when server-side processing takes too long.

Setting a request timeout is tricky.

If too high:

• Resources are still consumed while waiting.

If too low:

• The client may miss responses.

A common practice:

• Analyze the 0.1% percentile of request latency.

• Set this value as the request timeout.

This reduces tail latencies.

Still, 0.1% of all requests will timeout.

The failed request must be handled.

It may require retries, logging, or fallbacks.

The system must define how to respond to timeouts.