What to do with failed requests

When Service A calls Service B and the request fails, several root causes may exist.

• Service A cannot establish a connection to Service B due to a network partition.
• Service B fails to process the request and returns an exception.
• There is no response at all, triggering a timeout.

With such failures, we must respond using one of these strategies: cancel, retry, failover, or fallback.

Cancel means we report the exception and terminate the operation.

We tried, we failed, and we lock what happened—no further action is taken.

This strategy is appropriate when the failure is not transient.

Even if the request is resent, success is unlikely.

Example: A request fails due to invalid authentication.

No matter how many times the request is resubmitted, invalid credentials will always result in failure.

Such faults should be locked and handled by the service operator.

Proper logging and failure reporting will be covered later in the course.

Retry means we reattempt the failed operation—either immediately or after a delay.

In the case of a failed request, this means sending the same request again.

Possible reasons for success on retry:

• The first request hit a faulty instance of Service B.
• A second attempt may hit a different, healthy instance.
• A brief network hiccup resolved by the time of retry.

Retry increases the probability of success due to instance variability or transient issues.

Failover means switching to a redundant or standby system of the same type.

This concept is widely used in distributed systems.

Example: A database server.

If a single database server fails or becomes inaccessible, the entire system may go down.

To improve reliability, deploy a second database server in a different data center.

Data written to the primary server is replicated to the secondary.

When the primary fails, all database traffic is redirected to the secondary.

From that point, the secondary server serves all queries.

Hardware load balancers use failover similarly.

If the active load balancer fails, a standby device takes over.

Many distributed systems use leaders and followers.

The leader handles all operations.

Data is replicated to followers, which also serve requests.

If the leader fails, the failover mechanism promotes one follower to become the new leader.

We will discuss databases, load balancers, leaders, and followers in later course sections.

In the original example, when Service A fails to submit a message to Service in Service B,

Service A uses a different mechanism to deliver the message.

The Oxford dictionary defines fallback as an alternative plan for emergencies.

This definition applies to distributed systems.

Fallback provides a secondary path when the primary request fails.

It does not require a full system switch.

It enables message delivery or operation continuity even when the primary path fails.

The fallback mechanism is distinct from retry or failover.

It does not reattempt the original request.

It uses a different method entirely.

This ensures system resilience under failure conditions.

Several strategies exist for managing failed requests in distributed systems.

Local Disk Storage

Service A can store the message on its local disk and attempt re-delivery later.

Message Queuing

Service A can send the message to a queue.

• The queue may be internal, running in memory on the service A instance.
• Alternatively, it may be an external messaging system.

Service B may consume messages directly from that queue.

Or another system may process the messages instead.

Fallback mechanisms offer increased system reliability.

However, they come with significant drawbacks.

• A second data processing path must be implemented and maintained.
• Fallbacks are difficult to test thoroughly.
• They may remain rarely used.
• When the primary path fails, the fallback may not be prepared for increased load.

We can combine multiple approaches:

• Retry the request two to three times.
• If the request still fails, activate a failover to a fallback path.

This layered approach uses retries before falling back to alternative processing.