Why Retries Can Make Things Worse
Retries can degrade system performance if not designed carefully. If Service B is already under high load, retries add more demand.
This can trigger a retry storm—where failed requests cause cascading retries—amplifying pressure on Service B instances.
Exponential Backoff: Delaying Retries
Retries should not happen immediately. A client must wait before attempting a retry.
The wait time increases after each attempt.
This pattern is known as exponential backoff.
It is called exponential because the delay grows exponentially with each retry.
For example:
- First retry: wait 1 second
- Second retry: wait 2 seconds
- Third retry: wait 4 seconds
- Fourth retry: wait 8 seconds
And so on.
Limiting Retry Count
Retries should not be infinite.
A maximum number of retries must be defined.
For instance, a client may retry up to three times.
If the request remains unsuccessful after three attempts,
the system should apply alternative strategies:
- Cancel the request
- Failover to another service
- Use a fallback response
Avoiding Coordinated Retries with Jitter
Even with exponential backoff, clients may retry at similar times.
This causes traffic spikes after each delay.
To prevent this, we introduce jitter.
Jitter adds random variation to retry intervals.
This spreads out retry times and reduces load spikes.
Simple Formula for Delay
A practical formula for calculating retry delays is:
delay = base_delay * 2^attempt + random_jitter
Where:
base_delayis the initial delay (e.g., 1 second)attemptis the retry number (0-indexed)random_jitteris a random value between 0 and base_delay
Jitter Beyond Retries
Jitter is not limited to retry logic.
It is widely used in distributed systems.
One example: application monitoring.
An agent runs alongside an application.
The application logs exceptions to a file.
The agent reads the log and sends aggregated data to a monitoring service.
The agent sends data periodically—say, once per minute.
If all agents send data at the start of each minute,
the monitoring service faces large traffic bursts.
Adding jitter spreads the data transmission across the minute.
Applying Jitter to Periodic Workloads
Jitter is useful in any system that aggregates data over time.
For example, a daily job that runs at midnight.
If it spawns many tasks that share a resource,
they should not all start at once.
Jitter ensures tasks begin at staggered times.
Jitter Solves the Thundering Heart Problem
The thundering heart problem occurs when many processes wait for an event.
When the event happens, all processes begin competing.
Jitter helps by spreading the start times of these processes.
This reduces resource contention and improves system stability.
Questions
- Why can retries make a system's situation worse even if the service is already under stress?
- What is exponential backoff, and how does it work in practice?
- How does jitter improve retry behavior, and why is it necessary?
- What happens if all clients retry at the same time after a failure?
- Why is it important to limit the number of retry attempts?
- Can exponential backoff alone prevent traffic spikes in a distributed system?
- How does jitter apply beyond retries in distributed systems?
- What is the 'thundering heart' problem, and how does jitter help?
Key takeaways
- Retries can worsen load on a service already under stress, leading to a retry storm.
- Exponential backoff increases wait time between retries—starting at one second, doubling each time.
- Jitter adds randomness to retry intervals to prevent all clients from retrying simultaneously.
- Retries should be limited in number to avoid infinite loops and to trigger fallback strategies.
- Exponential backoff with jitter is simple to implement and effective in practice.
- The jitter concept is not limited to retries—it applies to periodic workloads and monitoring agents.
- Traffic spikes can occur when systems aggregate data and send it at fixed intervals; jitter spreads the load.
- Jitter helps prevent resource contention in systems with many processes waiting for an event.
- Without proper retry design, distributed systems can experience cascading failures or performance degradation.