Fault tolerance, resilience, and reliability

Fault tolerance is the ability of a system to continue operating properly when one or more of its components contain faults.

To use the term precisely, distinguish three related ideas:

1. An error is a mistake, often introduced while writing software.
2. The error may create a fault—a defect or bug that can cause incorrect behavior.
3. One or more faults may produce a failure, when the system cannot perform a required function.

A fault-tolerant system isolates faults so that they do not become a failure of the whole system.

The terms are often used interchangeably, but one useful distinction is downtime.

• A highly available system accepts that downtime may occur and tries to minimize it.
• A fault-tolerant system aims to continue operating without downtime while faults occur.

Under this distinction, fault tolerance is a stronger form of high availability. A fault-tolerant system is highly available, but a highly available system is not necessarily fault tolerant.

A car carries a spare tire. If a tire fails, the car stops while the tire is replaced. Service is restored quickly, but there is downtime. This resembles high availability.

An airplane cannot stop in mid-flight when an engine fails. Other engines must take over until the aircraft can land. This resembles fault tolerance.

Both systems use redundancy, but they provide different continuity guarantees.

A fault-tolerant design uses the same principles and processes as a highly available design.

Reaching a zero-downtime goal may require more redundant resources than ordinary high availability. That additional capacity raises the cost of the system.

In everyday engineering discussions, people may still use both terms loosely. Clarify the expected behavior instead of debating terminology alone.

A resilient system maintains an acceptable level of service while facing faults.

Resilience and fault tolerance have overlapping meanings and are often used interchangeably. Resilience and resiliency are also variants of the same noun.

The practical question is whether the mechanisms that handle faults continue to work when real faults occur.

Fault-handling mechanisms should be exercised continually. One option is to deliberately introduce faults, such as terminating random service instances.

Chaos engineering provides techniques for simulating system failures and observing system behavior.

It is related to a game day, but their emphasis differs:

• A game day focuses primarily on the team and its actions, building muscle memory for incident response.
• Chaos engineering focuses primarily on system behavior and can automate experiments that run repeatedly, including in production.

The lesson uses a compact model:

reliability = high availability + correctness + acceptable response time

Correctness means that the system returns the right result. Time means that the result arrives soon enough to remain useful.

A system can be continuously available yet return corrupted or incorrect data. It can also answer so slowly that the response no longer helps the client.

In both cases the system may be available, but it is not reliable.

Similar terms can carry different connotations.

High availability and reliability sometimes emphasize known, expected failures:

• server crashes;
• data-center power outages;
• network problems.

Resilience sometimes emphasizes less predictable conditions:

• short or sustained traffic surges;
• unexpected dependency failures;
• events whose timing and shape are unknown.

Calling a system both reliable and resilient can therefore emphasize its ability to handle expected and unexpected failures.

Not everyone makes this distinction. Some engineers treat resilience as a necessary part of reliability and use “reliable” for both categories.

High availability, fault tolerance, resilience, and reliability overlap. They lead to the same family of architectural principles and operational practices.

When reading or writing a design document, inspect what the author means:

• Is short downtime acceptable?
• Which faults must be isolated?
• What level of degraded service is acceptable?
• Must responses be correct and timely?
• Are expected and unexpected failures both in scope?

Precise guarantees are more useful than a list of impressive adjectives.