Durability

Durability means that once data is successfully submitted to a system, it remains intact—even if faults occur.

For example, when a user uploads a photo to Instagram, the system must guarantee that the photo will not vanish after the confirmation message is returned.

Systems achieve durability by creating and maintaining multiple copies of data.

Three common approaches are backup, write, and replication.

Backup: Periodic Data Copies

Backup involves copying data from non-volatile storage—like a disk—periodically.

For instance, a backup might occur once a day.

Such a copy is known as a backup.

Three strategies exist for creating backups:

• Full backup: A complete copy of all data is created.

  Full backups allow rapid data restoration after a failure.

  However, they are slow to create, especially with large datasets.

• Differential backup: Only the changes since the last full backup are saved.

  A full backup occurs once a week.

  Daily backups store only the differences since the last full backup.

  Differential backups are smaller and faster to create than full backups.

  To restore data, both the full backup and the last differential backup are required.

  This is slower than restoring from a single full backup, but still acceptable.

  However, differential backups grow in size over time.

  On Monday, the first differential backup may be small.

  On Saturday, the last differential backup may become large and slow to create.

• Incremental backup: Only the changes since the previous backup are saved.

  A full backup occurs every Sunday.

  Daily backups contain changes since the prior backup.

  Monday’s backup includes changes since Sunday.

  Tuesday’s backup includes changes since Monday.

  Each incremental backup is smaller and faster to create.

  The downside is longer restoration time.

  Restoration requires applying all incremental backups in sequence after the full backup.

Backup improves durability.

But it is insufficient on its own.

First, backups cannot be created immediately after data changes.

In the best case, backups happen every several minutes.

If a disk fails, several minutes of data may be lost.

Second, data restoration takes time.

A recovery process may require finding a new drive, installing it, and applying backups.

During this time, users cannot access their data.

To solve these issues, systems use RAID—Redundant Array of Independent Disks.

RAID combines multiple physical disks into a single logical unit.

From the application’s perspective, RAID appears as one virtual device.

Data is written to this virtual device and copied across all physical disks.

If one disk fails, data remains accessible because it exists on another disk.

This ensures durability even during hardware faults.

RAID can increase read throughput beyond simple redundancy.

Because data is duplicated, read requests can be serviced by any disk in the set.

This allows the application to read from multiple disks in parallel.

RAID configurations are known as RAID levels.

The one shown on screen is RAID1, also called a mirrored volume.

In RAID1, both disks contain identical data.

RAID0 uses any number of disks, but we'll use two for simplicity.

In RAID0, data blocks are written alternately across disks.

This technique is called data striping.

RAID0 has no data mirroring.

There is no data redundancy in RAID0.

Therefore, RAID0 does not increase durability.

In fact, RAID0 decreases durability.

A single disk failure in RAID0 causes complete data loss.

The more disks in RAID0, the higher the risk of total data loss.

Despite its durability flaws, RAID0 has one valid use case.

It improves performance.

Reads and writes can occur concurrently.

This results in higher throughput for both read and write operations.

Instead of copying data at the disk level, we can copy it at the application level.

This means multiple machines—possibly geographically distributed—each have their own disk or storage.

When data is written to one machine, it is quickly copied to several others.

This is the concept of replication.

Replication is a major topic in system design.

We will cover:

• Popular replication algorithms
• Trade-offs between synchronous and asynchronous replication
• How replication works
• Handling failed replicas
• Resolving inconsistencies between replicas

For now, remember that replication increases durability, availability, and read throughput.

The three mechanisms—RAID, replication, and backups—are not mutually exclusive.

High-durability systems typically use at least two, and often all three.

A backup mechanism is essential for high durability.

Replication is widely used today.

Public cloud storage relies heavily on replication.

Whether RAID is needed may not be obvious when replication is present.

However, both RAID and replication are used in practice.

For example, Cassandra's documentation states that while replication prevents data loss,

RAID1 should be used for the commit log disk if extra redundancy is needed.

Other techniques help ensure durability.

Versioning exists—like a backup—but applies to specific objects, not entire storage.

Systems may also include safeguards against accidental deletion.

High durability is not just about copying data.

It also requires ensuring all copies remain uncorrupted and healthy.

Disks fail frequently.

So systems need mechanisms to regularly verify data integrity.

This is typically achieved using checksums.

When a file is read into storage, the system calculates and stores a checksum.

When the file is retrieved, the client computes a checksum from the retrieved data.

It compares the result with the stored checksum.

If they match, the data is uncorrupted.

If they differ, the data was corrupted.

The system then repairs the data using redundant copies.

It creates a new copy from other healthy replicas.

For example, Hadoop Distributed File System (HDFS) runs background processes that regularly scan data.

These processes detect and remove corrupted data.

By copying data and verifying integrity regularly, high durability is achieved.

And here comes our standard question: How high?

No system can guarantee 100% durability over a long time period.

Data copies can be lost or corrupted due to:

• User errors
• Hardware failures
• Natural disasters (earthquakes, fires, floods)
• Theft

Despite these risks, systems can aim extremely close to 10