Regions, availability zones, data centers, servers

When designing a system for reliability, performance, and scalability, both software and hardware must work together.

Imagine a startup founder building a web application.

The app runs on a general-purpose server installed in their home garage.

It accepts HTTP requests from users worldwide, processes them, and stores data in a database.

Initially, both the web app and database run on the same physical server.

As traffic grows, the web app becomes CPU-bound, requiring more processing power.

The database, meanwhile, demands large amounts of memory and disk space.

A single server with balanced resources becomes expensive.

Splitting the workload improves cost efficiency.

Buy two servers: one compute-optimized with high-performance processors.

One storage-optimized with ample memory and disk space.

Move the database to its own dedicated server.

This setup functions well for a while.

One day, the founder’s dog turns off the application server.

This event reveals a critical vulnerability: single-point failure.

To address this:

• Purchase a server rack.

• Install both servers inside the rack.

• Secure the rack away from pets.

• Add a third server: one application server, one small load balancer.

The load balancer distributes incoming requests across the application servers.

To ensure data durability:

• Buy an additional database server.

• Implement data replication and automated backups.

This setup remains stable for a while.

A power outage in the area causes all servers to shut down.

This reveals another failure point: location-dependent power.

To improve resilience:

• Move one application server and the secondary database server to a friend’s garage.

• The friend’s garage is several blocks away.

• This creates a geographically distributed system.

If servers in the founder’s garage fail, the system remains available.

The friend’s garage servers continue serving users.

This design reduces the risk of total system failure.

The setup incurs additional costs: one more rack, one more load balancer.

Users in India report poor performance.

Latency between India and the US is high.

To address this:

• Contact friends in India.

• Ask them to build an identical infrastructure.

• Serve India users from India.

This reduces latency and improves user experience.

The system still has a single point of failure: the database.

Only one database server handles user requests.

The secondary database server replicates data for durability.

It does not serve user requests.

If the primary database fails, the secondary must become primary.

This transition requires a well-defined failover mechanism.

This issue will be discussed in later sections.

To build highly available, performant, scalable, and durable distributed systems:

• Software design principles are essential.

• Infrastructure design is equally critical.

• Software engineers must understand both layers.

• Only a few key infrastructure concepts are needed.

These include physical location, redundancy, and geographic distribution.

Understanding these enables engineers to build resilient systems without overcomplicating the design.

Servers have four key parameters: CPU, memory size, disk size, and network bandwidth.

Each component of a system determines the server resources it needs.

Among these, CPU is the easiest to scale.

Network bandwidth is typically the hardest to scale.

Multiple servers are mounted within a server rack.

Racks simplify server access, inspection, and physical manipulation.

Racks improve cooling efficiency and allow physical locking for enhanced security.

Each rack typically has its own dedicated network and power source.

While system designers usually don’t prioritize racks, exceptions exist.

We may want servers in different racks to ensure availability.

If one rack fails, spare hardware in other racks maintains system operation.

Alternatively, we may place all servers in a single rack to minimize communication latency.

This choice trades performance for availability.

A data center is a building housing many server racks.

It includes independent power, cooling, and physical security.

It functions like a startup’s garage but at a much higher cost.

A data center can become completely unavailable due to events like power outages, lightning strikes, tornadoes, or earthquakes.

When a startup’s garage loses power, servers are moved to a friend’s garage.

The same principle applies to data centers.

This concept is known as an availability zone.

An availability zone consists of one or more discrete data centers.

Each has independent power and networking.

Availability zones ensure high availability and scalability.

Instead of concentrating all hardware in a single data center, servers are distributed across multiple data centers.

If one data center fails or lacks capacity, systems can rely on others in the same availability zone.

A region is a group of isolated, physically separate availability zones.

Regions are spread across geographic areas.

For example, US and India represent two distinct regions.

A single country may contain multiple regions.

For instance, US East Coast and US West Coast are separate regions.

Each region typically includes at least two availability zones.

Availability zones are within a 100 km radius of each other.

AZs in a region are connected via high-bandwidth, low-latency networks.

Network latency between AZs is less than 2 ms.

Regions are spread globally.

Each region contains one or more availability zones.

Each availability zone contains one or more data centers.

Each data center holds many server racks.

Each rack mounts multiple servers.

Servers come in different types:

• Compute-optimized: many CPU cores

• Memory-optimized: large RAM

• Storage-optimized: high local disk throughput for reads and writes

When designing a system, non-functional requirements must consider deployment location and configuration.

Deployment decisions shape system availability, performance, and resilience.

To achieve worldwide scale, deploy a system copy in every region.

This improves performance by reducing latency—users access the system from a location physically closer to them.

It increases availability because if one entire region fails, user requests can be routed to other regions.

It enhances scalability by providing access to more hardware resources across regions.

Within each region, deploy the system across multiple availability zones.

This boosts system availability—failure in one zone does not bring down the entire system.

It improves scalability by distributing workloads across zones.

It increases data durability through rapid replication between zones.

Within each data center, deploy services across different server racks if high availability is required.

This provides redundancy at the physical level, minimizing single-point failures.

Different system components—such as microservices—should be deployed on servers optimized for their specific workload.

Workload characteristics include:

• Compute requirements
• Memory usage
• Disk storage needs
• Networking demands

Each component must be matched to server hardware that best supports its operational profile.

This ensures efficient resource utilization and performance consistency across the global system.