Autoscaling system design

A web service must be integrated with a monitoring system to track performance metrics, such as CPU utilization across all servers in the cluster.

Operators must define configuration parameters for scaling, including which metric to monitor and the threshold at which scaling should occur. These settings form an auto scaling rule.

A rule management component handles create, read, update, and delete (CRUD) operations for auto scaling rules. Rules are stored in a database.

A rule checker component periodically evaluates all active rules to detect when scaling is needed. It runs every minute or more frequently.

The rule checker must process many rules efficiently. One approach is to elect a leader instance in the rule checker cluster. This leader coordinates rule partitioning and assigns partitions to worker instances.

The leader uses consistent hashing to distribute rules across partitions.

An alternative avoids a central coordinator. Rules are split into a fixed number of partitions—such as 10—using range partitioning based on rule identifiers.

Ten worker machines are launched, each assigned one partition.

Each worker attempts to claim ownership of every partition using a distributed log mechanism.

Only one worker per partition succeeds in acquiring the log.

The first worker to secure the log becomes the owner of that partition.

As long as a worker maintains the log, it is the sole permitted processor of rules in that partition.

Each worker iterates over its assigned rules.

For each rule, it retrieves the metric name and fetches metric data from the monitoring system.

If the metric exceeds the threshold, the worker generates an auto scaling event—a task indicating scaling out.

Each such task is sent to a messaging system.

A task processing component consumes messages from the queue and handles resource allocation. This component is called a provisioning service.

Multiple instances of the provisioning service run as consumers of the message queue.

The competing consumers pattern is used to manage load. When task volume increases, more provisioning service instances are launched.

When a provisioning service instance receives a task, it identifies the target cluster and adds new machines to it.

Newly launched instances must advertise themselves so clients can discover and route traffic to them.

This is a known problem in distributed systems—service discovery.

A common system setup includes a load balancer in front of a web service cluster. This architecture enables dynamic scaling by allowing new machines to be registered with the load balancer during service provisioning.

The design presented is high-level but already practical. It demonstrates how basic auto-scaling concepts—specifically metric-based scaling—can be implemented with minimal prior knowledge.

This approach serves as a strong starting point for conversations with stakeholders. It enables clear, grounded discussions about system behavior and performance.

We already possess foundational understanding of messaging systems, including task producers, consumers, and internal task queue mechanics.

Later in the course, we will cover:

• Monitoring systems
• Leader election
• Distributed logs
• Database selection strategies

These include scalable, reliable, and fast communication between system components.

The ability to apply fundamental system design concepts allows for constructive, informed conversations with stakeholders.

This example illustrates how core system design principles function as building blocks for solving ambiguous, real-world problems.