What Is Overload?
Overload occurs when existing computational resources cannot process all incoming traffic.
Why Manual Scaling Is Ineffective
Adding servers manually during load spikes is not optimal.
- It becomes operationally expensive when done frequently.
- It increases the risk of overload if servers are under-provisioned for long periods.
- It leads to wasted costs if servers are over-provisioned for extended times.
The Challenge of Resource Estimation
Accurately predicting required computational capacity is a difficult problem.
The Goal of Autoscaling
Ideally, systems should automatically add servers when load increases and remove them when load decreases.
Autoscaling in Cloud Computing
Autoscaling is a core concept in cloud environments.
Benefits of Autoscaling
- Improves system availability by reducing the impact of traffic spikes.
- Reduces costs by only using resources when needed.
- Enhances performance by distributing load across more servers, reducing individual server load.
Three Types of Autoscaling
- Metric-based
- Schedule-based
- Predictive
Metric-Based Autoscaling
Uses performance metrics to trigger scaling decisions.
- Example metric: CPU utilization
- Example threshold: 80%
- Trigger: When average CPU utilization across all servers exceeds 80%, add servers.
Other common metrics include:
- Memory usage
- Disk usage
- Request count
- Active threads count
Schedule-Based Autoscaling
Applies scaling based on predefined time schedules.
- Example: Scale out during business hours, scale in at night and on weekends.
Predictive Autoscaling
Uses machine learning models to forecast traffic.
- Example metric: CPU utilization
- Example target: 50%
- Goal: Maintain average CPU utilization at 50% across all servers.
Models analyze historical data to predict future traffic.
- They forecast CPU utilization changes.
- Trigger autoscaling when a predicted increase or decrease is detected.
Combining Scaling Policies
Multiple policies can be used together.
- Metric-based scaling responds to real-time resource usage.
- Predictive scaling anticipates future load and scales in advance.
Applying Autoscaling to Messaging Systems
For systems with competing consumers (e.g., Revitem queue, SQS):
- Monitor queue depth
- Add consumer instances when depth exceeds a limit
- Shut down redundant consumers when queue depth decreases
For systems without competing consumers (e.g., Kinesis, Kafka):
- Use consumer offsets
- Scale message consumption by adding partitions
Partition Scaling in Kafka and Kinesis
Both Kafka and Kinesis offer APIs that enable increasing the number of partitions in a queue on demand.
Kafka currently does not support reducing the number of partitions in a queue.
The Competing Consumers Pattern
The competing consumers pattern can be used to design an autoscaling system.
How this works is demonstrated in the next video.
Questions
- What happens when existing computational resources are insufficient to process all incoming load?
- Why is manually adding servers to a cluster not a sustainable solution?
- What are the three types of autoscaling, and how do they differ in their decision-making approach?
- How does metric-based autoscaling use thresholds to trigger scaling?
- Can schedule-based autoscaling anticipate traffic patterns, or does it only respond to fixed times?
- What role does machine learning play in predictive autoscaling?
- How can a messaging system use the competing consumers model to implement autoscaling?
- What limitation exists in Kafka’s partition management that affects autoscaling in messaging systems?
Key takeaways
- Autoscaling helps prevent overload by automatically adding or removing servers based on load.
- Manual scaling is operationally expensive and can lead to either under-provisioning or over-provisioning.
- Resource capacity estimation remains a difficult problem in real-world systems.
- Metric-based autoscaling reacts to real-time metrics like CPU or memory usage with defined thresholds.
- Schedule-based autoscaling follows predefined times, such as scaling out during business hours.
- Predictive autoscaling uses historical data and machine learning to forecast traffic and scale in advance.
- Multiple scaling policies can be combined—metric-based for real-time response and predictive for proactive scaling.
- In messaging systems, autoscaling can be applied via queue depth or partition count, depending on support for competing consumers.
- The competing consumers pattern is a potential design for building an autoscaling system, though it requires specific system support.