Load shedding

Auto scaling is not instantaneous.

It involves multiple sequential steps:

• Monitoring systems collect and aggregate CPU usage metrics.
• Provisioning services allocate new machines.
• Web services must be deployed and bootstrapped on new machines.

These steps occur one after another.

Until new machines are added to the cluster, the service remains vulnerable.

A sudden traffic spike—such as a DDoS attack or a shift in client behavior—can overload all existing machines.

This causes a sharp drop in service performance.

To prevent outages, systems must include protective mechanisms.

Two key concepts are low-charging and rate limiting.

Low-charging means a server drops incoming requests when it approaches overload.

The goal is to ensure every accepted request is fully processed.

Each server accepts requests up to a defined threshold.

After that threshold, it drops incoming requests until more capacity is available.

Run a load test.

Generate high traffic on the server.

Monitor performance metrics such as CPU or memory usage, or request latency.

When any metric exceeds a defined limit, record how many requests are being processed.

This gives the maximum request-per-second rate the server can handle.

There are multiple ways to implement low-charging.

We must consider I/O models and thread structures.

Three common models:

• Thread per connection
• Thread per request with non-blocking I/O
• Event loop

Most real-world client-server applications use one of these models.

Option 1: Limit Connections

Limit the number of simultaneous connections the server accepts.

When the maximum is reached:

• The server can reject new connections.
• Or accept them and queue them, but not process them.

This limits the number of clients served at once.

Option 2: Limit Concurrent Requests

For thread-per-connection and thread-per-request models:

• Limit the number of active request-processing threads or worker threads.

For the event loop model:

• Limit the size of the task queue.

When the queue is full, new tasks are rejected.

Tuning thread pool size is challenging.

A large thread pool risks:

• Excessive memory usage
• High context-switching overhead
• Performance degradation

A small thread pool leads to:

• Underutilized CPU and memory
• Wasted resources

Low-charging tests reveal both extremes.

Option 3: Dynamic Threshold Monitoring

Monitor system metrics in real time.

When resource usage approaches a predefined limit, start dropping requests.

This provides adaptive protection.

RabbitMQ uses a mechanism called consumer prefetch.

It limits how many messages a consumer can receive before processing.

This prevents overwhelming a single consumer.

Prefetch controls the number of messages in flight.

It directly implements low-charging behavior in message delivery.

A prefetch buffer limits how many unacknowledged messages a consumer can receive.

When the buffer is full—meaning the consumer is still processing messages from the buffer—the broker stops pushing new messages.

The broker also removes all prefetched messages from the queue, making them invisible to other consumers.

Once any message in the buffer is acknowledged by the consumer, the broker pushes more messages—but never exceeds the buffer limit.

Consumer prefetch resembles TCP flow control.

In TCP, the receiver signals the sender how much data it can accept.

The sender transmits only up to that amount and waits for an acknowledgment.

The acknowledgment includes data on how many bytes the receiver is willing to buffer.

Server-side load shedding drops requests when too many arrive.

This is known as low-charging.

Backpressure involves notifying clients and asking them to reduce data publishing rates.

These concepts are conceptually close.

The distinction lies in client involvement.

Server-side mechanisms like low-charging may or may not notify clients.

If clients are notified and voluntarily reduce their load, it is backpressure.

Low-charging is necessary but not sufficient for backpressure.

Backpressure requires client cooperation.

Requests must be prioritized.

Some requests are more important than others.

For example, load balancers send health check requests to every service machine.

These pin requests must never be dropped.

Customer requests from real users are more important than robot-generated requests, like search crawlers.

Requests vary in cost.

Some are cheap—such as cached data retrieved quickly.

Others are expensive—requiring heavy computation.

Expensive requests can be dropped to save CPU cycles.

Traditional queuing uses FIFO—first in, first out.

With low-charging, LIFO—last in, first out—may be better.

When a service is overloaded, recent requests are more relevant.

A user likely refreshes the page after waiting too long.

Old requests are no longer valid.

If a request sits in a queue for a long time, the client likely has already timed out.

No need to process such requests.

The server uses timeout hints.

When a client sends a request, it includes a timeout value.

This value indicates how long the client will wait before retrying.

The server uses this to detect when a client has likely timed out.

This enables intelligent load shedding decisions.

The timeout hint is critical for identifying stale or obsolete requests.

It allows the system to avoid processing requests that are no longer relevant.

This ensures efficient resource use and improves overall system responsiveness.

The server evaluates a hint and drops the request if the hint has expired.

Auto scaling and low-charing can both be used in a service, but must be carefully configured and tested.

Low-charing may disable auto scaling by preventing CPU usage from reaching scaling thresholds.

For example:

• Auto scaling is set to launch new instances when CPU usage exceeds 70%.
• Low-charing begins dropping requests when CPU usage reaches 60%.
• CPU usage stabilizes at 60%, never reaching 70%.
• The system adds no new capacity.
• Requests are dropped instead of being served.

To avoid this conflict:

• Set a lower threshold for auto scaling (e.g., 60%)
• Set a higher threshold for low-charing (e.g., 70%)

This ensures:

• Auto scaling activates early when CPU usage reaches 60%
• Low-charing only engages when CPU usage reaches 70%

If auto scaling cannot provision capacity quickly enough, low-charing activates only as a last resort.

This sequence prevents request dropping during capacity growth.

The system scales out first, then drops requests only when capacity remains insufficient.