Performance

When people hear 'performance,' they typically think of either processing time or the rate at which something is processed. It can be both.

For example, when you press play on a video, you expect the content to reach your device quickly. A key non-functional requirement for a video hosting service is to deliver content rapidly—even under poor network conditions. Here, time matters.

But when you like or dislike a video, you don’t care whether your reaction is counted immediately or after several minutes. What matters is that the system can handle all incoming reactions at scale. In this case, the rate at which reactions are processed is critical.

The terms 'time' and 'rate' are too broad in software engineering. We use 'latency' and 'throughput' instead.

When a client sends a request, it travels across the network, arrives at the server, is processed, and then the response returns to the client. The total time from request to response is called response time.

Response time includes:

• Network delay: time the request spends traveling across the network in both directions
• Service time (processing time): time the server takes to process the request

We could further break this down to include time spent reading data from a socket on the server. However, this detail is not essential for current discussion.

The term 'latency' is used in many different ways. Common meanings include:

• Network delay (network latency)
• Service time (server-side latency)
• Full response time (client-side latency)

The word 'latent' means inactive or dormant. When data travels across a network, the action is not yet active—hence, 'latent'.

Because 'latency' can mean different things, context is crucial. If unsure, clarify with your inter-year.

If an inter-year asks about improving network latency, they likely want discussion on the OSI model and network protocols.

If they ask about server-side latency, they expect talk about server optimizations such as:

• Faster algorithms
• Memory versus disk tradeoffs
• Thread pools
• Parallel processing
• Local caching

If they ask about client-side latency, they expect discussion on:

• Blocking versus non-blocking I/O
• Message format design
• Data compression
• Content delivery networks (CDNs)
• External caching

These are just a few potential topics.

Latency is never constant. Even identical requests from the same client will have different network delays, processing times, and thus different response times.

Going forward, when I say 'latency,' I mean response time.

Although everything discussed here is foundational, it sets the stage for deeper exploration of performance in later sections.

Latency—whether network or processing—can be measured in two common ways. One is average latency, calculated by summing individual request latencies and dividing by the total number of requests.

While easy to compute, average latency is often misleading. It can mask poor performance because a few slow requests can skew the result. For instance, if most requests are fast but a few are extremely delayed, the average may appear acceptable while the user experience remains poor.

A better metric is percentiles. When requests are sorted by latency over a time interval—like one minute—the latency at the middle point is the 50th percentile (p50). This means 50% of requests had latency no greater than that value. For example, p50 of 70 milliseconds indicates 50% of requests completed in 70ms or less.

In practice, engineers focus on high percentiles like p99. If p99 is 3 seconds, then 1% of requests took 3 seconds or more. These represent the worst-case user experiences.

Our goal is to investigate high-latency requests, identify root causes, and reduce tail latencies. Reducing latency is essential—but how much?

In theory, latency should be as low as possible. In practice, diminishing returns set in. Further reductions require significant effort. Thus, we must optimize a chosen percentile—such as p99—to a realistic target. For example, target p9s at 1 second.

Once set, this target becomes a system promise. It is then communicated to clients as a performance guarantee.

This target is known as a slow. The latency value we commit to becomes the slow. This mechanism was introduced earlier in the discussion of availability.

Throughput is the rate at which a system processes operations. Examples include:

• The number of web requests processed per second
• The number of database queries handled per minute
• The number of network packets delivered successfully per hour

Higher throughput indicates better performance.

To increase throughput, two primary strategies exist:

1. Reduce operation latency

   • If each request takes less time to process, more requests can be handled in the same time period

2. Scale the system

   • More resources allow more operations to be performed

Real-world examples:

• Processing a large file: Split the file into chunks and assign each chunk to a separate worker machine. Process chunks in parallel, then aggregate results on a final machine. This is the core idea behind MapReduce.

• Handling high-volume message queues: If a single consumer cannot keep up with incoming messages, add more consumers. However, multiple consumers cannot typically process the same queue.

• Solution: Create multiple queues, distribute messages evenly, and assign one consumer per queue. This is a fundamental pattern in event stream processing.

• Increasing database write throughput: Deploy multiple database instances, each storing a subset of data. This enables parallel writes and improves overall write performance.

Each instance of distributed data storage is called a chart. Instead of writing to a single database instance, we distribute writes across multiple charts.

This approach—distributing writes across multiple charts—is the foundation of horizontal scaling for both SQL and NoSQL databases.

To increase read throughput, we often use an external cache. Data is replicated across several cache instances, allowing each replica to serve read requests. This is the principle behind horizontal scaling of reads in memory data stores.

Throughput refers to the rate of data transfer, typically measured in bits per second. Bandwidth is the maximum rate of data transfer across a given path.

Think of bandwidth as a tube and throughput as water flowing through it. When the tube is fully filled, throughput reaches maximum and equals bandwidth.

If the tube is only half full, throughput is half of bandwidth. Throughput is always limited by the available bandwidth.

To increase system throughput, especially for data transfer between servers, we may need to increase bandwidth. This often means using a larger network pipe—more bandwidth—to achieve higher throughput.

In network-heavy systems, increasing throughput often requires increasing bandwidth, not just adding more processing power.