What Is Batching?
Batching combines multiple requests into a single request to improve performance.
Why Batching Increases Throughput
Batching improves throughput by reducing the number of individual requests.
Each request consumes resources: HTTP headers, connections, memory, and threads.
Fewer requests mean:
- Fewer HTTP headers
- Fewer connections established
- Less memory allocation
- Fewer threads used (in thread-per-request models)
Cost and Throttling Benefits
Fewer requests reduce costs, especially in pay-per-request cloud services.
Lower request volume reduces throttling risk.
Services often limit request rates to prevent abuse.
Batching Introduces Complexity
Client-side complexity arises from accumulating messages.
Clients must:
- Store messages in a buffer
- Wait up to several seconds
- Send buffer content when buffer fills or timeout occurs
This adds implementation and configuration overhead.
Server-Side Challenges
Processing batched requests introduces ambiguity in error handling.
If a batch contains 10 messages and the 5th fails:
- Should processing continue?
- Should the entire batch succeed or fail?
- Should already processed messages be rolled back?
These decisions are non-trivial.
Two Server Handling Approaches
Option 1: Treat the batch as an atomic unit.
- All operations must succeed for the batch to succeed.
Option 2: Process operations independently.
- Report failures per message.
- Make progress on successful operations.
Option 2 is more common in practice.
Two Types of Batching
Option A: Batch multiple requests into one HTTP call.
- Concatenate individual requests with a separator.
- Each request includes its own headers and body.
Option B: Batch multiple resources into one call.
In Option A, the server processes each nested request and returns a single HTTP response.
The response includes status codes for each individual request.
This format allows clients to identify which operations succeeded or failed.
Batch Request Format in Google APIs
Many Google APIs support batching and follow a standardized batch request format.
Example: Google Drive Batch API
The Google Drive batch API demonstrates this pattern by allowing multiple operations in a single request.
SQM Message Batch API
SQM provides a send message batch API that submits multiple messages to a specified queue.
Instead of concatenating individual requests, the client provides a list of messages.
Batch Response Structure
The service returns a standard HTTP response containing a list of results.
Each result object includes:
- The message identifier
- The status of the operation
Status indicates whether the message was successfully enqueued.
Outcome of Batch Operations
A batch request can result in a mix of successful and unsuccessful operations.
Client Response Handling Options
Three strategies exist for handling partial responses:
-
Retry the entire batch request as is
This is valid when each nested operation is critical.
The server replays all operations.
Previously succeeded operations remain unchanged.
Failed operations get retried.
This option is easy to implement on the client side.
-
Retry each failed operation individually
-
Create a new batch request with only failed operations
The last two options require more client effort.
The client must:
- Detect individual operation failures
- Construct one or more new requests
These options do not require the batch API to be 'ad-important'.
Example: SQS Delete Message Batch API
SQS supports retrieving up to 10 messages per pull request.
The consumer is responsible for deleting successfully processed messages.
The delete message batch API accepts a list of message identifiers.
The response includes the deletion status for each message.
Failed deletions remain in the queue.
When visibility timeout expires, such messages are retrieved by other consumers.
The consumer must scan the response to identify deletion failures.
If any failure is found, it can apply one of the three response handling options.
Kafka and Batching
Kafka relies heavily on batching.
On the producer side, messages are batched before sending to brokers.
On the consumer side, messages are retrieved in batches.
Batching is one of the key factors enabling Kafka’s high throughput.
Questions
- When a batch request fails during processing, should the entire batch be rolled back or only the failed operation be retried? What determines which approach is correct?
- How does the server decide whether to treat a batch request as an atomic unit or as a collection of independent operations?
- If a client receives a partial success in a batch response, what are the practical implications of each retry strategy — retrying the whole batch, retrying individual operations, or creating a new batch with only failed messages?
- Can batching be effectively used in systems where every message must be processed with absolute reliability? Why or why not?
- What happens to messages that fail to be deleted in a batch delete operation — are they automatically removed or left in the queue?
- Does the use of batching in Kafka imply that it is only effective under specific conditions, or is it universally beneficial?
- How does the client-side buffer (e.g., memory buffer) affect the timing and reliability of batched requests?
- What are the potential risks of relying on a batch API that returns mixed success/failure results without clear error isolation?
Key takeaways
- Batching increases throughput and reduces cost by minimizing the number of requests sent to a service.
- Batching reduces overhead such as HTTP headers, connection establishment, and thread allocation per request.
- Batching introduces complexity on both client and server sides, especially in handling failures and retries.
- Server-side failure handling is ambiguous: it can either treat the batch as atomic or process operations independently.
- A batch request may contain both successful and failed operations, requiring the client to handle partial results.
- The client has three options for handling partial failures: retry the whole batch, retry individual operations, or re-submit only failed messages.
- In systems like SQS, the consumer must scan the batch response to identify failed deletions and take corrective action.
- Kafka uses batching on both producer and consumer sides, which contributes to its high throughput, but this does not guarantee reliability for every message.
- Batching is not universally optimal — its benefits must be weighed against implementation complexity and failure handling challenges.