Problems in Distributed Messaging Systems
Distributed systems face a wide range of challenges, not just in messaging, but across all layers.
These problems include transient network issues, long-lasting network partitions, and consumer application failures.
The broker must ensure messages are delivered to consumers.
If delivery fails at the moment, the broker retains the message temporarily.
Key topics include timeouts, retries, idempotency, backoff with jitter, failover, fallback, message delivery guarantees, and consumer offsets.
Reliable delivery alone is insufficient.
Speed matters.
If messages arrive faster than consumers can process them, they accumulate on the broker.
This accumulation can fill the broker’s local disk.
Bursts in message volume or gradual traffic growth can trigger this.
Even with stable message rates, performance may degrade.
For example, message size increases, requiring more processing resources.
To address this, systems must either reduce processing latency or increase throughput—both are essential.
Topics include batching, compression, horizontal and vertical scaling, autoscaling, and partitioning.
Both low latency and high throughput are critical.
However, brokers must avoid overwhelming consumers in push models.
In pull models, multiple consumers may overload the broker with polling requests.
Achieving effective flow control is difficult.
Brokers cannot reliably maintain a perfect message delivery rate.
It must not be too slow or too fast.
Design concepts help applications protect themselves.
These include load shedding and rate limiting.
Applications must defend against aggressive clients.
Clients must also protect themselves from application failures.
When a broker pushes messages to multiple consumers, and some fail or lag, the broker must not degrade.
It should continue sending messages to healthy consumers.
Design patterns such as circuit breaker and bulkhead support this.
These problems are common in system design interviews.
They are prevalent in distributed systems.
Consider the classic three-tier architecture.
The presentation tier displays data.
In web apps, this includes HTML, CSS, JavaScript, and frameworks like React or Angular.
The application tier, or business logic tier, runs core functionality.
It uses languages like Java, Python, or Ruby.
The data tier handles data persistence.
This includes SQL or NoSQL databases.
When a user makes a request, it travels through all tiers, from top to bottom.
The response returns upward.
What if the request fails between any two tiers?
What if one tier becomes slow?
What if traffic spikes dramatically?
The system may exceed its capacity.
It may not handle the load beyond a certain threshold.
Will the Entire System Crash?
A critical concern in distributed systems: whether a failure in one component will bring down the entire system.
How to Avoid System-Wide Failures
System design includes specific concepts to prevent cascading failures.
These concepts are not limited to messaging systems.
Broad Scope of System Design Problems
The questions about system reliability are fundamental and widely applicable.
There are numerous system design concepts developed to address these issues.
Coverage of Key Concepts
We will discuss many of these concepts in this section.
Later in the course, additional concepts will be introduced.
Universal Applicability
The messaging system serves as a demonstration model.
These principles apply to nearly all distributed systems.
Real-World Relevance
Consider the system you are currently working on.
Have you faced any of these failure scenarios?
Core Design Principles
System design concepts are used to solve both reliability and scalability challenges.
These concepts are essential for building robust distributed systems.
Both reliability and scalability are addressed through these principles.
The same design patterns apply across different system types.
Failure prevention is not a theoretical concern—it is a practical necessity in real-world deployments.
Questions
- What specific problems arise when messages are pushed or pulled in a distributed system, and how do they differ from messaging-specific issues?
- How can a broker ensure reliable message delivery when consumers may fail or experience transient network issues?
- What happens if messages arrive faster than consumers can process them, and what are the practical consequences for system performance?
- How do batching, compression, and scaling help address both latency and throughput in message consumption?
- In what ways might a push model overwhelm a consumer, and how might a pull model overwhelm a broker?
- What design patterns can help prevent a broker from degrading when some consumers fail or are slow?
Key takeaways
- Reliable message delivery requires handling transient network issues, long-lasting partitions, and consumer failures through mechanisms like retries and message persistence.
- Message accumulation on the broker can occur when incoming rates exceed processing capacity, leading to disk full errors or performance degradation.
- Speed matters not just in message delivery but in overall system performance, especially when message size or processing complexity increases over time.
- Flow control is difficult to achieve perfectly—brokers cannot reliably maintain a balanced message rate to consumers due to unpredictable workloads.
- Design concepts like load shedding and rate limiting help protect applications from aggressive clients and prevent cascading failures.
- Circuit breaker and bulkhead patterns allow systems to isolate failures and continue operating even when some components are unhealthy.
- The three-tier architecture illustrates how failures at any layer—presentation, application, or data—can impact end-to-end system behavior, highlighting the need for resilience at each level.