What Kind of Database Should We Use?
When we think of a database, we typically imagine a dedicated server running a database system—either SQL or noSQL.
These databases are accessed remotely by client applications via network calls. The setup may involve a single server or multiple servers forming a distributed system.
Such databases are usually deployed outside the applications that use them.
Problems with Remote Databases in Messaging Systems
Remote databases introduce significant challenges for messaging systems.
First, latency.
High throughput requires processing messages as soon as they are produced and retrieved as quickly as possible.
Remote calls increase latency, directly reducing system throughput.
Second, failure modes.
The system must handle multiple failure scenarios:
- Database downtime
- Performance degradation during message bursts
- Network partitions
- Memory exhaustion on the broker, requiring immediate message storage
The application must manage how many database calls are made to avoid overwhelming the backend.
Third, cost.
At scale, databases become expensive.
One mitigation is to use a cache.
However, in messaging systems, messages are typically read once and then deleted.
This short lifespan makes caching ineffective for reducing database reads.
When Is a Remote Database Acceptable?
A remote database is acceptable only when:
- The system scale is low
- Throughput is not a critical concern
Why Embedded Databases Offer a Better Solution
If the database runs locally on the broker machine, it becomes an embedded database.
Embedded databases are tightly integrated with the application.
They are lightweight, use minimal resources, and eliminate network calls.
This results in faster message storage and retrieval.
Types of Embedded Databases
Embedded databases can be SQL-based, such as:
- Apache Derby
- SQLite
Or noSQL-based, such as:
- LevelDB
- ROGDB
They support multiple storage modes:
- In-memory
- On-disk
- Combined (in-memory and on-disk)
Real-World Usage of Embedded Databases
Multiple distributed systems use embedded databases as internal storage engines.
Embedded databases are present in everyday devices:
- Mobile phones
- IoT devices
- Sensors
- Thermostats
- Cameras
- Game consoles
- Cars
- Drones
- Robots
If your application manages many ad hoc files on disk, consider using an embedded database as an alternative.
Use Cases for Embedded Databases
Embedded databases support use cases such as:
- Data enrichment
- Real-time analytics
- Machine learning
These will be explored in detail later in the course.
Example: ROGDB
ROGDB is a popular embedded database used today.
It exemplifies the design principles of embedded databases.
We will now examine the key design principles behind ROGDB.
Questions
- Why might a remote database introduce latency in a messaging system?
- What are the failure modes that become more complex when using a remote database?
- How does the cost of a remote database scale with system size?
- Why is a cache ineffective for a messaging system that processes red messages?
- Under what conditions is using a remote database acceptable for a messaging system?
- What makes an embedded database different from a remote database in terms of operation?
- Which types of applications commonly use embedded databases today?
- What are the key design principles behind ROGDB, as mentioned in the lesson?
Key takeaways
- Remote databases introduce latency because they require network calls to store and retrieve messages.
- Remote databases create complex failure modes, such as network partitions and server downtime, that must be managed.
- Database cost increases significantly at scale, especially when handling high throughput and frequent access.
- Caching does not help in messaging systems because messages are typically red and deleted quickly.
- Using a remote database is acceptable only when scale is low and throughput is not a critical concern.
- An embedded database runs locally on the application machine, eliminating network calls and reducing latency.
- Embedded databases are lightweight, fast, and designed to integrate tightly with the application.
- They support various storage modes—such as in-memory, on-disk, or both—and can be used for local caching or internal storage.
- Embedded databases are used in mobile apps, IoT devices, and real-time analytics systems where local data management is essential.