Host discovery

To communicate, machines must discover each other. A consumer needs to connect to a broker, either directly or through a network path. The consumer must know the broker’s IP address or hostname.

If the broker’s name is a hostname, the consumer uses DNS to resolve it to an IP address.

DNS maintains a directory of domain names and maps them to IP addresses. It contains hundreds of millions of domain names and receives millions of lookup requests daily.

A single server cannot handle this volume. The phone book must be split into smaller pieces, each stored on a separate server.

A B-tree index helps split large datasets into manageable chunks and guides navigation. Each node provides a hint about where to go next during a search.

When a consumer looks up mail.google.com:

• The root server receives the domain name.
• It identifies the top-level domain (TLD), such as .com.
• It forwards the query to a TLD server.

The TLD server examines the domain name and identifies the second-level domain, like google.com.

It then directs the query to a next-level server in the chain.

Each server in the hierarchy reduces the data it must store.

• The TLD server holds only .com domains.
• The next server holds only subdomains under google.com.

At the final server, the domain name is resolved to an IP address, such as mail.google.com.

If a domain has many subdomains, the hierarchy can extend further.

Each level reduces the dataset size, allowing deeper resolution.

By adding more levels to the tree, the system can scale to store billions of domain names.

Despite the hierarchical design, the system faces a challenge: handling a large number of concurrent lookup queries.

The structure ensures data is partitioned and navigated efficiently, but query load remains a critical concern.

Each level in the hierarchy reduces the data set size, enabling scalability.

The root server acts as the entry point, routing queries based on top-level domains.

TLD servers act as intermediaries, routing based on second-level domains.

Final servers store only specific subdomains and return IP addresses.

This tree-based structure enables efficient, scalable domain name resolution across vast networks.

The design leverages B-tree navigation principles to guide queries through a layered system.

Each server in the chain handles a smaller subset of domain names.

The system scales by increasing the depth of the hierarchy.

The number of lookup queries remains high, requiring robust performance under load.

The structure ensures that no single server bears the full load of domain name resolution.

The hierarchy distributes the workload across multiple servers.

This design supports billions of domain names while maintaining efficient query resolution.

The root server does not store domain data—it only routes queries.

TLD servers do not store all domains—they store only those under a specific top-level domain.

Final servers store only the subdomains they serve and return IP addresses.

The system is built on a tree structure that splits domain names by domain level.

Each level reduces the data volume, enabling scalability.

The design is effective for large-scale distributed systems.

The challenge of high query volume remains unresolved in this part.

If all DNS lookup queries pass through a single root server, it creates a bottleneck and a single point of failure.

This risk is mitigated by distributing root server functionality across multiple instances.

Every node in the DNS hierarchy—root, TLD, and authoritative—represents a set of servers, not a single one.

Data is replicated across all servers in each set to ensure availability and consistency.

Replication enables horizontal scaling of lookup queries.

Caching reduces the load on servers by storing frequently accessed data.

DNS data changes infrequently, making it highly cacheable.

Caching occurs at multiple levels:

• Local DNS resolver in the operating system
• ISP DNS resolver server

These caches store results of previous lookups.

When a lookup is performed, the system checks the cache first.

If the result is not found, it proceeds to the next server in the chain.

A consumer machine requests the IP address for brokera.mydomain.com.

The request begins with the local DNS resolver in the OS.

If the IP is not cached, the local resolver forwards the request to the ISP’s DNS resolver.

The ISP DNS resolver checks its own cache.

If the IP is not found, it queries a root server.

The root server responds with a pointer to a TLD server.

The TLD server knows where to route requests for .com domains.

It forwards the request to an authoritative name server.

The authoritative name server holds full records for mydomain.com.

It returns the IP address of the broker machine.

It also returns a time-to-live (TTL) value, indicating when the result must be refreshed.

There are hundreds of root name servers globally.

They are divided into 13 sets.

Each set is operated by a different organization.

Each set has its own IP address.

Every ISP DNS resolver maintains a list of these 13 IP addresses.

Each IP address serves multiple servers behind it.

Requests are distributed using any CAST routing.

Any CAST allows multiple servers to share a single IP address.

Network routers select one server from the group based on routing metrics.

Metrics include proximity and cost.

For example, client A’s request may be routed to either root server A or root server B.

Selection is based on the lowest cost path.

This ensures load balancing and proximity optimization.

CacheN is used to offload root servers.

Web servers and local DNS resolvers cache DNS data.

As a result, root servers handle only a small fraction of all queries.

Redundancy, any CAST routing, and caching are the core mechanisms that enable DNS scalability and reliability.

A CAST (Cost-Aware Server Topology) enables efficient host discovery by evaluating path costs to root servers. When multiple servers share the same host identifier, the system selects the one with the lowest cost to the requesting client.

For Client B, requests are routed to Root Server A because it has the lowest path cost. This decision is based on the cost metric, not server identity.

CAST improves high availability. If a server fails, requests are automatically redirected to healthy, available servers within the network.

CAST supports load balancing when multiple servers announce identical cost values. This allows traffic distribution across servers with equal proximity.

CAST networks are widely used in DNS and content delivery networks.

CAST mitigates DDoS attacks by distributing traffic across multiple servers, preventing any single server from being overwhelmed.

DNS stores several record types. Three key ones are:

• A records: Point to an IP address.
• CNAME records: Map one domain name to another.
• TXT records: Store arbitrary text-based information.

These records enable DNS to manage host discovery, domain mapping, and metadata storage.