DNS Optimization: Half the Build Time and 85% Memory Savings

In this article, we'll explore how we significantly reduced the time required to apply changes to DNS records while dramatically decreasing memory consumption. The key was transitioning from complete builds to targeted updates.

DNS Optimization: Half the Build Time and 85% Memory Savings

The increasing number of events in the DNS system necessitated optimizing the propagation chain. However, the real problem was not the volume of events, but the complexity of working with enormous zones, which today contain tens of millions of records and are still growing in size.

Every change triggers a process called "build," which converts the current database state into a format usable by our authoritative DNS servers and records changes in the distributed storage. Previously, partial record updates were significantly faster than a complete zone build. This led us to aim to apply a similar principle to zone setting changes as well.

Although it is logical to update only changed zone settings, determining when such an approach is safe and consistent presented the greatest challenge. While individual record changes clearly identify what was modified, events related to zone settings are more complex. They can originate from various internal systems, often do not specify the exact nature of the change, and can affect data shared by multiple zones with the same domain name. An incorrect choice of work scope would lead to either inefficiency or outdated DNS data.

We developed a targeted build scheduler that selects the smallest safe set of changes to perform, and a new unified planner that optimizes processing for the same zone. As a result, changes to zone settings no longer trigger a complete recalculation of millions of records. This led to an 85% reduction in peak memory consumption and allowed us to shorten waiting times across the entire propagation chain, resulting in half the time from a DNS change to its write to distributed storage.

A DNS zone represents a summary of records and settings for a given domain. For their global distribution, we use a multi-stage propagation process. A change, initiated by a customer or an internal system, is first stored in the database. From there, an event is generated, which the DNS Aggregator combines with other events and passes to the Zone Builder. The Zone Builder then reads the necessary zone data, converts it into a format for authoritative DNS servers, compares it with the distributed storage, and writes only the performed changes, which then propagate across the entire network. This process is simplified; the Zone Builder does not copy data directly. Instead, it combines records with zone settings, applies rules for DNS handling, and organizes the results so that authoritative DNS servers can use them efficiently.

The original Zone Builder was designed such that it did not need to know the reason for a zone change; it always completely rebuilt the desired state from the current database. The entire process involved loading the entire zone, including settings and all DNS records, converting them into keys for distributed storage, comparing them with existing data, deleting outdated and writing new keys, and updating zone metadata. Although only changed keys were written, the entire process still required loading, encoding, and comparing each record and holding the entire zone in memory. This was acceptable for small zones, but for the largest ones, with tens of millions of records, the zone would not even fit into memory.

Zone setting changes complicated targeted updates. An example is enabling a feature that updates metadata and can affect records at the root domain, such as CAA records. While a complete build only needed the information that the zone had changed, a targeted build had to answer more complex questions: Which DNS names might change? Could shared metadata or the list of published zones change? What previous state must be removed from distributed storage?

To safely answer these questions, five main problems needed to be solved: events had to specify the change, each change had to be mapped to the smallest correct build, the system had to account for shared data for zones with the same name, the database could change between the event and the build, and combining work must not lead to loss of messages from the event stream.

We modified event generators to include information about changed fields and settings, including previous and new values. We performed a detailed analysis of how each setting affects DNS output. Not all events were this specific. Some database functions only update a timestamp, indicating "something in this zone has changed." Until we have precise identification of the caller and its purpose, such timestamp updates still trigger a complete build. The same rule applies to missing details or any changes that could affect records anywhere in the zone – the scheduler never reduces the scope of work based on uncertainty.

The output from distributed storage has two key parts: record keys, containing DNS responses for a specific zone and name, and a smaller shared metadata key, which informs authoritative DNS servers about zone usage, including special records. The scheduler leverages this distinction. It can choose one of five build types: "Resources Only" (reconstructing records at specific DNS names), "Resources and Metadata" (reconstructing specific names and shared metadata), "Apex and Metadata" (reconstructing the root domain and metadata), "Purge" (removing a zone that is no longer to be published), and "Full Zone" (a complete rebuild when smaller work cannot be safely proven).

Dominik Medal

Got an idea for a website or app?

Let's talk it through — non-binding, with no pressure, and a concrete next step.

The principle is simple: every key written by a targeted build must be bit-for-bit identical to what a complete build would write. If this cannot be proven, a complete build is performed. Deleting the old state works similarly to before: the Zone Builder searches distributed storage for existing keys under each name it is rebuilding and removes those not written by the new build. Complete builds remain necessary when a change could affect records anywhere in the zone, or when a zone is published for the first time.

However, there is a deliberate exception for the largest zones. Above a certain record count threshold, the Zone Builder never escalates to a complete build. Instead, it performs a reconstruction of the root domain, metadata, and all names identified in the event queue. This compromise is accepted because a complete build of these enormous zones would not complete and would block the processing of all other changes on its partition in the event stream.

Distributed storage stores record keys for each zone separately. However, there is one shared metadata key for all zones with an identical domain name. This shared key acts as a directory and informs authoritative DNS servers which internal zone IDs they can use when answering a query. The reason one domain can have several internal zone IDs is the onboarding, offboarding, and other customer lifecycle processes, where multiple internal zone objects can temporarily exist for the same public domain. For example, changing a TXT record in one of the internal zones will only affect the record keys of that specific zone. Changes to nameservers or wildcard records are different because information about them is stored in the metadata key. Such builds must therefore update both zone records and the metadata key. When launched, the Zone Builder reloads the current list. If it is writing a metadata key, it rebuilds the entire key from this fresh perspective instead of patching an older copy.

An event describes a past change, but a build may wait while further events arrive and the database changes. Therefore, planning occurs twice. First, an event creates a conservative work list. Immediately before performing the build, the Zone Builder loads a fresh publication snapshot: the current zone settings, whether the zone is to be published to its network, and which zone IDs with the same name belong to the metadata key. The fresh state can change the outcome in four ways: a targeted build might become a complete build if the zone now requires full-zone work; a build might change to a purge if the zone is no longer to be published; a build might be skipped if the triggering zone no longer belongs to the metadata key; or a metadata build might reconstruct the shared key from the current set of candidates.

Loading the fresh state occurs from read-only database replicas to prevent collisions with customer write operations. Message delivery and database replication happen independently: an event might arrive before its database change is visible on the replica. The DNS Aggregator therefore delays events long enough to cover normal replication lag, and a final database check will refuse reads if the replica is still behind.

Previously, work with individual records and complete zones was handled by separate schedulers. We replaced these with a single unified planner. The planner now decides what needs to be built, and the scheduler then determines when it will be executed. If several events for one zone arrive before its build begins, the planner combines their work. For example, a change to an A record and a wildcard record for the same domain will be merged into a single operation, encompassing both names and metadata updates. Merging only adds work. If any event requires a complete zone, the merged build becomes a complete build. A purge operation overrides any reconstruction, as its goal is to stop publishing the zone.

The planner separately tracks DNS work and messages awaiting acknowledgment. When an operation is retried, both these components are preserved. Even if a complete build or purge no longer requires individual record names, messages remain attached until the result is safely written. A slow build can block other customers, even if it doesn't concern them. The system divides messages into ordered partitions and maintains one acknowledged offset for each. We use "soft committing." A build can complete out of order and be marked as finished in memory, but the acknowledged position only advances after all earlier messages have completed. If the Zone Builder restarts, unfinished messages are replayed. We consciously choose repeated work over losing a change.

Targeted plans reduce the time the oldest message remains unfinished. The scheduler also prevents one busy zone from consuming all work resources. After a zone build, the next build waits for a reconstruction interval, which varies by service type. Changes that arrive in the interim are merged into the next build. The scheduler usually adheres to this interval but allows old work in the queue to run before blocking a partition for too long.

To ensure safety and correctness, we implemented equivalence tests to compare the outputs of complete and targeted builds, including bit-for-bit key comparisons. We also tested the functionality of the planner and scheduler. During rollout, we concurrently ran a complete build alongside each targeted build and compared their outputs. Furthermore, we used a mirror instance of the Zone Builder. This consumes the production event stream, reads the production database state, but writes to a separate distributed storage that does not serve customer traffic.

A comparator checked a 20% sample of zones for missing, superfluous, or differing keys. All results were visualized on a dedicated dashboard. The rollout of the new scheduler was gradual, dividing zones by the hash of their name. This ensured that zones with the same name were processed by the same scheduler, preventing conflicts when writing shared metadata.

Within 24 hours, 92.95% of builds avoided a complete rebuild of the entire zone. Only 7.04% of builds were complete. The number of complete builds decreased by approximately 40%. The remaining complete builds took an average of only 0.35 seconds, as they mostly concerned new, small zones. The unified planner eliminated the need for a batching system. Message consumption from the event stream now closely mirrors their production, instead of previously lagging and catching up during peak times.

Build completion time, measured from a DNS record or zone setting change to its write into distributed storage, showed significant improvement. The peak p99 value, i.e., the time within which 99% of events are completed, dropped from 14.6 minutes to 1.21 minutes, a twelve-fold reduction. The median time decreased from approximately 7.7 seconds to roughly 3.3 seconds. Aggregate memory consumption, actively used by the Zone Builder deployment, dropped from an average of 49 GiB to 7 GiB, representing an 86% reduction. Peak consumption decreased from 79 GiB to 12 GiB, or 85%. These results also allowed us to shorten the zone reconstruction interval and the event delay time in the DNS Aggregator.

It turned out that completely eliminating full builds is neither realistic nor practical. A zone published for the first time requires writing all records. Changes that could affect records anywhere in the zone must be fully reviewed. An event that does not specify the nature of the change still leads to a complete build because the scheduler never reduces the scope of work based on uncertainty. Furthermore, for small zones, a complete build is still inexpensive and fast. We will continue to move other event types to targeted builds. However, the largest benefits have already been realized – events that previously triggered complete rebuilds of our largest zones no longer do so, and the remaining complete builds mostly concern events like initial publication, which affect new and small zones.

Dominik Medal

Let's talk about your project

Tell me what you need and together we will work out the best way forward.

Stop scrolling, call me

+420 735 505 585