Resilient internal DNS: redundant resolvers, serve-stale and a sinkhole

The problem

When internal DNS stops answering, authentication, file shares and consoles fail together.

The goal was a self-hosted resolver, authoritative for internal names and forwarding the rest, with four properties:

• Redundant nodes that all answer, so losing one is not an outage.
• Continued answers for known names when every upstream fails.
• Policy-based blocking that can never take an internal name with it.
• Latency that holds under concurrency, measured rather than assumed.

Edge filtering by DNS interception cannot tell an internal name from an external one, so it can break internal resolution. Only the resolver knows which names are internal, so filtering belongs there.

What was built

A resolver designed to run as redundant nodes that all answer.

• Authoritative and forwarding. Internal zones answer from memory. Other names are forwarded, optionally over TLS (RFC 7858), and cached; negative answers are cached per RFC 2308, bounded by the SOA minimum and a hard cap.
• Split horizon. Internal zones can carry names that also exist publicly; a separate scheduled check compares any such internal answer with public resolution so drift is visible.
• Serve-stale (RFC 8767). An expired NOERROR or NXDOMAIN entry is kept for a bounded period. The stale answer, with a short TTL, is used only when every upstream fails or none answers in time; after a failed refresh it is served at once for a short hold before upstream is retried. Never for local zones, blocked names or SERVFAIL.
• Fast failover. A failed upstream is marked down and skipped while a healthy one exists; one background probe decides when it returns.
• Anti-spoofing and exposure. Random transaction IDs upstream, EDNS cookies (RFC 7873) and reply validation before caching; a recursion ACL refuses off-network clients.

Design decisions

Where the block check sits

Each query is checked in this order: local zone, recursion ACL, blocklist, then forwarding, where the cache lives.

• After the zone check, so an internal name can never be blocked, whatever a fetched list says.
• After the ACL, so an off-network client gets REFUSED rather than an answer that reveals what is filtered.
• Before the cache, so a name cached before its block stops being served at once.

Matching works outward from the full name, allowlist first at each level, and stops one label short so no rule can block a top-level domain. Blocked names get NXDOMAIN for every record type by default, since clients notice inconsistent answers across types.

Remote lists are untrusted input

An update is refused if it parses to nothing, shrinks sharply from that source’s last count, or names an internal zone. A refused fetch does not keep its ETag, so one bad download cannot leave the source stuck behind a 304. Entries that collide with known external dependencies are reported for a decision rather than pre-allowed; only a handful of names are seeded into the allowlist, because an exception list nobody revisits is how blocking quietly stops working. Blocking is opt-in per deployment, after the list review above.

Change detection by version, not by clock

Zones rebuild into memory continuously, within a two-second freshness bound. A list of about 150,000 rows cannot. A version counter, bumped in the same transaction as the rows, is read once per cycle, and the sets rebuild only when it changes.

Evidence and limits

Finding the bottleneck by dose-response

Figures here are from lab hardware, not a deployed node. Before the in-memory snapshot, authoritative throughput fell roughly ninefold from one client to sixteen, while cached names held steady. Ablations cleared the per-datagram connection, WAL locking, database size, cache settings and the interpreter’s thread-switch interval.

The deciding test: SELECT count(*) over a zone’s rows is one SQLite step and ran 30,000 q/s, rising to 49,000 with four threads. Fetching the same rows takes one step per row, ran 1,000 q/s and collapsed under concurrency. Each step releases and re-acquires Python’s global interpreter lock, so cost tracks rows crossing the boundary, not I/O. The collapse worsened with zone size, from 1.2x to 7.7x.

After the fix, the answer path makes no SQLite calls. Measured on a representative lab dataset:

Measure	Result
Sixteen-client authoritative throughput	About 50x pre-fix, within 5% of one client
Sixteen clients under concurrent writes	Within 2% of the unloaded rate, zero errors
Worst case before a new record is served	1.94 s
Query-log trim lock hold (separate fix)	6.467 s to 0.196 s

Limits

• The first benchmarks measured only the loaded case. Profiling showed an idle resolver paying the rebuild on almost every query, so the rebuild moved to a background thread.
• A speed benchmark cannot see a wrong answer: one prototype returned SERVFAIL for every authoritative query and benchmarked well. An answer-pinning suite now gates performance changes.
• Three times, a suite passed because it exercised a path other than the production code path. The latest fixed suite runs queries from real handler threads and fails when the earlier design is restored.
• Serve-stale and failover evidence is from offline suites with fake and real clocks, plus the revert-proof harness.
• Blocking is network-wide or off; per-client scoping is out of scope.

Where it applies

Healthcare payers and regulated federal contractors. SC-22 asks for fault-tolerant name resolution with internal and external roles separated. A redundant design addresses the fault-tolerance half; role separation depends on where external-facing names are served. For an assessment package, the useful evidence is measured failure behaviour and a test record.

Manufacturing and industrial suppliers. Plant networks depend on internal names for HMIs, historians and licence servers, and edge interception can break them. Resolver-side filtering, internal zones first, answers the questionnaire item on DNS filtering and resilience.

Fractional security leadership. The method transfers: pin correct behaviour, rule out suspects by ablation, confirm the cause by dose-response, and record what each test cannot see.

Independent work by Wade Blackwell. Each write-up is written from scratch for this site, without client, employer or personal data. Names, hostnames, addresses and configuration detail are left out on purpose.