Operations monitoring with recovery that is tested, not assumed

The problem

Monitoring systems are easy to leave untested: backups that are never restored, advisory boards that match on vendor alone, failover that is drawn but not exercised.

The system described here is a network and security operations monitor with a recovery standby. The rebuilt system follows one rule: every claim it makes about its own state (“this device is affected”, “the standby is current”, “the backup restores”) must be measured, and anything unmeasured must say so.

What was built

• Asset-matched advisories. Public vulnerability data, pulled on a schedule and on demand, enriched with the CISA Known Exploited Vulnerabilities catalog (including its ransomware flag) and EPSS scores, matched against versions read from each monitored element.
• Active and standby roles. Role changes are automated and run one at a time.
• A warm standby. A replication follower with fail-closed promote and build guards keeps a ready-to-open copy of each database on the standby, so takeover can skip a full restore.
• A scheduled test restore. The standby restores every replicated database to scratch space, runs an integrity check and pushes the verdict to the system’s status page.
• A retention map. Every mechanism that ages out, archives or deletes data, with the value it had when the map was compiled from the running system.

Design decisions

Unknown is neither “affected” nor “safe”

The first matcher matched on vendor alone and counted unknown versions as hits. Every red row that was checked against live measurements turned out to be already patched or not applicable. The rebuilt matcher has three verdicts: affected needs a measured version inside the range; not affected means measured and ruled out; unverified carries a written reason and never counts as a hit.

A measured version less precise than the advisory’s bound is unverified. Operating-system packages take their verdict from the host’s own package security state, because distributions backport fixes without changing the upstream version number.

The standby fails closed

• The follower applies only complete, verified changes in order, and its restore stops at any gap instead of restoring past missing data.
• Building a warm copy requires an explicit allow from a guard. A timeout, a guard that cannot run, or any answer other than an explicit allow counts against the build. A promotion stops a running build, and an aborted build removes its partial copy, leaving the previous copy untouched.
• An integrity check condemns a warm copy only on an actual verdict of corruption.
• Lag is null when not measured, never zero. While polls fail, reported lag grows, so the lag alert fires instead of a fresh timestamp sitting over a frozen status.
• Takeover uses the fastest path only when it is safe to, and otherwise the slower full restore.

Archive before delete, verify before replace

Tiers kept for history are written to read-only archive shards before hot rows are deleted; short-lived raw tiers are deleted outright, and the retention map says which is which. Export tools verify copied data before replacing a stored file and never replace a longer file with a shorter one. They never delete anything.

Evidence and limits

Each exercise found something:

Exercise	Found	Fixed by
First scheduled test restore	Ran out of disk during replay; a real promotion restores the same way and could have failed	More disk and a bounded replay size; re-run passed
Early failback	Overlapping role-change steps corrupted a database; recovered from the pre-promotion copy, minutes of data lost	Role changes serialized, with a safe fallback
Planned role swap	Role-change automation was silently skipped; the role moved but the database did not promote	A post-deploy check now verifies that role-change automation is armed

Other measured points:

• Against the previous matcher, affected rows fell by more than 99%, with every remaining row tied to a measured version.
• The follower carries 62 test functions (integrity checks, crash re-apply, partial writes, restore gaps) and a lab harness that kills it mid-apply.

Limits:

• Claims are limited to what each exercise measured; recovery-time figures are not published.
• Package-state verdicts can only cover flaws for which the distribution has published a fix; that boundary is shown with the verdict rather than hidden.

Where it applies

• Federal and health-sector authorization packages (FedRAMP, NIST 800-53). Contingency testing and backup (CP-4, CP-9) are often evidenced by a policy and a schedule. A dated, machine-checked restore verdict that degrades health status when it fails or goes stale shows the control operating. For RA-5, an “affected” row that names the measured version and its source answers the follow-up question in advance.
• Manufacturing suppliers (IEC 62443, SEMI E187, NIST 800-171). Equipment versions are often unknown. “Unverified, with a reason” is a defensible questionnaire answer; an unmeasured “not affected” is not. Backup and recovery (SR 7.3, whose first enhancement asks for backup integrity verification, and SR 7.4) are best evidenced by a restore that has actually been run.
• Fractional security leadership. A practical first review: map where data ages out, then restore something and record what breaks.

Independent work by Wade Blackwell. Each write-up is written from scratch for this site, without client, employer or personal data. Names, hostnames, addresses and configuration detail are left out on purpose.