Evolve Infrastructure

Problem

A single self-hosted environment was carrying the operational weight of several internal platforms with no formal backup discipline, no tested recovery process, and no documented answer to "what happens if this dies." Backups that have never been restored are not really backups, and infrastructure that's understood only in one person's head is a single point of failure independent of any hardware. The gap wasn't lack of scripts — it was the absence of governance: a place where recovery objectives, known risks, and what's protected against (versus knowingly not protected against) are written down, scored, and revisited rather than assumed. The failure modes that mattered most weren't hypothetical either: a nightly backup job had been silently failing for several nights running before anyone checked, a live production database was being pointed at the wrong path so it was never actually captured, and a set of containers were configured to stay down instead of recovering after a reboot — none of it visible until someone went looking for it on purpose rather than trusting that "there's a cron job for that" meant it was working.

What was built

A backup and recovery discipline covering the full operational footprint: nightly encrypted backups to offsite object storage, validated end-to-end through an actual restore drill rather than trusted on faith, plus a weekly automated integrity check that reads back a sample of stored data to catch silent corruption before it's needed. Recovery is documented as a runbook with concrete target timeframes for different failure scenarios rather than a vague "we'd figure it out," and every container-style service is configured to survive an unexpected restart automatically. A standing risk register scores active risks by likelihood and impact, assigns an owner, and gets reviewed on a quarterly cadence plus after any material change, with resolved risks tracked separately rather than silently dropped. A destructive-command guardrail blocks a defined set of high-confidence dangerous patterns before they run, and every push to the infrastructure repo is scanned for accidentally committed secrets before it merges.

Technical approach

The design principle is defense in depth applied to operational discipline, not just to the systems it protects: backup coverage, restart policy, and secret scanning are each independent controls, so no single failure mode (a bad deploy, a mistyped command, a leaked credential) cascades into total loss. The recovery process is explicitly rated by recovery time and recovery point objectives per scenario, and the single acknowledged unrecoverable failure mode — losing the backup encryption passphrase before it's mirrored somewhere independent of the primary host — is called out directly in every relevant document rather than left as an unstated assumption, which is the kind of gap that usually only gets discovered during the actual incident. The risk register distinguishes between risks that are mitigated, risks that are accepted at the current scale with a documented reason, and risks still open pending an owner action, which keeps the register honest rather than becoming a list everything eventually gets marked "handled." A cost baseline is tracked as its own artifact, so the actual monthly spend to run this is a known number rather than something inferred after the fact from a bill.

Each active risk is scored on two independent axes — likelihood, ranging from "possible but unlikely without a specific change" up to "realistically expected this quarter," and impact, ranging from "annoying but operational" up to "multi-day outage, data loss, legal exposure, or material revenue impact" — with critical-impact items ranked above everything else even when their likelihood is low, on the reasoning that a single bad event in that tier is irrecoverable regardless of how rare it is. The database-restore path is a concrete example of the discipline paying for itself: a real drill against the production database surfaced structural corruption (circular foreign keys and orphaned rows) that a naive single-pass restore couldn't tolerate, so the documented recovery procedure now runs as three separate passes — globals, schema, then data — specifically because the drill, not a design review, found the failure mode.

Creative approach

Craft

The documentation is deliberately layered by audience rather than written once and reused: a short executive overview answers "what would break and how fast does it recover" for a non-technical reader in under two minutes, while a separate security document, a technical threat model, a data-classification and third-party-processor record, and a step-by-step disaster-recovery runbook each exist for a specific reader — a board member, an auditor, privacy or legal counsel, an insurance underwriter, a vendor doing diligence, a new engineer coming in cold. The overview itself even routes each of those readers to the specific document meant for them rather than assuming one document can serve all of them. That's the same underlying discipline presented at as many altitudes as there are audiences, not one document straining to answer every question at once.

Reframe

The reframing insight is that backup and recovery are a governance problem before they're a tooling problem. A working backup script proves nothing about whether recovery actually works; only a real restore drill does, and only a written, dated risk register with an owner per item keeps known gaps from quietly becoming permanent blind spots. Treating the infrastructure repo itself as the source of truth — rather than tribal knowledge — is what makes a single-operator setup survivable if that operator is ever unavailable. The register also draws a line most informal setups never draw: a risk that's accepted with a documented reason is treated as a different category from a risk that's simply unaddressed. An accepted risk carries a written "why we accept it" — a cost-benefit judgment made once and revisited on a cadence — rather than being indistinguishable, in hindsight, from a risk nobody ever looked at. That distinction is what lets a single operator say, honestly, which gaps are a considered bet and which are just untouched.

Process and what failed

The risk register shows real, dated closures rather than a static wish list — several identified risks (an over-scoped access credential, a missing storage protection feature, a disabled system firewall) were found, tracked, and explicitly closed on a specific date rather than left open indefinitely, which is the evidence that the register gets acted on rather than just maintained as documentation theater. Other risks are knowingly left open with a documented reason (a small hardware gap around power-loss protection, a rotation deadline calendared rather than urgently fixed), which reflects a considered judgment about where effort is and isn't warranted at the current scale rather than an oversight.

Outcome

The backup and recovery discipline is live and running on an automated nightly cadence with a weekly integrity check, and has been validated end-to-end through a real restore drill rather than only tested in theory — including proving out the three-pass database restore procedure the drill itself forced into existence. The risk register is actively maintained with a quarterly review cadence, dated risk closures, and clearly owned open items; the closed-risk table alone shows a silently failing backup job, a database that wasn't actually being captured, and container restart policies that wouldn't have survived a reboot, each found, fixed, and dated rather than left as an assumption. The audience-layered documentation set — executive overview, security doc, threat model, data-classification and subprocessor record, disaster-recovery runbook — is the artifact a board member, an auditor, or an insurance underwriter would actually be handed today, not a hypothetical future deliverable. This is standing operational infrastructure in continuous use, not a one-time setup exercise.