A backup job reporting success is a claim about writing, not about recovering. The way to convert that claim into knowledge is to rehearse the recovery on a schedule and time it honestly, which is a different exercise from writing a plan. This covers how to structure those rehearsals so they surface the gaps instead of confirming what you hoped.
Almost every organisation has a recovery plan and almost none has a number. Ask how long a full restore takes and the answer is an estimate from whoever wrote the document. A drill programme replaces that estimate with a measurement, and the gap between the two is usually the finding.
Four Drill Types, and What Each Proves
Treating "test the backups" as one activity is why most programmes stall. Different drills cost different amounts and prove different things, and a sensible calendar mixes them.
| Drill | Effort | What it proves |
|---|---|---|
| Integrity check | Automated, minutes | The archive is readable and complete |
| Single-item restore | Under an hour | One file or table can be recovered on request |
| Full system restore | Half a day | A whole service can be rebuilt, and how long it takes |
| Unannounced drill | Half a day, plus nerve | Whether anyone can do it without the person who wrote the plan |
The last row is the one that finds the real problems, and it is the one nobody schedules. A drill run by the engineer who built the backups, with their own notes open, tests the backups. It does not test the organisation.
The Only Number That Matters
Time the drill from the moment somebody says "go" to the moment the service is verified working. Not from when the restore command starts.
Those two figures diverge sharply, because the elapsed time is dominated by things nobody counts: locating the right backup, finding credentials, provisioning a target, waiting for a download, and the twenty minutes spent reading documentation that turned out to be stale. The restore command itself is often the fastest part.
Compare the measured figure against your stated recovery time objective. If the objective is four hours and the drill took nine, you do not have a four-hour objective; you have an aspiration. Either invest until the measurement matches, or change the number in the document to the truth. Both are defensible. Believing the wrong one is not.
Six Ways a Drill Gives False Assurance
A drill that passes for the wrong reason is worse than no drill, because it converts uncertainty into misplaced confidence.
Restoring onto the original host. The host still has the packages, the configuration, the SSH keys and the encryption key cached. Restore to a freshly provisioned machine or you have tested nothing about rebuilding.
Restoring with the key already present. Encrypted backups restore beautifully on a machine that holds the key. Do the drill on a host where you have to retrieve the key from wherever it is kept, because that retrieval is part of the recovery.
Checking the database and not the application. A dump can restore cleanly and still be inconsistent from the application's perspective, particularly if it was taken while writes were in flight. Log in, run a report, open a record with an attachment.
Restoring the newest backup only. Ransomware is discovered days after it arrives, so the backup you need is the one from before. Restore from a week ago at least once a year.
Using the runbook's author as the operator. They fill gaps from memory without noticing. Hand the document to somebody else and watch where they stop.
Announcing it a week ahead. Useful for a first drill. After that, the value is in finding what people cannot do without preparation.
A Calendar That Survives Contact With Work
Ambitious programmes get abandoned in the second quarter. This one does not.
Continuously: automated integrity verification of every backup, with an alert on failure. If your backup system can re-read and checksum stored data, that is this line, and it needs no human.
Monthly: one single-item restore, rotating through systems. Fifteen minutes, one person, logged.
Quarterly: one full system restore of a tier-one service, to a new host, timed. Rotate which service.
Annually: one unannounced drill, and one restore from a backup at least a week old.
Put these in the same calendar as everything else that must happen, with a named owner per instance. A drill that depends on somebody remembering is a drill that happens once.
Write Down What the Drill Found
The record is the deliverable, and it needs to be short enough that people actually produce it. Six lines: what was restored, from which backup, to what target, who ran it, how long from "go" to verified, and what went wrong.
That last field is the valuable one. Every drill produces at least one surprise: a document that named a decommissioned server, a credential nobody had, a dependency that was not in the backup at all. Fix each one and re-run the affected step. A drill whose findings are never actioned is theatre.
These records also happen to be exactly what an auditor asks for. Frameworks with continuity requirements do not want your plan; they want evidence that it was exercised, with dates. A drill log satisfies that in a way a policy document never will.
What the Drill Cannot Tell You
A restore drill proves you can recover from the backups you have. It says nothing about whether those backups exist for everything that matters.
So pair the drill with an inventory pass: walk the list of systems and ask what is backed up, how often, and to where. The systems that fail this are rarely the databases. They are the certificate authority somebody set up, the CI secrets, the DNS zone file, the config on a firewall, the one virtual machine that was never added to the schedule. Our guide to backup encryption and key recovery covers the key material category specifically, which is the one whose absence turns every other backup into noise.
The drill also cannot help if the backups are reachable from the compromised environment. That is a design property rather than a testing one, and our guide to immutable backups against ransomware covers the object lock and retention arrangements that address it.
The Infrastructure That Makes Drills Cheap
The reason quarterly full restores get skipped is the cost of a target to restore onto. When that means requisitioning hardware, it does not happen.
On per-resource pricing, a restore target is sized to the job and destroyed afterwards: MassiveGRID's Linux VPS is $2.87 per CPU core, $0.80 per GB of RAM and $0.01 per GB of SSD per month, so a 4 vCPU, 8 GB, 100 GB drill instance is $18.88 for a full month and a fraction of that for an afternoon. Backup services provide the block-level incremental backups with AES-256 encryption at $0.01 per GB, stored on RAID10 in Tier-3 and Tier-4 datacenters with destinations in the US and EU, plus two free VPS snapshots. Underneath, Proxmox high-availability clustering with automatic failover over Ceph storage replicating every block three times across independent NVMe drives handles the hardware failures that drills are not about, and NOC services can own the drill calendar where nobody internally will.
Restore targets can be ordered across a partner footprint of more than 700 datacenters in 85 metros, 30 countries and six continents, with auto-provisioning in New York, London, Frankfurt and Singapore, so a drill can prove recovery into a different region rather than the one that just failed.