A recovery document states:
RPO:
15 minutes
RTO:
60 minutes
The numbers look precise.
They fit neatly into an architecture diagram.
They may have been copied into:
- a risk register
- a service profile
- a disaster-recovery plan
- an audit document
- a platform standard
But where did they come from?
Perhaps backups run every fifteen minutes.
Perhaps somebody estimated that restoring the database should take less than an hour.
Perhaps the numbers represent what the business needs.
Perhaps they represent what the architecture can actually deliver.
Those are not necessarily the same thing.
A restore drill turns the objectives from promises on paper into evidence.
RPO and RTO become real when a recovery exercise demonstrates which state can be restored and how long it takes to return a usable service.
Before the drill, the objectives describe intention.
After the drill, the organisation has measurements.
Those measurements may confirm the design.
They may also reveal that the system has been carrying a sixty-minute recovery promise on top of a three-hour recovery process.
Objectives and demonstrated results are different
The recovery point objective defines the maximum data loss the organisation has judged acceptable.
Suppose the target is:
RPO target:
15 minutes
If the database fails at 14:30, the recovery capability should be able to return to a safe point no earlier than:
14:15
14:15 14:30
│ │
├──── acceptable loss window ─┤
│ │
recovery point failure
The objective does not promise zero loss.
It defines the tolerated recovery-point gap.
The recovery time objective defines the maximum acceptable elapsed time from an agreed disruption boundary until a defined service outcome has been restored.
Suppose the target is:
RTO target:
60 minutes
The organisation must define both ends of that measurement.
START
failure, user impact,
detection or declaration
END
critical journey restored
or full service restored
Without those boundaries, two teams can report different recovery times and both be mathematically correct.
A drill produces different numbers:
RPO target:
15 minutes
Demonstrated recovery-point gap:
18 minutes
RTO target:
60 minutes
Demonstrated recovery time:
78 minutes
The targets describe what the service should provide.
The demonstrated results show what the tested recovery path achieved under a declared scenario. Both should remain visible.
A drill needs a declared scenario
A restore exercise should begin with a specific failure.
For example:
Scenario:
The bfstore production order database
has been deleted accidentally.
The application account remains trusted.
The backup account, primary region,
artefact registry, deployment metadata
and infrastructure source remain available.
A different scenario would produce different results:
production account compromised
primary region unavailable
database logically corrupted
encryption key inaccessible
Kubernetes control plane lost
One drill cannot demonstrate every recovery path. The scenario should identify:
- the initiating event
- the expected blast radius
- which systems remain available
- which recovery path should be used
- which objectives apply
- when timing begins
- which outcome stops the clock
Without a clear scenario, teams may quietly use whichever path makes the exercise easiest.
The recovery point is a complete, usable state
Backup frequency is only one input into the recovery-point objective.
Suppose bfstore creates a database backup every fifteen minutes.
It is tempting to conclude:
backup interval:
15 minutes
therefore RPO:
15 minutes
That assumes the newest backup is complete, readable, available, consistent, decryptable and compatible with the restore tooling.
During a drill, the team may discover that the latest copy cannot be used.
Perhaps it was still being transferred when the failure occurred.
Perhaps a required transaction-log segment is missing.
Perhaps corruption began before the backup was created.
The latest safe recovery point may be forty-five minutes old even though the backup process ran every fifteen minutes.
RPO target:
15 minutes
Demonstrated recovery-point gap:
45 minutes
The schedule describes how often a copy is attempted.
The drill reveals which recent state can actually be recovered.
Logical corruption may require an older point
The newest available backup may already contain the damage.
Suppose:
corrupt migration started:
09:47
latest clean recovery point:
09:45
incident detected:
10:30
The demonstrated recovery-point gap is forty-five minutes.
Backups may have run successfully after 09:47.
They preserved the corruption successfully too.
RPO must be assessed against the latest safe recovery point, not merely the newest stored copy.
The application must match the restored data
A service evolves over time.
Suppose the database is restored to a point before a schema migration.
The current application image expects the newer schema.
database:
schema v12
application:
expects schema v13
The database is readable, but the application cannot use it safely.
A complete recovery set may include:
RECOVERY POINT
│
├── database version
├── application image
├── schema version
├── configuration version
├── secrets and keys
└── deployment metadata
Immutable artefacts and deployment history matter during recovery.
A backup without the matching application and configuration can restore data into a service that no longer understands it.
Distributed state must be reconciled
The bfstore order service may depend on:
- MySQL state
- Kafka messages
- consumer offsets
- outbox records
- payment-provider outcomes
- inventory reservations
Suppose MySQL is restored to 10:00 while Kafka retains messages through 10:30.
MYSQL:
10:00
KAFKA:
10:30
The recovery process must decide what to replay, which events may duplicate, whether restored outbox rows will republish, whether consumers are idempotent and how external payments are reconciled.
The database recovery-point gap and the final business loss may not be identical.
RECOVERY-POINT GAP
state absent from
the restored system
PERMANENT BUSINESS LOSS
state that cannot be reconstructed
through replay, reconciliation
or an external source
Some customer actions may be reconstructed from event history or provider records. Others may require manual work.
A restore drill reveals whether the recovery point is coherent across the complete business workflow.
RTO measures the complete recovery path
A storage system may report:
snapshot restore completed:
18 minutes
That is useful.
It is not necessarily the service RTO.
The complete path may include:
detect incident
declare recovery
obtain emergency access
select recovery point
provision replacement infrastructure
restore data
apply grants and configuration
deploy the matching application
verify business behaviour
redirect traffic
SNAPSHOT RESTORE
│
▼
DATABASE READY
│
▼
APPLICATION READY
│
▼
BUSINESS JOURNEY READY
A process becoming technically alive is not the same as its service becoming operationally useful.
For a checkout service, recovery might mean a customer can complete a test purchase safely.
For an internal platform, it might mean a team can perform a successful infrastructure run.
Where partial service is acceptable, the objective should name the exact capability required, such as critical reads within thirty minutes and order creation within sixty. The clock stops only when that declared outcome is available.
The clock needs an agreed starting point
Possible starting points include:
when the failure occurs
when users are affected
when monitoring detects it
when the incident is declared
when the restore command starts
Suppose:
failure:
10:00
detected:
10:12
recovery declared:
10:25
restore started:
10:41
service verified:
11:18
Measured from restore start:
37 minutes
Measured from failure:
78 minutes
Both calculations are correct. Only one may match the agreed objective.
For user-facing recovery, an earlier boundary is usually more honest because detection and coordination are part of the operational capability.
A system the organisation cannot recognise as failed is not secretly recovering.
Access and coordination belong inside RTO
Recovery often requires privileged access.
The team may need:
- a backup restore role
- decryption-key access
- cloud administrator access
- emergency database credentials
- DNS or traffic-control permission
During a drill, the team may discover:
restore procedure:
32 minutes
access approval:
54 minutes
The access process dominates the recovery time.
That delay is part of the recovery system, not an administrative detail outside the architecture.
Emergency access should be protected, narrowly scoped, auditable, available during the expected failure and regularly tested.
A secure role nobody can activate during the incident protects the system from both attackers and recovery.
The same is true of human coordination.
A recovery plan may depend on:
incident commander
application owner
platform engineer
database operator
security responder
network operator
The drill may reveal that nobody can declare recovery, teams are waiting for each other, or no one owns final business verification.
The architecture may be automated, but recovery remains a socio-technical system.
Milestones reveal where time is lost
A single recovery-time number can hide the delay.
A useful drill records intermediate milestones:
| Milestone | Elapsed time |
|---|---|
| Incident detected | 8 minutes |
| Recovery declared | 19 minutes |
| Access obtained | 31 minutes |
| Recovery point selected | 38 minutes |
| Infrastructure ready | 54 minutes |
| Data restored | 71 minutes |
| Application started | 83 minutes |
| User journey verified | 96 minutes |
The RTO target was sixty minutes.
The demonstrated recovery time was ninety-six.
The table shows whether the largest delays come from restoration, detection, authority, decision-making or verification.
Without milestones, the team may optimise database copy speed while ignoring the hour spent finding the correct role.
Verification must use a real service outcome
Infrastructure checks are useful:
database accepts connections
Pods are ready
health endpoint responds
Kafka is reachable
They do not demonstrate that the business capability works.
For the order service, verification might include:
retrieve an order created
before the recovery point
confirm item totals
confirm payment reference
confirm order status
create a new test order
verify expected event publication
INFRASTRUCTURE READY
is necessary
USER JOURNEY READY
is the recovery outcome
The RTO clock should not stop because the first Pod became ready while every checkout request still fails.
Drills expose whether recovery is operable
A runbook may contain the expected recovery steps.
The drill tests whether those instructions remain accurate.
Common findings include:
command no longer valid
role name changed
backup location moved
required tool not installed
DNS step missing
verification query outdated
Every deviation should be recorded.
A successful recovery completed through undocumented improvisation still shows that the documented process cannot support another operator reliably.
The exercise may also expose hidden dependencies:
- one engineer’s laptop
- a package registry
- a container image no longer retained
- the failed identity provider
- DNS inside the unavailable region
- a secret stored only in the lost cluster
- automation that depends on the infrastructure being restored
RECOVERY TOOL
│
▼
DEPENDS ON FAILED SYSTEM
That circular dependency needs another route. Recovery forces the team to recreate the path rather than assume everything remains present.
Automation can reduce variance
Manual recovery introduces variation.
One operator may restore successfully in forty minutes.
Another may take two hours.
A recovery workflow might automate:
select backup
provision isolated database
restore data
apply grants
deploy matching application
run integrity tests
record timing
This reduces transcription errors, forgotten steps and dependence on individual memory.
Automation does not remove judgement.
The team must still choose the scenario, safe recovery point, whether to fail over, whether restored data is acceptable and when to return traffic.
The machine should perform repeatable work.
Humans should retain decisions whose context matters.
The drill itself needs guardrails
A restored production application might:
- send customer emails
- call a payment provider
- publish duplicate Kafka events
- alter live inventory
- register public DNS
- consume production traffic
The drill environment should isolate those effects.
RESTORED APPLICATION
│
├── email disabled
├── payment stubbed
├── Kafka isolated
├── public DNS absent
└── production writes blocked
The test should remain realistic while preventing unintended external actions.
Recovery testing is incomplete when the service is restored into a padded room that bears no resemblance to production.
It is also unwise to restore it directly into the ballroom and see whom it invoices.
Demonstrated results should sit beside the objectives
After the drill, the service record might show:
RPO target:
15 minutes
Demonstrated recovery-point gap:
18 minutes
RTO target:
60 minutes
Demonstrated recovery time:
96 minutes
The tested capability has missed both objectives.
The team can improve the recovery capability, change the architecture, automate slow steps, reduce access and coordination delays, or deliberately revise the objectives with the business.
The wrong option is to keep reporting the target as though it were achieved.
A missed objective does not automatically justify stricter process. If restore approval took forty minutes, another approval may make the next result worse.
Improvement should target the demonstrated cause.
Possible changes include pre-approved recovery authority, automated isolated environments, clearer incident roles, retained compatible artefacts, scripted verification and tested emergency credentials.
The drill should remove uncertainty and toil rather than growing more process around every recovery command.
Drills should increase in scope
A first exercise may restore one database into an isolated environment.
Later exercises can test wider scenarios.
Component restore
restore one database
Service recovery
restore database,
application and dependencies
Environment recovery
recreate the complete environment
Account-loss exercise
recover into a replacement account
Regional recovery
restore service outside
the primary region
SMALL DRILL
demonstrates mechanics
LARGER DRILL
demonstrates coordination,
dependencies and architecture
Confidence can grow progressively rather than depend on one annual exercise that spends much of the day rediscovering passwords.
Recovery objectives shape ordinary architecture
A drill may show that the RPO target requires:
- more frequent backups
- transaction-log retention
- durable event history
- idempotent replay
- stronger cross-system consistency
The RTO target may require:
- pre-provisioned capacity
- infrastructure automation
- warm standby systems
- simpler dependency chains
- retained artefacts
- faster traffic switching
RECOVERY OBJECTIVE
│
▼
ARCHITECTURE REQUIREMENT
An RTO of eight hours may support cold reconstruction.
An RTO of fifteen minutes probably cannot depend on rebuilding everything from an empty account through several manual approvals.
Recovery objectives are not labels added after the architecture is designed.
They place constraints on the architecture itself.
A bfstore order-service drill
Suppose bfstore defines:
RPO target:
15 minutes
RTO target:
60 minutes
The scenario is:
The production order database
has been deleted accidentally.
The backup account, primary region,
artefact registry, deployment metadata
and infrastructure source remain available.
The exercise records:
Failure:
10:00
Detected:
10:06
Recovery declared:
10:14
Latest usable recovery point:
09:42
Database restored:
10:47
Application ready:
11:02
Order journey verified:
11:18
The demonstrated recovery-point gap is:
18 minutes
The demonstrated recovery time is:
78 minutes
The exercise misses both objectives. Findings include:
- the 09:57 backup lacked a required log segment
- restore access took twelve minutes to arrange
- database grants were absent from the runbook
- the matching application image was difficult to identify
- business verification remained manual
Improvement work might include:
- validating transaction-log continuity automatically
- pre-authorising a protected restore role
- backing up grants and database configuration
- recording the image digest beside every backup point
- scripting the order-integrity checks
The next drill can show whether those changes close the gap. That is more valuable than changing the report to say:
Recovery successful.
Success without timing and data-loss evidence does not test the objectives.
The questions I now ask
Scenario
Which failure is being tested,
and what remains available?
Recovery point
What is the latest safe,
complete and compatible state?
Clock
When does recovery timing begin,
and which outcome stops it?
Recovery set
Which data, artefacts, configuration,
keys and dependencies are required?
Verification
Which user journey demonstrates
that the service is usable?
Measurement
What recovery-point gap and
recovery time did the drill demonstrate?
Improvement
Which architectural or operational
change will close the gap?
These questions turn a restore exercise into an engineering experiment rather than a ceremonial tour of the backup console.
The mental model I am keeping
My earlier model was:
RPO
backup frequency
RTO
estimated restore duration
The stronger model is:
FAILURE SCENARIO
│
▼
RECOVERY CLOCK STARTS
│
┌────────────────┴────────────────┐
│ │
▼ ▼
SELECT SAFE RECOVERY POINT RESTORE SERVICE
│ │
▼ ▼
RECOVERY-POINT GAP ELAPSED RECOVERY TIME
│ │
└────────────────┬────────────────┘
▼
VERIFY USER JOURNEY
│
▼
COMPARE RESULTS WITH TARGETS
│
┌────────────────┴────────────────┐
│ │
▼ ▼
TARGETS MET GAP DISCOVERED
│ │
└────────────────┬────────────────┘
▼
IMPROVE AND RETEST
RPO and RTO are valuable because they translate disruption into tolerable limits.
But a target does not create the capability required to meet it.
A fifteen-minute RPO needs recoverable state close enough to the incident.
A sixty-minute RTO needs detection, access, infrastructure, restoration and verification to fit inside one hour.
Only a drill brings all those dependencies into the same room.
A recovery objective becomes credible when a restore exercise measures the real recovery-point gap, the complete recovery duration and the user journey that demonstrates the service has returned.
Until then, RPO and RTO remain architectural intentions.
Useful intentions, certainly.
But still numbers waiting for reality to mark their homework.