A dashboard shows:
Backup job:
successful
Last run:
02:00
Objects copied:
418,732
Errors:
0
The status is green.
The storage system contains files with recent timestamps.
The organisation concludes:
Our data is backed up.
Then the production database is lost.
The recovery team begins restoring it and discovers:
- the backup is incomplete
- the encryption key is unavailable
- the recovery account lacks permission
- the database version has changed
- application configuration is missing
- nobody knows which recovery point is consistent
- the documented command no longer works
The backup process ran successfully.
The service cannot be recovered.
That distinction is the centre of this article:
A backup is a candidate recovery input. A recoverability contract defines the outcome it must support, and a recovery exercise provides evidence that the contract can be met.
Copying data is necessary.
It is not the outcome the organisation needs.
The actual outcome is:
After loss, corruption or compromise,
can the service be returned
to an acceptable state
within an acceptable time?
That question includes:
- data
- encryption keys
- configuration
- software versions
- infrastructure
- identity
- dependencies
- procedures
- people
- verification
A backup job proves that one step completed.
A recovery exercise proves whether the recovery system works.
Recovery is a service capability
Backup, restore and recovery are related but different.
BACKUP
preserved recovery input
RESTORE
recreated component state
RECOVERY
usable service returned
A database can be restored while the application remains unavailable.
An application can start while restored data is inconsistent.
A service can appear healthy while customers are missing recent orders.
The organisation does not really need archived bytes.
It needs the ability to restore a working outcome.
The capability should therefore be defined at the level of the service promise, not the backup tool.
Start with a recoverability contract
A recovery design becomes clearer when it records the scenario, outcome, recovery set and evidence in one place.
For example:
RECOVERABILITY CONTRACT
Service:
order-service
Scenario:
accidental database deletion
Required outcome:
order placement and retrieval restored
Target RPO:
15 minutes
Target RTO:
60 minutes
Recovery set:
database
schema and migrations
application artefact
configuration
keys and identities
event position
Required dependencies:
backup storage
cloud control plane
secret manager
DNS
recovery automation
Verification:
retrieve existing order
create new order
publish and consume order event
Last demonstrated:
2025-05-18
Observed result:
RPO 12 minutes
RTO 48 minutes
The contract answers:
Which service outcome
must return?
Which failure scenario
are we designing against?
How much data loss
is acceptable?
How long may recovery take?
Which complete set of state
must be restored?
Which user journey proves success?
The backup system is one part of that contract.
The contract is the actual promise.
Define RPO and RTO precisely
Two recovery objectives shape the design.
Recovery Point Objective
RPO describes the maximum acceptable amount of recent data loss.
RPO:
15 minutes
Backup frequency does not prove that RPO.
BACKUP FREQUENCY
how often copies are attempted
RECOVERABLE POINT
latest copy proven usable
OBSERVED RPO
distance between the incident
and the latest business-consistent
state successfully recovered
A snapshot timestamp may say 10:00.
If the latest state that passes business verification is 09:45, the observed recovery point is 09:45.
The practical RPO must account for:
- complete transactions
- usable log segments
- consistency across related systems
- event replay
- external side effects
- verification of the recovered state
Recovery Time Objective
RTO describes how long the service may remain unavailable.
The measurement boundary must be explicit.
FAILURE OCCURS
│
▼
DETECTION
│
▼
RECOVERY DECLARED
│
▼
TECHNICAL RESTORE
│
▼
BUSINESS VERIFICATION
│
▼
SERVICE RETURNED
Useful timing evidence separates:
detection time
decision time
recovery execution
validation time
A database restore may take twenty minutes.
The complete service recovery may take three hours.
Both measurements can be correct.
Only one describes the user’s experience.
Define the complete recovery set
A successful job may produce an incomplete recovery input.
A MySQL backup may omit:
- users and grants
- stored procedures
- configuration
- encryption material
- external binary objects
A Kubernetes backup may preserve Deployments and ConfigMaps while omitting:
- external databases
- persistent storage
- secret-manager content
- cloud load balancers
- identity-provider configuration
BACKUP TOOL SCOPE
│
▼
WHAT THE TOOL CAN SEE
│
╳
│
COMPLETE SERVICE STATE
The backup system may be working exactly as configured.
The configuration may describe an incomplete recovery set.
For bfstore order recovery, the set may include:
database state
schema and migration history
database users and grants
application artefact
runtime configuration
workload identity
network and DNS
Kafka offsets and outbox state
secrets and encryption keys
Related state must be time-consistent
Suppose:
database:
restored to 10:00
Kafka consumer offset:
restored to 10:20
The service may skip messages.
The opposite mismatch may replay messages that already affected the database.
Recovery therefore needs consistency groups, not merely individual copies.
The design should answer:
- which data stores must align
- which event positions correspond to the selected point
- which duplicates may occur
- which external side effects cannot be replayed
- which business state needs reconciliation
Distributed state turns recovery into a coordination problem.
Configuration, artefacts and infrastructure belong in the set
Version-controlled desired state helps recreate:
- application configuration
- infrastructure declarations
- policies
- deployment manifests
- migration definitions
- runbooks
But Git may contain references rather than live values.
Secrets, keys, state and external data still need separate protection.
GIT
describes what should exist
BACKUP
preserves valuable runtime state
CONTROL PLANE
recreates and reconciles resources
A complete design uses all three intentionally.
Infrastructure recovery may also require:
- remote state
- provider versions
- modules
- registries
- bootstrap resources
- scoped credentials
- an alternative path when the normal control plane is unavailable
A recovery path that depends entirely on the failed automation platform is not independent.
Know what each protection mechanism proves
Different mechanisms protect against different failures.
Job success and checksums
A successful job may prove:
- the source was read
- data was written
- transfer checksums matched
- the destination accepted the objects
Checksums protect integrity.
They do not prove completeness, consistency, compatibility or usability.
A perfectly preserved incomplete backup remains incomplete.
Replication
Replication creates another current copy.
REPLICATION
another current copy
for availability
It can protect against some infrastructure failures.
It may also reproduce logical corruption, destructive writes or compromised application behaviour.
Backup
A backup preserves a historical recovery point.
BACKUP
historical state
for restoration
It supports recovery from deletion or corruption when the selected point remains usable.
Archive
An archive serves a different purpose.
ARCHIVE
long-term retained information
for legal, historical
or analytical use
An archive may not support rapid service recovery.
A backup may support operational recovery without satisfying long-term record-retention requirements.
Snapshots
A crash-consistent snapshot may resemble sudden power loss.
It may not capture application-consistent business state.
Application consistency may require:
- flushing buffers
- coordinating volumes
- pausing writes
- database-native tools
- recording transaction positions
Point-in-time recovery
Point-in-time recovery can select a moment before corruption.
It still requires the team to understand:
- when the harmful action began
- what valid work occurred later
- how events will be replayed
- how external effects will be reconciled
- which application version can use the restored state
No one mechanism proves recoverability by itself.
Protect the recovery inputs and the path to use them
Backups often contain sensitive information.
They should be encrypted.
That creates a dependency:
ENCRYPTED BACKUP
│
▼
HISTORICAL KEY
│
▼
USABLE RESTORE
The recovery design should answer:
- where keys are stored
- who may use them
- how access is audited
- how historical keys survive rotation
- how the key system itself is recovered
Key recovery must be tested.
A runbook that says “decrypt the backup” is incomplete when nobody has proved that the recovery identity can access the correct key.
Recovery authority must be independent enough
The restore path may require access to:
- backup storage
- encryption keys
- cloud APIs
- databases
- clusters
- DNS
- secret managers
Ordinary production access may be unavailable or deliberately disabled during an incident.
The design needs narrow emergency authority that is:
- strongly authenticated
- auditable
- documented
- tested
- available without relying entirely on the failed system
An emergency role that has never been assumed successfully is not a recovery capability.
It is a theory with an IAM policy attached.
Separation must match the scenario
A backup stored in another account can reduce the blast radius of workload compromise or accidental production deletion.
It may still share:
- organisation administrators
- the same identity provider
- the same encryption authority
- the same region
- central automation
The recovery contract should name the scenario the boundary is intended to survive.
The backup must sit outside the expected blast radius of the failure it is intended to recover from.
A backup inside the blast is debris.
A backup beyond the blast is recovery.
Immutability needs policy and cost discipline
Retention-locked storage can protect against:
- accidental deletion
- malicious deletion
- ransomware
- compromised automation
- premature expiry
It also creates consequences for privacy, legal deletion and cost.
Immutability should follow the retention requirement and threat model.
It should not become a ceremonial padlock whose key is cost surprise.
Clean recovery may require a trusted environment
An isolated restore test protects production from accidental writes.
A cyber-recovery scenario needs more.
Restoring the latest state may recreate:
- compromised credentials
- malicious configuration
- poisoned artefacts
- vulnerable software
- persistence added by an attacker
A clean recovery environment may require:
- trusted infrastructure definitions
- known-good artefacts
- rotated credentials
- restricted connectivity
- integrity or malware checks
- investigation of when compromise began
- validation before traffic is restored
PROTECTED RECOVERY INPUTS
│
▼
CLEAN RESTORE ENVIRONMENT
│
▼
TRUST VALIDATION
│
▼
CONTROLLED RETURN TO SERVICE
Recovery is not only about restoring state.
It is also about deciding which state is still trustworthy.
Test at several levels
Not every test proves the same thing.
BACKUP VERIFICATION
Was data copied, retained
and protected as expected?
RESTORE VALIDATION
Can one backup be read,
decrypted and restored?
COMPONENT RECOVERY TEST
Can a database, cluster state
or application component recover?
SERVICE RECOVERY TEST
Can the complete service return
and pass business verification?
DISASTER-RECOVERY EXERCISE
Can people, authority, infrastructure,
dependencies and communication work
under a defined failure scenario?
A successful database restore does not prove application recovery.
A successful automated service restore does not prove incident coordination during a provider outage.
The levels are complementary.
Automation should test repeatable mechanics
A scheduled workflow may:
select a backup
create an isolated target
restore data
run integrity queries
start the application
execute business checks
record duration
destroy test resources
The automation itself needs guardrails:
- verify the destination is not production
- record source and target identities
- disable public DNS
- redirect event publication
- prevent customer notifications
- restrict outbound communication
- control cleanup
- preserve evidence
The recovery tool is a high-blast-radius control.
Rotate which recovery points are tested
Testing only the latest backup can miss:
- historical key problems
- old-format incompatibility
- broken incremental chains
- retention errors
- damaged older recovery points
A programme should sometimes select:
- the latest point
- a random recent point
- the oldest retained point expected to work
- a point requiring incremental or log replay
Test on a calendar and after material change
Cadence should reflect:
- service criticality
- change rate
- RPO and RTO
- compliance
- cost
Tests should also be triggered after changes such as:
- database-engine upgrades
- key rotation
- IAM redesign
- backup-agent changes
- schema changes
- infrastructure-module upgrades
- secret-manager migration
- control-plane changes
A quarterly test still leaves months of uncertainty if the recovery path broke yesterday.
Use recovery tiers
Not every service requires the same programme.
TIER 1
critical service
automated restore validation
frequent service recovery exercises
strict RPO and RTO evidence
TIER 2
important service
periodic component and
service recovery tests
TIER 3
recreatable or low-impact service
documented recovery
and sampled testing
The tier should follow business consequence, data importance and the cost of extended outage.
The goal is proportionate evidence, not identical ceremony.
Measure, record and learn
A recovery exercise should record:
- scenario
- selected recovery point
- start and end boundaries
- observed RPO
- observed RTO
- procedure followed
- deviations
- failed steps
- business verification
- owners
- improvement actions
For example:
Service:
order-service
Scenario:
accidental database deletion
Incident time:
10:00
Recovery declared:
10:03
Recovered business point:
09:48
Service verified:
10:51
Observed RPO:
12 minutes
Observed RTO:
48 minutes
Outcome:
passed with actions
A green result without detail teaches little.
The evidence should answer:
Did the contract hold?
What nearly failed?
Which manual steps slowed recovery?
Which dependency was missing?
What changed since the last exercise?
Failed exercises are valuable
A failed restore exercise is one of the safest times to discover that recovery does not work.
RESTORE TEST FAILED
│
▼
NO CUSTOMER INCIDENT
│
▼
KNOWN IMPROVEMENT WORK
The wrong response is to soften the test until it passes.
The right response is to repair the recovery system and repeat the exercise.
A failed test is evidence that the test was necessary.
Ownership and practice must survive staff change
A recovery plan should identify:
- incident commander
- service owner
- data owner
- platform owner
- security contact
- communication owner
Runbooks, diagrams and commands matter.
Practised operators matter more.
DOCUMENTED PROCESS
plus
PRACTISED OPERATORS
A backup that everybody assumes somebody else can restore has no practical owner.
The capability should survive holidays, role changes and inconvenient incidents at 03:17 on a Sunday.
A bfstore order-service recovery contract
For bfstore, the order service might use:
Service:
order-service
Tier:
1
Scenario:
production database deletion
Required outcome:
retrieve existing orders
create new orders
publish and consume order events
Target RPO:
15 minutes
Target RTO:
60 minutes
Recovery set:
MySQL data
schema and migrations
users and grants
application artefact
configuration
workload identity
secrets and keys
outbox state
Kafka positions
infrastructure state
Recovery boundary:
protected backup account
isolated recovery environment
Verification:
order total equals item totals
payment reference remains attached
confirmed orders contain items
new order IDs remain unique
duplicate event handling remains safe
A realistic exercise would:
- declare the recovery scenario
- record the timing boundary
- select a business-consistent recovery point
- create an isolated target
- restore data, users and grants
- recover configuration, keys and identity
- deploy the matching application artefact
- disable external side effects
- align outbox and Kafka positions
- run business invariants
- create a test order
- verify event processing
- calculate observed RPO and RTO
- record deviations and actions
- destroy the isolated environment
RECOVERABILITY CONTRACT
│
▼
PROTECTED RECOVERY SET
│
▼
ISOLATED RESTORE
│
▼
BUSINESS VERIFICATION
│
▼
OBSERVED RPO AND RTO
│
▼
IMPROVEMENT ACTIONS
No single backup system recreates the whole service.
Recovery depends on the complete set working together.
Questions I now ask
Outcome
Which service outcome and
failure scenario are protected?
Objectives
Which RPO and RTO
does the scenario require?
Recovery set
Which complete, time-consistent
state is needed?
Dependencies
Which keys, identities and
bootstrap services enable recovery?
Separation
Are the recovery inputs outside
the expected failure boundary?
Test level
Which exercise demonstrates
the required outcome?
Verification
Which business checks prove
the restored service is correct?
Evidence
When was the contract last demonstrated,
and what changed since?
A backup policy without answers to these questions describes storage activity more clearly than recovery capability.
The mental model I am keeping
My earlier model was:
BACKUP
copy important data somewhere safe
The stronger model is:
RECOVERABILITY CONTRACT
│
┌─────────────────┼─────────────────┐
│ │ │
▼ ▼ ▼
SCENARIO RPO RTO
│ │ │
└─────────────────┼─────────────────┘
▼
RECOVERY SET
│
┌─────────────────┼─────────────────┐
│ │ │
▼ ▼ ▼
DATA CONFIGURATION KEYS AND IDENTITY
│ │ │
└─────────────────┼─────────────────┘
▼
PROTECTED RECOVERY INPUTS
│
▼
ISOLATED RESTORE
│
▼
BUSINESS VERIFICATION
│
▼
OBSERVED RPO AND RTO
│
▼
LEARN AND IMPROVE
Backups are necessary because data can be lost, corrupted, encrypted, deleted or made unreachable.
But the backup itself is not the promise.
The promise is that the organisation can recover.
That requires selecting a usable recovery point, retrieving protected inputs, unlocking them, restoring compatible infrastructure, reconnecting the application and proving that important user journeys still work.
A backup becomes trustworthy only when a recovery exercise has shown that the data, the system and the organisation can find their way back together.
Until then, the organisation has copies.
It may even have excellent copies.
What it does not yet have is evidence of recovery.