Cloud Sandbox
All writing

DNS failover: where Route 53 failover routing earns its keep

Route 53 failover routing can redirect new DNS resolutions from a primary endpoint to a secondary endpoint. It is valuable when the secondary is independently usable, the health signal represents the service capability, and caching, data recovery, failback and network reachability have all been designed.

about 26 minutes min read

The bfstore production workloads send telemetry to:

otel.prod.aws.internal.bfstore.example
        │
        ▼
primary internal load balancer
        │
        ▼
OpenTelemetry ingestion gateway
        │
        ▼
self-hosted observability platform

The service is private. Its name lives in a Route 53 private hosted zone associated with the production VPC, and the network path is available through bfstore’s private routing architecture.

Then the primary observability environment suffers a serious failure.

bfstore could change DNS manually, restore the primary in place, accept telemetry loss, or direct clients to a separately prepared recovery service. Route 53 failover routing addresses that last option.

Two records share the same name and type:

otel.prod.aws.internal.bfstore.example

primary
    → primary observability endpoint

secondary
    → recovery observability endpoint

When Route 53 considers the primary healthy, it returns the primary record. When the primary is unhealthy and the secondary is healthy, it returns the secondary. Failover routing is Route 53’s active-passive policy and is supported in private hosted zones.

That is useful, but incomplete.

DNS failover can change the answer returned to a future query. It cannot create or warm the recovery environment, replicate data, establish routes, copy credentials, repair dependencies, move existing connections, flush cached answers or prove that the secondary can carry production load.

DNS failover redirects discovery. Disaster recovery makes the discovered destination worth reaching.

Route 53 earns its keep when both sides of that sentence have been built.

What Route 53 actually decides

Failover routing is active-passive

Route 53 supports several routing policies that can consider health. Failover routing specifically models an active-passive relationship.

NORMAL OPERATION

clients
   │
   ▼
primary endpoint


FAILOVER OPERATION

clients
   │
   ▼
secondary endpoint

Only the primary should normally be returned. The secondary waits until the primary is considered unhealthy.

Active-active designs use other health-aware policies, such as weighted or latency routing, where several healthy records can be eligible at once. Those environments receive ordinary traffic and are exercised continuously.

An active-passive secondary may receive little or no ordinary traffic. Its readiness must therefore be demonstrated by probes, drills and operational controls.

ACTIVE-PASSIVE

one destination is ordinary

one destination is a promise
about future readiness

Route 53 implements the selection. It does not prove that the promise is true.

Two records form one decision

A basic failover group contains one primary record and one secondary record with the same name and type, distinct identifiers, and health behaviour that lets Route 53 choose between them.

records:
  - name: otel.prod.aws.internal.bfstore.example
    type: A
    routing_policy: failover
    failover_role: primary
    identifier: primary-observability
    target: primary-observability-endpoint

  - name: otel.prod.aws.internal.bfstore.example
    type: A
    routing_policy: failover
    failover_role: secondary
    identifier: recovery-observability
    target: recovery-observability-endpoint

The pair can use alias or non-alias records, and one may be an alias while the other is not, provided the names and record types match.

IS PRIMARY HEALTHY?
        │
    ┌───┴───┐
   YES      NO
    │        │
    ▼        ▼
PRIMARY   IS SECONDARY HEALTHY?
              │
          ┌───┴───┐
         YES      NO
          │        │
          ▼        ▼
      SECONDARY  SPECIAL FAILURE BEHAVIOUR

That final branch is easy to overlook.

If both destinations are unhealthy, Route 53 returns the primary

If both failover records have health checks and both are considered unhealthy, Route 53 still returns the primary record.

PRIMARY UNHEALTHY

SECONDARY UNHEALTHY
        │
        ▼
ROUTE 53 RETURNS PRIMARY

Route 53 needs to return a DNS response and treats all records as healthy as a last resort. The failover policy then selects the primary.

The operational consequence is important:

Failover routing does not guarantee that every returned endpoint is healthy.

When all options are unhealthy, clients still need deadlines, bounded retries, circuit breaking and useful error handling. DNS is one selector in the reliability system, not its final safety mechanism.

A secondary with no health evaluation is considered healthy

If the primary becomes unhealthy and the secondary has neither an associated health check nor alias target-health evaluation, Route 53 considers the secondary healthy and returns it.

PRIMARY

health checked
unhealthy


SECONDARY

no health check
no target-health evaluation
therefore considered healthy
        │
        ▼
SECONDARY ANSWER RETURNED

An alias record is different when EvaluateTargetHealth is enabled: Route 53 inherits health from the alias target.

An unchecked secondary can be appropriate for a deliberately simple destination, such as a static maintenance page. It deserves more caution for a private database, payment service or telemetry pipeline that may be under-sized, stale, missing credentials or blocked by a route.

An unchecked secondary means:

If the primary fails,
send traffic here without asking
whether this destination can receive it.

That should be an explicit architectural decision, not an accidental omission from the Terraform.

How to create a meaningful health signal

Health must represent the capability being failed over

The key question is not:

What metric do we already have?

It is:

What must be true for this endpoint
to fulfil the service contract?

For observability ingestion, useful health may require the load balancer to accept connections, at least one gateway target to be ready, authentication to work, the queue to be writable and downstream storage to remain within safe capacity.

A weak signal such as:

EC2 instance is running

can stay green while the process has crashed, the disk is full, the certificate has expired or every request fails.

A stronger signal asks whether a production-shaped telemetry request can be accepted and durably handed to the next stage.

Health checks should be meaningful enough to represent the capability, cheap enough to run continually, independent enough not to amplify an outage and stable enough not to flap. The best check is the one that answers whether traffic should continue arriving.

Alias health can follow a load balancer

When a failover alias points to an Application Load Balancer or Network Load Balancer, EvaluateTargetHealth can make Route 53 inherit the health of that destination.

FAILOVER ALIAS RECORD
        │
        ▼
EVALUATE TARGET HEALTH
        │
        ▼
INTERNAL LOAD BALANCER
        │
        ▼
TARGET-GROUP HEALTH

For an ALB or NLB to be considered healthy, every target group that contains targets must contain at least one healthy target. A target group with no registered targets is considered unhealthy.

This works well when load-balancer target health accurately represents the DNS name’s service contract.

It can be too broad when one load balancer serves several capabilities. A target group unrelated to the specific DNS name can influence the alias health decision. The health boundary and the DNS name should describe the same service.

Highly available AWS endpoints may need an application health check

EvaluateTargetHealth is not equally useful for every alias target.

AWS notes that highly available services such as VPC interface endpoints, API Gateway, S3 and VPC Lattice may provide little operational benefit from alias target evaluation. Failover involving these services may need a Route 53 health check that represents the application’s ability to use them.

For a bfstore PrivateLink endpoint, the interface endpoint may remain infrastructurally healthy while the provider application is unavailable, authentication fails, storage cannot be read or every request returns an error.

INFRASTRUCTURE HEALTH

interface endpoint exists
network interfaces available


APPLICATION HEALTH

approved module can be retrieved
and authenticated successfully

Failover should normally follow the second when that is the service promise being protected.

Standard endpoint health checks cannot target private addresses

Route 53 endpoint health checkers operate outside the VPC and cannot target local, private, non-routable or multicast IP address ranges.

For private hosted-zone failover, a common alternative is a Route 53 health check based on a CloudWatch alarm data stream.

PRIVATE SERVICE
      │
      ▼
IN-VPC PROBE OR SERVICE METRIC
      │
      ▼
CLOUDWATCH ALARM
      │
      ▼
ROUTE 53 HEALTH CHECK
      │
      ▼
FAILOVER RECORD

A purpose-built probe runs where the private service is reachable, publishes a metric and lets CloudWatch evaluate it. The probe can include private routing, transport, authentication and dependency behaviour without exposing the service merely so external health checkers can reach it.

The CloudWatch alarm needs an explicit insufficient-data policy

A Route 53 health check based on a CloudWatch alarm monitors the alarm’s data stream rather than simply reading its current label.

The health check is healthy when the stream indicates OK, unhealthy when it indicates ALARM, and follows the configured insufficient-data behaviour when the stream lacks enough evidence:

  • healthy;
  • unhealthy;
  • last known status.

Route 53 does not support cross-account CloudWatch alarms for this relationship. The alarm must also use a standard-resolution metric. Metric math and “M out of N” alarms are not supported.

If several signals must be combined, publish a purpose-built readiness metric or use a calculated health check.

What should missing data mean?

HEALTHY

Assume the service is healthy
unless failure is proven.


UNHEALTHY

Assume missing evidence means
the service cannot be trusted.


LAST KNOWN

Continue using the previous state.

Healthy can keep traffic on an unobservable failure. Unhealthy can trigger failover because the monitoring path failed. Last known can preserve a stale decision. The choice should reflect the independence and reliability of the monitoring path.

Which account owns the alarm?

If the private hosted zone and health check live in bfstore-network while the metric lives in bfstore-observability or bfstore-prod, the design must account for the same-account requirement. Options include publishing a purpose-built metric into the health-check account, moving the health check and alarm with the workload, or revisiting which account owns the DNS decision.

Central ownership is useful only when the required signals can reach it cleanly.

Calculated health checks can express quorum

A calculated health check can monitor several child health checks and compare the number of healthy children with a threshold. The published Route 53 quota allows up to 255 child checks.

PRIMARY OBSERVABILITY HEALTH

child 1:
    primary-specific probe from production VPC

child 2:
    primary-specific probe from staging VPC

child 3:
    ingestion backlog below threshold

child 4:
    durable storage writable


parent healthy when:
    at least 3 children are healthy

This can reduce sensitivity to one faulty probe, but it can also create a formula nobody understands during an incident. Calculated health checks cannot monitor other calculated health checks, so the design must remain shallow.

Combine only independent evidence that genuinely determines whether the service can receive traffic, and make the formula explainable in a runbook.

What DNS cannot do

TTL determines how quickly new queries learn the decision

Route 53 may mark the primary unhealthy quickly, while resolvers and clients continue using a cached primary answer.

T0  primary fails
T1  health state changes
T2  Route 53 returns secondary
T3  cache expires
T4  client queries again
T5  client connects to secondary

Experienced failover time is closer to:

health detection
+ health-state evaluation
+ remaining DNS cache lifetime
+ application retry or reconnect
+ secondary connection establishment

For non-alias health-checked records, AWS recommends a TTL of 60 seconds or less when clients need to react quickly. Shorter TTLs increase query frequency, so the setting must match the recovery objective.

Alias records do not set their own TTL. Route 53 uses the TTL of the AWS resource or record that the alias targets.

Existing connections do not move

Suppose a Go service has an established gRPC connection to:

10.96.12.45

DNS later begins returning:

10.160.12.45

The existing TCP or HTTP/2 connection remains attached to the original destination until it fails, the client closes it, an idle timeout expires or the application refreshes it.

DNS failover redirects future discovery. It does not migrate live sessions.

Client behaviour therefore belongs in the recovery design:

  • connection deadlines;
  • bounded retries;
  • reconnect logic;
  • appropriate DNS refresh;
  • idempotent operations;
  • awareness of long-lived pools.

Telemetry collectors may keep persistent export connections. MySQL pools may retain sessions. Kafka clients may rely on broker metadata after the initial bootstrap lookup. The failover test must use the real client, not only dig.

Route 53 cannot repair a missing route

A workload may receive the secondary answer:

otel.prod.aws.internal.bfstore.example
    → 10.160.12.45

The production VPC still needs a route to the recovery network, a return route, security-group and firewall permission, the correct hosted-zone association and authentication accepted by the secondary.

DNS ANSWER
    │
    ▼
ROUTE TABLE
    │
    ▼
TRANSIT OR OTHER PRIVATE PATH
    │
    ▼
SECURITY CONTROLS
    │
    ▼
SECONDARY SERVICE

A correct failover record can return an unreachable address. Private recovery paths should exist and be tested before the incident. Creating them during the incident is emergency network construction, not automated failover.

The secondary must be able to carry the traffic

A cold standby may contain no running replicas, stale data, missing secrets, outdated infrastructure or untested network policy. DNS can still direct production traffic to it.

Before enabling a secondary failover record, bfstore should know:

  • Capacity: can it handle full production demand?
  • Data: is required state replicated, restored or deliberately absent?
  • Identity: will current workloads and clients be accepted?
  • Secrets: are certificates, tokens and keys available?
  • Dependencies: can it reach the services it needs?
  • Operations: are dashboards, alerts and runbooks active?
  • Ownership: who declares it ready?
DNS SECONDARY

must represent

RECOVERY CAPABILITY


not merely

ANOTHER ADDRESS

For stateful order processing, readiness may also require database promotion, write fencing, event-offset reconciliation and idempotency state. Route 53 should move traffic only after those concerns have been handled.

Multi-AZ load balancing usually does not need Route 53 failover

An internal ALB or NLB can already operate across Availability Zones and route traffic to healthy targets. An Application Load Balancer normally spans at least two enabled zones.

orders.prod.aws.internal.bfstore.example
        │
        ▼
internal ALB
        │
        ├── targets in AZ A
        ├── targets in AZ B
        └── targets in AZ C

Adding Route 53 failover between two load balancers that duplicate the same multi-AZ failure boundary often adds complexity without creating meaningful independence.

Route 53 is more useful between independent service cells:

PRIMARY CELL              SECONDARY CELL

load balancer             load balancer
compute                    compute
dependencies               dependencies

The boundary may be another Region, VPC, service cell, recovery stack or maintenance destination. Two records pointing into the same failure domain do not create redundancy.

Shared dependencies can defeat failover

Suppose primary and secondary order-service endpoints both depend on the same regional database, Kafka cluster, KMS key, identity provider, Transit Gateway route or egress path.

If the primary fails because the shared dependency is unavailable, the secondary fails for the same reason.

PRIMARY ENDPOINT ───┐
                    ▼
               SHARED KAFKA
                    ▲
SECONDARY ENDPOINT ─┘

DNS failover protects only against failures outside the shared dependency set.

A recovery review should ask:

Which failure makes
the primary unhealthy?

Can that same failure
disable the secondary?

The secondary does not need to duplicate every component. It needs enough independence to survive the failure scenarios the policy claims to handle.

How to operate the recovery system

Automated health and operator-controlled failover solve different problems

Automated failover is attractive: the service fails, health changes and Route 53 redirects future resolutions.

The difficult question is whether every detected failure should move traffic automatically.

Some failures are clear:

  • the primary endpoint is unreachable;
  • all application targets are unhealthy;
  • the environment has been deliberately isolated.

Others are ambiguous:

  • latency is elevated;
  • one dependency is degraded;
  • replication is behind;
  • the recovery environment may be less ready;
  • the monitoring system is unstable.

Amazon Application Recovery Controller can provide operator-controlled routing switches integrated with Route 53 health checks. ARC routing controls change the associated health-check state and can use safety rules to prevent dangerous combinations. They are traffic switches, not application-health monitors.

AUTOMATED FAILOVER

observed service health
        │
        ▼
DNS changes automatically


CONTROLLED FAILOVER

operator or recovery automation
        │
        ▼
routing-control state changes
        │
        ▼
DNS changes deliberately

A stateless private ingestion endpoint may tolerate automatic failover. A stateful order system with asynchronous cross-Region replication may require verification before writes move.

Mature automation is not merely fast. It moves traffic only when the destination is ready.

Failback can be more dangerous than failover

The primary recovers and its health check turns green. Automatic failback may still be premature.

During the outage, the secondary may have accepted writes, advanced event consumers, changed caches or reversed replication direction. The primary may have missed events.

PRIMARY HEALTHY AGAIN
        │
        ▼
IS ITS STATE CURRENT?
IS REPLICATION SAFE?
IS IT READY FOR AUTHORITY?

Failback should consider data reconciliation, replication direction, event offsets, connection draining, customer impact and operational approval.

For stateful services, automated detection with controlled failback is often safer. A green process or load-balancer target does not prove that the primary should reclaim authority.

Health checks need hysteresis

If a service alternates between healthy and unhealthy, Route 53 can repeatedly change the returned destination. Clients hold answers for different TTL windows, so traffic may split across primary and secondary.

A stable design should account for check interval, failure threshold, CloudWatch evaluation periods, missing-data behaviour, client TTL, cooldown and the evidence required before failback.

Route 53 endpoint health checks can run at 10-second or 30-second intervals and use a configured failure threshold. A compatible CloudWatch alarm can require consecutive breaching periods, but Route 53 supports only standard-resolution metrics and does not support metric math or “M out of N” alarms.

If bfstore needs asymmetric failure and recovery thresholds, implement them upstream and publish one binary readiness metric or controlled routing state.

# Conceptual upstream readiness policy,
# not a direct Route 53-compatible alarm definition.

readiness_policy:
  declare_unready:
    success_rate_below_percent: 95
    sustained_minutes: 3

  declare_ready_again:
    success_rate_at_or_above_percent: 99
    sustained_minutes: 10
    operator_approval: true

The numbers require testing. The principle is to prevent one noisy measurement from moving production traffic repeatedly.

Testing must include cached answers

A failover drill should test the complete path.

Baseline

Confirm that the primary is returned, clients connect, the secondary is independently healthy and both network paths work.

Trigger failure

Use a controlled health-check override, failed synthetic probe, ARC routing control or target withdrawal in a test environment.

Observe Route 53 and caches

Confirm that the primary health changes, the secondary remains healthy and fresh queries return the secondary.

Run:

dig otel.prod.aws.internal.bfstore.example
getent hosts otel.prod.aws.internal.bfstore.example

from several clients and resolvers. Record the answer, TTL and time until cached primary addresses disappear.

Observe real applications

Confirm that long-lived clients reconnect, retries remain bounded, authentication works, no duplicate operation is created and latency remains acceptable.

Observe the secondary

Check capacity, queue depth, throttling and downstream storage.

Test failback separately

Restore the primary without immediately returning traffic. Validate it, perform controlled failback and test clients again.

THE TEST IS COMPLETE WHEN

real clients reach a valid secondary

and return safely

A DNS response proves routing intent. A completed operational or business transaction proves recovery.

Metrics and evidence

The design should expose several layers.

Route 53 health

  • health-check state and transition time;
  • child-check state;
  • CloudWatch alarm data stream;
  • insufficient-data events.

DNS

  • Resolver query logs;
  • response data and code;
  • source VPC;
  • old and new answers;
  • observed TTL.

Resolver query logging records queries that reach Route 53 VPC Resolver. Queries answered from a resolver cache are not logged individually.

Client and recovery

  • connection failures, retries and reconnect time;
  • successful requests by destination;
  • secondary target health, capacity and queue depth;
  • replication state and failover approvals.

A useful incident timeline distinguishes detection, DNS selection, cache expiry, client reconnect and secondary stabilisation. That reveals where the recovery time was actually spent.

Route 53’s DNS data plane is separate from configuration work

Route 53 separates the DNS-serving data plane from the control plane used to create and update resources. AWS recommends relying on preconfigured health checks and failover records during recovery rather than making control-plane changes at the moment of failure.

WEAK RECOVERY

incident
   │
   ▼
engineer creates records
and health checks


STRONGER RECOVERY

incident
   │
   ▼
preconfigured health changes
   │
   ▼
existing records select secondary

The secondary configuration should exist before the primary fails. Recovery should execute prepared decisions rather than discover missing permissions, invalid targets or untested scripts during the incident.

Applying the model at bfstore

A bfstore private failover design

A useful first bfstore case is the observability ingestion endpoint.

Telemetry is important, but its front-door ingestion path can be largely stateless. Moving it between service cells is more manageable than moving the authoritative order database.

Stable service name

otel.prod.aws.internal.bfstore.example

Destinations

primary observability ingestion
eu-west-2

recovery observability ingestion
separate VPC or Region

The production, staging and platform VPCs can reach both destinations through approved private paths.

Separate control-loop probes

Each branch must be tested directly. A branch health probe must not resolve the failover name whose answer it controls.

otel-primary.prod.aws.internal.bfstore.example
        │
        ▼
primary ingestion endpoint
        │
        ▼
primary capability metric


otel-recovery.prod.aws.internal.bfstore.example
        │
        ▼
recovery ingestion endpoint
        │
        ▼
recovery capability metric

Each synthetic OpenTelemetry probe:

  1. resolves its endpoint-specific name;
  2. establishes the required transport;
  3. authenticates with a probe identity;
  4. submits a marked signal;
  5. verifies acceptance by the intended gateway;
  6. publishes a branch-specific metric.

Using otel.prod.aws.internal.bfstore.example for branch health would create a circular control loop. After failover, that name may resolve to the recovery endpoint, so the probe would no longer prove the health of the primary.

End-to-end service probe

A separate probe resolves the stable production name and measures the client path and service SLO. It confirms which branch Route 53 selected and whether clients can use it, but it does not directly determine either branch’s health.

BRANCH-SPECIFIC PROBES
        │
        ▼
FAILOVER DECISION


STABLE-NAME PROBE
        │
        ▼
END-TO-END RECOVERY EVIDENCE

Failover records

dns_failover:
  name: otel.prod.aws.internal.bfstore.example
  type: A

  primary:
    identifier: primary-observability
    target: primary-ingest-load-balancer
    health: primary-capability-health

  secondary:
    identifier: recovery-observability
    target: recovery-ingest-load-balancer
    health: recovery-capability-health

The recovery gateway maintains enough baseline capacity to accept production telemetry or can scale within the recovery window. It may use an independent queue and storage, or an explicitly accepted period of reduced retention.

Failback is operator-approved after the primary-specific probe stays healthy, storage and queues are reconciled, capacity is confirmed and the stable-name probe succeeds after traffic returns.

This is a realistic use of Route 53 failover: both destinations accept the same protocol and identity, the secondary is independently useful, and DNS has a precise job.

A weaker bfstore use case: failing over the order database name

Consider:

mysql.orders.prod.aws.internal.bfstore.example

Changing its DNS answer from a primary RDS MySQL writer to a recovery writer does not prove that the secondary has been promoted, contains current data, matches the schema or has fenced the old primary from writes. It also does not reconnect existing pools or reconcile event consumers and idempotency state.

replicate or restore
        │
        ▼
promote secondary
        │
        ▼
fence old primary
        │
        ▼
validate state
        │
        ▼
change DNS
        │
        ▼
reconnect and verify writes

DNS may be one step in the procedure. It should not be the entire procedure. A record that moves before the database can safely change authority is unsafe.

Where failover routing earns its keep

Route 53 failover is a strong fit when:

  • clients must keep one stable name;
  • both destinations accept the same protocol and identity;
  • consuming networks can already reach both;
  • health represents the service capability;
  • TTL and reconnect behaviour fit the recovery objective;
  • the secondary has tested capacity, state and dependencies;
  • failback requires deliberate evidence.

It is weaker when the secondary shares the primary’s failure domain, needs a long rebuild, cannot be reached privately, or must complete stateful promotion before accepting traffic. It also adds little when a load balancer already handles the relevant boundary.

Failover routing is a selector between prepared alternatives.

Pre-deployment review

Failure domain

  • Which failure should move traffic?
  • Can it also disable the secondary?
  • Which critical dependencies are shared?

Health

  • Does each branch-specific check represent the service contract?
  • Is the secondary checked, target-evaluated or intentionally unchecked?
  • What does insufficient data mean?
  • What happens when both branches are unhealthy?

Reachability, state and capacity

  • Can every consuming VPC reach both destinations?
  • Will the same identities and credentials work?
  • Is the secondary’s state current enough?
  • Can it carry the failover load?
  • What promotion, fencing or reconciliation must happen first?

Clients and operations

  • How long may resolvers retain the old answer?
  • When do long-lived clients resolve again?
  • Are retries bounded and operations idempotent?
  • Is failover automatic or controlled?
  • What evidence permits failback?
  • Has the complete path been tested with real clients?

The mental model

The incomplete model is:

PRIMARY FAILS
      │
      ▼
ROUTE 53 NOTICES
      │
      ▼
EVERYTHING MOVES
TO SECONDARY

The useful model is:

PRIMARY CAPABILITY
        │
        ▼
BRANCH-SPECIFIC HEALTH
        │
        ▼
ROUTE 53 DECISION
        │
        ├── primary healthy
        │       └── primary answer
        │
        └── primary unhealthy
                ├── secondary healthy
                │       └── secondary answer
                └── secondary unhealthy
                        └── primary may still return
        │
        ▼
CACHE EXPIRES
        │
        ▼
CLIENT RESOLVES AND RECONNECTS
        │
        ▼
DESTINATION ACCEPTS WORK

The records decide where traffic may go. Health decides whether each destination can fulfil the contract. TTL and clients determine when the change takes effect. Network reachability and recovery readiness determine whether it succeeds.

Route 53 is the switch in a recovery system, not the recovery system itself.

It earns its keep between two already viable destinations.

References and further reading