Cloud Sandbox
All notes

Reading a route table from destination outward

An AWS route table is easier to understand when read as a destination lookup rather than a sequence of rows. Start with the packet destination, find the longest matching prefix, follow the selected target and then verify the onward and return paths.

about 15 minutes min read

An application sends a packet to:

10.80.12.45

Its subnet route table contains:

Destination          Target

10.64.0.0/16         local
10.80.0.0/16         tgw-0123456789abcdef0
0.0.0.0/0            nat-0123456789abcdef0

I used to read the table from top to bottom.

That is visually accurate and operationally useless. AWS does not choose a route because its row appears first.

It examines the packet destination and selects the most specific matching route. This is longest prefix match.

For 10.80.12.45:

Does 10.64.0.0/16 contain it?

No.


Does 10.80.0.0/16 contain it?

Yes.


Does 0.0.0.0/0 contain it?

Yes.

Two routes match.

10.80.0.0/16 is more specific than 0.0.0.0/0.

The packet is sent towards the Transit Gateway.

A route table is not a checklist executed in display order. It is a collection of candidate answers to one question: where should traffic for this destination go next?

Choose the first routing table

This article begins with traffic originating from a resource inside a VPC.

For that traffic, I start with the packet’s effective source network interface and the route table associated with its subnet.

SOURCE WORKLOAD
      │
      ▼
EFFECTIVE SOURCE ADDRESS
AND NETWORK INTERFACE
      │
      ▼
SOURCE SUBNET
      │
      ▼
ASSOCIATED ROUTE TABLE

Each subnet uses one subnet route table at a time. The association can be explicit, or the subnet can inherit the VPC’s main route table. One table can serve several subnets.

The first troubleshooting question is therefore:

Which route table does the effective
source network interface actually use?

Opening a route table with a promising name is not enough.

bfstore-prod-private-routes

may look correct while the source subnet still inherits:

bfstore-prod-main

The unattached table has no effect on that packet.

Find the effective source, not merely the workload label

Suppose an EKS Pod runs on a node in:

bfstore-prod-application-a

The relevant routing origin depends on the packet’s effective source address and network interface after the cluster’s networking and any source-address translation have been applied.

Depending on the design, the source may be a Pod address, a secondary address on the node interface, or the node address after source NAT. I do not assume the node subnet is the routing origin merely because the Pod runs there.

Ingress can begin with a gateway route table

Traffic entering a VPC through an internet gateway or virtual private gateway can follow a different first lookup.

AWS allows a route table to be associated with either gateway. A gateway route table can redirect incoming traffic through a network interface or Gateway Load Balancer endpoint before the traffic reaches the destination subnet.

TRAFFIC ORIGINATING IN A SUBNET

source interface
    → source subnet route table


TRAFFIC ENTERING THROUGH A GATEWAY

gateway route table, where associated
    → middlebox or destination subnet

The method remains the same; the first table changes with the routing origin.

Perform the route lookup

Once I know the applicable table, I work through four steps.

1. Write down the exact destination

A route lookup begins with an IP address, not a service name or resource label.

A useful investigation records:

source:
  account: bfstore-prod
  subnet: bfstore-prod-application-a
  address: 10.64.96.34

destination:
  service: central-observability-ingest
  address: 10.80.12.45
  port: 4317
  protocol: tcp

The route-table decision uses:

10.80.12.45

The port and protocol matter later to security controls and the application.

They do not select the VPC route.

2. List every matching destination

Consider:

Destination          Target

10.64.0.0/16         local
10.80.0.0/16         tgw-0123456789abcdef0
10.80.12.0/24        pcx-0123456789abcdef0
0.0.0.0/0            nat-0123456789abcdef0

For:

10.80.12.45

the matching routes are:

10.80.0.0/16

10.80.12.0/24

0.0.0.0/0

The local route does not match because the destination sits outside 10.64.0.0/16.

3. Choose the longest prefix

Compare the matching prefix lengths:

/24

/16

/0

The largest prefix is the most specific.

10.80.12.0/24

wins.

The packet is sent to:

pcx-0123456789abcdef0

DESTINATION

10.80.12.45
      │
      ├── matches 0.0.0.0/0
      ├── matches 10.80.0.0/16
      └── matches 10.80.12.0/24
                         │
                         ▼
                  MOST SPECIFIC
                         │
                         ▼
                  PEERING TARGET

A route is a claim about destinations.

10.80.0.0/16 → Transit Gateway

means that addresses in that range use the Transit Gateway unless a more-specific route wins. It says nothing about target health, onward routing, the return path, security controls or application health.

4. Treat the default route as the fallback

The IPv4 default route is:

0.0.0.0/0

It matches every IPv4 address because it fixes no destination bits.

It handles a packet only when no more-specific matching route wins.

For example, 10.64.110.20 uses the local route, 10.80.12.45 uses the Transit Gateway route, and 1.1.1.1 uses the default route.

For IPv6, the default destination is:

::/0

IPv4 and IPv6 routes are evaluated independently.

Understand the local routes

Every VPC route table contains local routes for the VPC’s associated IPv4 and IPv6 CIDR blocks.

A VPC with several CIDR blocks therefore has several local routes.

For a VPC using:

10.64.0.0/16

the ordinary route is:

10.64.0.0/16 → local

A packet from 10.64.96.34 to 10.64.110.20 uses that route. The local route does not mean the packet is accepted automatically. Routing selects the next hop; security controls decide whether the traffic is permitted; the application must still be listening and healthy.

A route table directs traffic. It is not a firewall rule written sideways.

AWS supports replacing a local-route target or adding a route more specific than the local route for advanced middlebox designs.

For a more-specific route inside the VPC:

  • the destination must match the complete CIDR of a VPC subnet;
  • the target must be a NAT gateway, network interface, or Gateway Load Balancer endpoint.

Any route that redirects traffic away from the ordinary local path deserves a clear architectural explanation and a tested return path.

Resolve ties, origins and route state

Longest prefix match is always the first selection rule.

Additional rules matter only when routes remain equivalent.

Static and propagated routes

When a static and propagated route use the same destination, the static route wins.

The reading process is:

1. Find all matching destinations.

2. Choose the longest prefix.

3. If equivalent routes remain,
   apply route-origin priority.

A propagated /24 still beats a static /16.

Specificity comes first.

Prefix-list routes

A route can use a managed prefix list as its destination.

For example, an S3 gateway endpoint may add:

Destination          Target

pl-0123456789abcdef0 vpce-0123456789abcdef0

The route matches when the packet destination belongs to a CIDR inside the prefix list. The CIDR’s specificity, not the prefix-list ID itself, participates in route selection.

AWS applies these additional rules:

  1. Longest prefix still wins.
  2. A prefix-list route beats an equally matching propagated route.
  3. If several prefix-list routes contain overlapping CIDRs and use different targets, AWS chooses one route, then continues to prefer that route.

The third case is not a useful policy mechanism.

I avoid overlapping prefix lists with different targets.

Route state and blackholes

A matching route is not necessarily usable.

If a target such as a NAT gateway is deleted, the route may remain in blackhole state.

Transit Gateway route tables can also contain deliberate blackhole routes.

10.80.12.0/24 → blackhole

0.0.0.0/0     → Transit Gateway

The /24 remains the most-specific match.

The healthy default route does not become an automatic fallback.

After selecting the winner, I check its state, target existence, attachment or propagation state, and target health.

Routes installed by VPC Route Server

Amazon VPC Route Server can learn routes through BGP and install or withdraw them in associated VPC and internet-gateway route tables. The destination-first lookup still applies, but troubleshooting may also require the peer, BGP or BFD state, RIB, FIB and route-table propagation state.

Repeat the lookup at every routing boundary

The selected target is only the next hop.

Suppose the route is:

0.0.0.0/0 → nat-0123456789abcdef0

The conclusion is:

The source subnet sends the packet
to this NAT gateway.

It is not:

The packet reaches the internet.

I continue outward.

PRIVATE SUBNET ROUTE TABLE
          │
          ▼
NAT GATEWAY
          │
          ▼
NAT SUBNET ROUTE TABLE
          │
          ▼
INTERNET GATEWAY
          │
          ▼
DESTINATION

For public IPv4 egress, the NAT gateway must be in the intended subnet and that subnet must route to an attached internet gateway. The destination must also return traffic successfully.

Transit Gateway adds another routing domain

A route to a Transit Gateway means that the VPC has handed the packet to its attachment.

The investigation continues through:

SOURCE WORKLOAD SUBNET ROUTE TABLE
        │
        ▼
TRANSIT GATEWAY ATTACHMENT
        │
        ▼
TGW ROUTE TABLE ASSOCIATED
WITH THE SOURCE ATTACHMENT
        │
        ▼
DESTINATION VPC ATTACHMENT
        │
        ▼
ATTACHMENT-SUBNET ROUTE TABLE
        │
        ▼
DESTINATION SUBNET

The attachment subnets are the VPC’s entry and exit points for Transit Gateway traffic.

Their associated route tables must be able to reach the destination inside the VPC.

The workload subnet also needs its route back towards the Transit Gateway for external destinations.

I verify the source attachment’s Transit Gateway route table, the winning transit route, destination attachment state, selected attachment subnets, their VPC route tables, the destination subnet path and the return route.

The return path is a separate investigation

Routing is evaluated independently in each direction. The forward lookup for 10.64.96.34 → 10.80.12.45 does not prove the return lookup for 10.80.12.45 → 10.64.96.34.

This matters especially with:

  • Transit Gateway;
  • VPC peering;
  • VPNs;
  • inspection appliances;
  • centralised egress;
  • asymmetric routing.

A route table showing a perfect forward route is only half a successful conversation.

A public subnet is defined by routing

A subnet named bfstore-prod-public-a is not public when its table contains only the local route. A differently named subnet with:

10.64.0.0/16 → local
0.0.0.0/0    → internet gateway

has the public routing path.

The route alone does not give every IPv4 resource public connectivity.

A resource also needs an appropriate public address or public-facing service relationship, and the surrounding security controls must allow the traffic.

The subnet name is documentation.

The route is behaviour.

Keep the main route table quiet

AWS creates a main route table with the VPC. A subnet without an explicit association inherits it and may receive a route never reviewed as part of that subnet’s design.

NEW SUBNET
    │
    ▼
NO EXPLICIT ASSOCIATION
    │
    ▼
MAIN ROUTE TABLE
    │
    ▼
INHERITED NETWORK PATH

For bfstore, I prefer:

  • a quiet main route table;
  • custom route tables for intentional subnet classes;
  • explicit associations managed as code.

The main table becomes a safe fallback rather than a cupboard of inherited surprises.

My bfstore reading method

Suppose order-service must reach:

10.80.12.45

I work through the path as follows.

1. Identify the effective source

Record:

source address
source network interface
source subnet

For Kubernetes, confirm whether the packet leaves with the Pod address or a translated node address.

2. Find the first route table

For workload-originated traffic, inspect the source subnet’s actual association.

For gateway ingress, inspect the gateway route table where one is associated.

3. Write the destination

10.80.12.45

4. List every matching destination

Include CIDRs and matching entries inside prefix lists.

5. Select the longest prefix

Apply origin priority only when equally specific routes remain.

6. Check route state

Confirm that the winning route is active and its target exists.

7. Follow the target

At a Transit Gateway, NAT gateway, peering connection, endpoint, middlebox or gateway, continue into the next routing domain.

8. Read the return path

For Transit Gateway paths, include the attachment-subnet route tables. Then begin again with the original source address as the destination.

9. Inspect non-routing controls

Check:

  • security groups;
  • network ACLs;
  • DNS resolution;
  • service listener;
  • TLS configuration;
  • application health.

The route table is one chapter in the packet’s journey.

I no longer ask it to explain the entire novel.

Questions I now ask

Origin

Which interface and routing boundary receive the packet first?

Destination

What is the exact destination IP address?

Match

Which CIDRs or prefix-list entries contain it?

Selection

Which match has the longest prefix?

Tie

Do equally specific routes need origin-priority rules?

State

Is the winning route active, propagated and attached to a healthy target?

Onward path

Which routing table is consulted after the target?

Return

How does the response reach the original source?

Permission

Which controls allow or deny the correctly routed traffic?

This turns a wall of CIDRs into a sequence of forwarding decisions.

The mental model I am keeping

My earlier model was:

ROUTE TABLE

read from top to bottom

find a row that looks relevant

assume the target reaches
the destination

The stronger model is:

                         PACKET ORIGIN
                               │
              ┌────────────────┴────────────────┐
              │                                 │
              ▼                                 ▼
       SOURCE SUBNET                    GATEWAY INGRESS
        ROUTE TABLE                      ROUTE TABLE
              │                                 │
              └────────────────┬────────────────┘
                               ▼
                    EXACT DESTINATION ADDRESS
                               │
                               ▼
                    ALL MATCHING DESTINATIONS
                               │
                               ▼
                      LONGEST PREFIX MATCH
                               │
                               ▼
                    TIE-BREAK AND ROUTE STATE
                               │
                               ▼
                         SELECTED TARGET
                               │
                               ▼
                     NEXT ROUTING DOMAIN
                               │
                               ▼
                       FINAL DESTINATION
                               │
                               ▼
                         RETURN LOOKUP

The routing origin answers:

Which route table is consulted first?

The destination answers:

Which entries are candidates?

Longest prefix match answers:

Which candidate is most specific?

Origin priority answers:

Which equivalent route wins?

Route state answers:

Can the selected route still forward?

The target answers:

Where does the packet go next?

The return lookup answers:

Can the destination reply?

And the security controls answer:

Is the correctly routed traffic
actually permitted?

A route table makes one decision at one routing boundary.

Begin with the destination. Work backwards to the most specific matching prefix. Then move forwards through the selected target.

For 10.80.12.45, ask which routes contain it.

For the winning route, ask whether it is active and where it points.

At that target, ask which table decides next.

Then repeat the process in the opposite direction for the reply.

The packet does not care which row looked important.

It follows the narrowest applicable signpost, one routing domain at a time.

References and further reading