NAT gateways: why private workloads still need an exit

Suppose bfstore, the fictional organisation used throughout this series, runs an application Pod in a private subnet.

It accepts customer requests through an internet-facing Application Load Balancer.

It has no public IPv4 address.

Its subnet has no direct route to an internet gateway.

The application is private from a routing perspective.

It still needs to pull images and packages, call payment and shipping providers, retrieve certificate-status information and report to external services.

The workload has avoided a direct internet entrance.

It still needs an exit.

INTERNET USER
     │
     ▼
PUBLIC APPLICATION LOAD BALANCER
     │
     ▼
PRIVATE BFSTORE WORKLOAD
     │
     ├── payment provider
     ├── shipping provider
     ├── software repository
     └── public API

The inbound and outbound paths solve different problems. The Application Load Balancer provides a controlled route into the application; the egress path lets it initiate connections out to dependencies.

For IPv4 workloads in private subnets, an AWS NAT gateway is one common way to provide that exit. The private resource sends traffic to the NAT gateway, which replaces the resource’s source address before forwarding the connection. External systems can return traffic belonging to that connection, but they cannot use the NAT gateway to initiate an unsolicited connection to the private resource.

A private subnet removes the direct internet route. A NAT gateway adds a controlled outbound path without restoring that direct inbound route.

That is useful, but it is not the whole egress design.

Example-address note: Public IPv4 addresses in this article use the RFC 5737 documentation ranges 198.51.100.0/24 and 203.0.113.0/24. They are illustrative and must not be used as production addresses.

Private does not mean self-sufficient

A private workload may depend on:

public services such as payment APIs, repositories and vendor endpoints
AWS services such as STS, ECR, S3 and CloudWatch Logs
private destinations such as shared VPCs, partners and on-premises networks

Those destinations should not all use the same path merely because they sit outside the source subnet.

Requirement	Preferred path
Same VPC	Local VPC routing
Another bfstore VPC	Transit Gateway, PrivateLink or another private connection
Supported AWS service	Gateway or interface VPC endpoint where appropriate
Public IPv4 service	Public NAT gateway and internet gateway
Direct public IPv6 connectivity	Internet gateway
Outbound-only public IPv6 connectivity	Egress-only internet gateway
IPv6 workload reaching IPv4	DNS64 and NAT64 where appropriate
Forbidden destination	No route or an explicit egress-policy denial

The IPv6 rows assume public IPv6 space. AWS private IPv6 ranges cannot use an internet gateway or egress-only internet gateway for direct internet egress; AWS drops that traffic at the gateway edge.

The question is not:

Does the private subnet need internet?

It is:

Which destination does the workload require,
and what is the narrowest suitable path?

NAT is a general-purpose translated IPv4 path, not the answer to every outbound connection.

NAT translates addresses and ports

Suppose a private workload has:

Private workload address:
    10.64.20.34

Zonal NAT gateway private address:
    10.64.0.10

NAT gateway Elastic IP:
    198.51.100.20

The original packet may begin as:

Source:
    10.64.20.34:49152

Destination:
    203.0.113.80:443

A public NAT gateway maps the workload source to a NAT address and translated port.

For a zonal public NAT gateway, the internet gateway then maps the NAT gateway’s private address to its Elastic IP.

Conceptually:

10.64.20.34:49152
        │
        ▼
10.64.0.10:translated-port
        │
        ▼
198.51.100.20:translated-port
        │
        ▼
203.0.113.80:443

The external service sees:

198.51.100.20

rather than:

10.64.20.34

That can provide a stable allow-listed source address when the NAT design uses explicitly governed Elastic IP addresses.

A Regional NAT Gateway can operate in automatic or manual address-management mode. Automatic mode lets AWS manage addresses and Availability Zone expansion. Manual mode lets the customer control addresses per Availability Zone. A vendor allow-listing design should therefore document the complete approved EIP set and how it changes as the regional gateway expands or contracts.

The shared source address also reduces attribution at the destination.

Several workloads may appear as one public IP.

The internal evidence must preserve which workload initiated each connection.

NAT is stateful, but it is not an egress policy

A public NAT gateway permits responses to connections initiated from behind it. It does not let an external host initiate an arbitrary connection to a private workload.

That is useful inbound protection, not a complete outbound security policy.

A NAT gateway does not decide:

which domain the application may access
whether a download is malicious
whether a SaaS destination is approved
whether data should leave the workload

NAT GATEWAY

translation
connection state
managed forwarding


NOT BY ITSELF

URL filtering
application inspection
malware scanning
data-loss prevention

Security groups cannot be attached directly to a NAT gateway. Outbound permissions remain on the source resources.

For zonal NAT, network ACLs on the source and NAT subnets can affect the path. Regional NAT is not hosted in a subnet, so there is no NAT-hosting subnet ACL.

A controlled path may combine:

WORKLOAD SECURITY GROUP
        │
        ▼
FIREWALL OR PROXY
        │
        ▼
NAT GATEWAY
        │
        ▼
INTERNET GATEWAY

NAT provides the exit mechanism. Egress policy decides which journeys are allowed through it.

AWS now offers several NAT models

The word NAT gateway no longer describes only one availability model.

Model	Placement	Availability model	Typical use
Zonal public NAT	Public subnet	One gateway in one Availability Zone	Internet egress with explicit per-AZ control
Regional NAT	VPC-level resource	Automatically spans workload AZs in automatic mode	Simplified multi-AZ public egress
Zonal private NAT	Subnet	One gateway in one Availability Zone	Translation between private networks
Centralised public egress	Egress VPC	Zonal or Regional NAT behind transit routing	Shared egress and inspection across VPCs

A public NAT gateway provides translated access to public IPv4 destinations.

A private NAT gateway translates between private networks. It cannot provide internet egress through an internet gateway. Regional NAT does not support private connectivity, so private NAT remains a zonal model.

The right question is not:

Do we use NAT Gateway?

It is:

Which NAT availability and connectivity model
fits this traffic, failure domain and operating model?

The zonal public NAT path

A zonal public NAT gateway used for internet egress lives in a public subnet.

It has:

a private IPv4 address from that subnet
one or more associated Elastic IP addresses
a route from its subnet to an internet gateway

The private workload subnet sends internet-bound traffic to the NAT gateway.

PRIVATE APPLICATION SUBNET
10.64.20.0/22
        │
        │ 0.0.0.0/0
        ▼
ZONAL PUBLIC NAT GATEWAY
10.64.0.10
        │
        │ 0.0.0.0/0
        ▼
INTERNET GATEWAY
        │
        ▼
PUBLIC IPV4 DESTINATION

The two route tables perform separate jobs.

Private application route table

Destination          Target

10.64.0.0/16         local
0.0.0.0/0            nat-application-a

Public NAT subnet route table

Destination          Target

10.64.0.0/16         local
0.0.0.0/0            internet-gateway

The first route does not point directly to the internet gateway.

The application subnet remains private.

The second route gives the zonal NAT gateway its public path.

If either route is missing, the complete path fails.

A NAT gateway placed in a subnet called public does not become useful through nomenclature. Its actual route table must contain the required internet-gateway route.

A zonal design needs an explicit availability decision

A zonal NAT gateway is implemented with redundancy inside one Availability Zone.

It is not automatically multi-AZ.

If private subnets in several zones all route through one zonal NAT gateway, they share that gateway’s Availability Zone as an egress dependency. If that zone becomes unavailable, workloads in the other zones can lose that path too.

A resilient zonal design uses same-zone egress:

AVAILABILITY ZONE A

private-application-a
        │
        ▼
nat-a in public-egress-a
        │
        ▼
internet gateway


AVAILABILITY ZONE B

private-application-b
        │
        ▼
nat-b in public-egress-b
        │
        ▼
internet gateway


AVAILABILITY ZONE C

private-application-c
        │
        ▼
nat-c in public-egress-c
        │
        ▼
internet gateway

This improves resilience and avoids sending ordinary egress traffic across zones before it reaches NAT.

It also creates:

three NAT Gateway-hour charges
three resources to monitor
three address sets
additional route-table configuration

For a short-lived development environment, one zonal NAT gateway may be an accepted trade-off.

environment: development

nat:
  availability_mode: zonal
  count: 1
  accepted_risk:
    - internet egress unavailable during NAT zone failure
    - possible cross-AZ processing

For production, the choice is no longer limited to one risky zonal gateway or several manually deployed zonal gateways.

Regional NAT provides another model.

Regional NAT removes the hosting subnet

A Regional NAT Gateway is created for the VPC rather than in a public subnet. Private subnets across Availability Zones can point to the same gateway ID.

PRIVATE SUBNET A ─┐
PRIVATE SUBNET B ─┼── 0.0.0.0/0 → regional NAT ID
PRIVATE SUBNET C ─┘
                         │
                         ▼
                  REGIONAL NAT GATEWAY
                         │
                         ▼
                  INTERNET GATEWAY

AWS creates a dedicated route table for the gateway with a preconfigured internet-gateway route. It can also contain return routes to supported middlebox paths, including Transit Gateway.

Regional NAT supports:

Automatic mode

AWS manages Availability Zone expansion, contraction and address allocation.

Manual mode

The customer controls configured zones and Elastic IP addresses per zone.

Manual mode provides explicit address control but returns more lifecycle responsibility to the platform team.

Expansion into a newly used Availability Zone can take up to 60 minutes. Until it completes, traffic from that zone may be processed cross-zone by an existing presence.

Regional NAT simplifies the steady-state architecture. It does not remove the need to understand its changing zonal footprint.

Capacity depends on the NAT model

A NAT gateway starts at 5 Gbps and can scale to 100 Gbps. It starts at one million packets per second and can scale to ten million.

Each IPv4 address supports up to 55,000 simultaneous connections to each unique destination tuple:

destination IP
destination port
protocol

A fleet calling one SaaS API can therefore encounter port pressure before aggregate bandwidth becomes the problem.

A zonal NAT gateway supports up to eight IPv4 addresses, subject to quotas. Regional NAT supports up to 32 per Availability Zone.

Responses include:

adding addresses or gateways
using Regional automatic address management
reusing application connections
replacing public traffic with a private endpoint
distributing destinations where supported

AWS publishes bytes, packets, connections, port-allocation errors, idle timeouts and packet drops.

Zonal metrics use the NAT Gateway ID dimension. Regional metrics use the gateway ID and Availability Zone, so monitoring should preserve the zonal view.

NAT connection state has a 350-second idle timeout

A NAT gateway expires a flow after 350 seconds without activity.

If a TCP client later attempts to continue that expired connection, the gateway can return a reset rather than a graceful close. The application must reconnect.

This matters for:

connection pools
long-lived API clients
streaming protocols
database connections crossing NAT
infrequent keepalive traffic

A workload may report:

The connection works.

Then it fails after several quiet minutes.

The route may be correct.

The NAT gateway may be healthy.

The application’s connection lifecycle may be incompatible with the translation timeout.

Possible responses include:

application-level retries
TCP keepalives below the timeout
shorter connection-pool lifetimes
avoiding NAT for that destination
using a private routed path

Routing creates the path.

Applications still need to behave appropriately on it.

Cost has more than two dimensions

NAT Gateway cost includes:

NAT Gateway-hours
data processed
standard data-transfer charges
possible cross-AZ transfer
transit or inspection charges in centralised designs

The hourly model differs by availability mode.

Model	Hourly basis	Common additional cost
Zonal NAT	Per provisioned gateway	Cross-AZ transfer when clients use another zone’s gateway
Regional NAT	Per active Availability Zone	Data processing and internet transfer
Centralised egress	NAT plus transit infrastructure	Transit Gateway, inspection and cross-VPC path costs

A single Regional NAT Gateway ID active across three Availability Zones is billed for three NAT Gateway-hours.

Regional NAT’s primary advantages are not necessarily a lower fixed NAT bill. They are:

no public NAT subnets
one NAT Gateway ID
automatic multi-AZ expansion
simpler route management
higher per-zone address limits

Cost optimisation should start with traffic classification.

Which traffic genuinely needs
general public IPv4 egress?

Which traffic can remain private?

Which traffic can use
a service-specific endpoint?

The cheapest NAT byte is often the one that never needed to cross NAT.

S3 and DynamoDB can avoid NAT through gateway endpoints

Amazon S3 and DynamoDB support gateway VPC endpoints.

A gateway endpoint inserts service-prefix routes into selected route tables and allows private connectivity without requiring an internet gateway or NAT device. AWS currently does not charge hourly or data-processing fees for gateway endpoints.

A private route table may contain:

Destination          Target

10.64.0.0/16         local
pl-s3                 s3-gateway-endpoint
0.0.0.0/0            nat-gateway

Traffic to S3 matches the more-specific prefix-list route.

Other public IPv4 traffic uses the default NAT route.

PRIVATE WORKLOAD
      │
      ├── S3 destination
      │       ▼
      │  gateway endpoint
      │
      └── public API
              ▼
          NAT gateway

Gateway endpoints are Regional and apply through associated route tables. S3 and DynamoDB resources should be evaluated in the same Region as the endpoint. Gateway endpoints are not extended through VPC peering, Transit Gateway, VPN or Direct Connect in the way an interface endpoint may be used for broader connectivity patterns.

They are highly useful.

They are not a universal route to every copy of S3 or DynamoDB from every connected network.

Interface endpoints replace selected public AWS paths

Many AWS services support interface VPC endpoints through AWS PrivateLink.

An interface endpoint creates private endpoint network interfaces in selected subnets. Workloads can reach a supported service without a NAT gateway, internet gateway or public IP.

Typical endpoint candidates in a private EKS environment include:

Amazon ECR API
Amazon ECR Docker registry
AWS STS
Amazon EKS
EKS Auth for Pod Identity
Amazon EC2
Elastic Load Balancing
CloudWatch Logs
Systems Manager

A fully private EKS data plane must provide private paths to the APIs and registries its nodes and Pods use. The exact endpoint set depends on features and identity mechanisms.

Interface endpoints are not free replacements for NAT.

They carry hourly and data-processing charges, consume subnet addresses and require security-group and DNS configuration.

The choice should compare:

number of endpoints
number of Availability Zones
traffic volume
security requirements
NAT processing cost
service availability through PrivateLink
operational complexity

Sometimes the endpoint is the obvious answer.

Sometimes one lightly used API does not justify endpoint interfaces across every zone.

No general internet access is a real architecture

A private workload can operate without general outbound internet access.

Every dependency then needs another path.

Requirement	Private path
Kubernetes API	EKS private cluster endpoint
Container images	Private ECR plus required ECR and S3 endpoints
Pod Identity	EKS Auth interface endpoint
IRSA credentials	Regional STS interface endpoint
CloudWatch Logs	Logs interface endpoint
EC2 API	EC2 interface endpoint
Application packages	Internal mirror or prebuilt image
External payment API	Controlled public egress still required

This reveals an important distinction:

PLATFORM CAN RUN WITHOUT INTERNET

does not imply:

APPLICATION HAS NO PUBLIC DEPENDENCIES

bfstore may build a private Kubernetes control and platform path while still requiring public egress for:

payment services
shipping services
email delivery
fraud APIs
other external business integrations

The exit should remain narrow and observable.

It should not disappear from the diagram merely because the cluster can bootstrap without it.

Decentralised, Regional and centralised egress optimise different things

A decentralised zonal model places NAT gateways in each workload VPC and Availability Zone.

AZ A: application-a → nat-a
AZ B: application-b → nat-b
AZ C: application-c → nat-c

It provides explicit zonal routing, local failure ownership and no transit dependency, but repeats cost, addresses and operations.

A Regional NAT model stays inside the workload VPC while removing public NAT subnets and customer-managed per-AZ gateways.

application-a ─┐
application-b ─┼── regional NAT ID
application-c ─┘

It provides one gateway ID, automatic multi-AZ availability and zonal affinity.

A centralised model sends traffic through Transit Gateway or Cloud WAN to an egress VPC.

WORKLOAD VPC
     │
     ▼
TRANSIT GATEWAY
     │
     ▼
EGRESS VPC
inspection → NAT → internet gateway

Centralisation can provide common policy, logging and egress addresses. It also introduces transit cost, complex return paths and a shared failure domain.

Central egress is an independent platform service. It needs ownership, SLOs, capacity controls, failure testing and cost allocation.

bfstore’s intended exit model

bfstore should keep four paths visibly separate.

Consumer ingress

Internet → public ALB → private EKS workloads

This is not NAT egress.

Internal and AWS-service traffic

private workload
   ├── private routing → shared services
   └── VPC endpoint   → supported AWS service

Internal telemetry should not leave through public NAT merely because its destination is in another account.

External business APIs

private workload
   │
   ▼
controlled public NAT
   │
   ▼
payment / shipping / email provider

The implementation may be zonal, Regional or centralised.

Data tier

RDS subnet → approved internal routes only

Database subnets should not inherit a general default route because application workloads need one.

A conceptual contract is:

subnet_class: private-application

routes:
  internal_vpc:
    destination: 10.64.0.0/16
    target: local

  central_network:
    destination: bfstore-private-summary
    target: transit-gateway

  public_ipv4:
    destination: 0.0.0.0/0
    target: controlled-egress

public_ipv4_assignment: disabled

Routes provide paths. Security and inspection policy decide which destinations may use them.

The exit needs evidence

A shared NAT address reduces the attribution available to the external destination.

Several workloads may all appear as:

198.51.100.20

Internally, bfstore should still be able to answer:

which account originated the traffic
which VPC and subnet it came from
which network interface initiated it
which destination it reached
how many bytes crossed the path
whether the connection failed
whether the NAT gateway approached capacity
which Availability Zone processed a Regional NAT flow

Evidence can include:

VPC Flow Logs
NAT gateway CloudWatch metrics
firewall logs
proxy logs
DNS query logs
application traces
external API audit records

Useful alarms include:

Unexpected increase in processed bytes

Port-allocation pressure

Packets dropped by the NAT gateway

Rise in idle connection timeouts

Zonal clients using another zone's NAT

Regional NAT not expanded into an active zone

Traffic to an unapproved destination class

The NAT gateway is part of the platform data path.

It should not be monitored only by discovering that package downloads have stopped.

Troubleshoot the path by NAT model

Start with the common path:

Confirm DNS returns the expected address family and destination.
Record the workload, interface, subnet and Availability Zone.
Read the source route table and check for more-specific or blackhole routes.
Verify source security groups and network ACLs.

Then follow the selected model.

Zonal public NAT

Confirm the gateway is Available and in the expected zone.
Verify its Elastic IP configuration.
Read the NAT subnet route table.
Confirm 0.0.0.0/0 → internet gateway.
Check the NAT-subnet ACL and gateway metrics.

Regional NAT

Confirm the source route targets the Regional NAT Gateway ID.
Verify that the gateway is active in the source zone.
Account for expansion time in a newly used zone.
Inspect the Regional NAT route table.
Verify automatic or manual address configuration.
Review metrics by gateway ID and Availability Zone.

For either model, check the internet-gateway attachment, port errors, drops, idle timeouts and the remote service’s allow list, protocol, TLS and credentials.

The useful question is not:

Is NAT broken?

It is:

At which routing, translation,
security or application boundary
does the packet stop?

The decision table

Area	Question
Dependency	Why does the workload need to leave its subnet or VPC?
Destination	Is the target internal, an AWS service, public IPv4 or public IPv6?
Alternative	Can a private route or VPC endpoint replace general NAT?
NAT model	Does the design need zonal, Regional, private or centralised NAT?
Availability	What happens when one Availability Zone or the shared egress service fails?
Addressing	Which public EIPs must remain stable for allow lists?
Policy	Which control restricts destinations beyond address translation?
Capacity	Could one destination exhaust connection-port capacity?
Cost	What hourly, processing, transit and cross-AZ cost does the path create?
Evidence	Can the original workload and processing Availability Zone be identified?
Recovery	Has the failure and replacement path been tested?

These questions turn NAT from an icon between two subnets into an egress architecture.

The mental model I am keeping

My earlier model was:

PRIVATE SUBNET

no internet

safe

The stronger model is:

PRIVATE WORKLOAD
      │
      ▼
DESTINATION CLASS
      │
      ├── internal    → private routing
      ├── AWS service → VPC endpoint
      └── public      → controlled egress
                              │
                  ┌───────────┼───────────┐
                  ▼           ▼           ▼
              zonal NAT  Regional NAT  central egress
                  └───────────┼───────────┘
                              ▼
                       public destination

The NAT model answers:

How should translated IPv4 egress
be provided and made resilient?

A VPC endpoint asks whether the service can be reached privately. A firewall or proxy determines which destinations and behaviours are allowed. Observability identifies the originating workload and processing zone. FinOps reveals traffic that is expensive only because it used general-purpose NAT.

Private workloads still need exits because useful software has dependencies.

The goal is to make each exit intentional.

A NAT gateway should be the path for traffic that genuinely requires translated public IPv4 egress, not the drain into which every unidentified outbound packet disappears.

Use private routing for internal services, endpoints for suitable AWS services, native IPv6 where supported and NAT64 where IPv6 workloads must reach IPv4.

Choose zonal, Regional or centralised public NAT for the public IPv4 dependencies that remain.

Then monitor, price and test that exit as part of the platform.

Give the workload a deliberate exit.

Put the correct road signs around it.

And keep the receipt.

References and further reading

RFC 5737: IPv4 Address Blocks Reserved for Documentation Defines the public IPv4 example ranges used in this article.
NAT gateways Defines public and private NAT gateway connectivity and address-translation behaviour.
Regional NAT gateways for automatic multi-AZ expansion Covers VPC-level Regional NAT, automatic and manual modes, route tables, zonal expansion and IP limits.
NAT gateway basics Covers zonal availability, bandwidth, packet rates, connection capacity, NAT64 and security limitations.
Amazon VPC pricing Documents zonal and Regional NAT Gateway-hour, data-processing and data-transfer charging.
Troubleshoot NAT gateways Covers routes, state, security rules, ACLs, timeouts, protocol support and connection limits.
NAT gateway metrics and dimensions Defines the NAT Gateway ID and Availability Zone metric dimensions.
Gateway endpoints Explains private S3 and DynamoDB routes without a NAT or internet gateway.
Gateway endpoints for Amazon S3 Covers same-Region behaviour, route-table associations and connectivity limitations.
AWS PrivateLink and interface endpoints Covers private service connectivity through endpoint network interfaces.
Deploy private EKS clusters with limited internet access Lists endpoint and registry dependencies for EKS nodes and workloads without general internet access.
Using NAT Gateway for centralised IPv4 egress Describes shared egress through Transit Gateway and an egress VPC.
IP addressing for VPCs and subnets Explains public and private IPv6 behaviour, including internet-gateway restrictions for private IPv6.