AWS VPC Routes and Network Reachability

When an instance in a VPC can’t reach something, people tend to jump straight to security groups. Often the problem sits one layer below: the packet had nowhere to go, or it went somewhere and the reply had no way back. VPC routing is a small set of rules, and once you can apply them by hand, most “it can’t connect” problems become a short checklist.

What a route table is and who uses it

A VPC has an address range, say 10.0.0.0/16, carved into subnets such as 10.0.1.0/24. Each subnet is associated with exactly one route table. If you don’t associate one explicitly, the subnet uses the VPC’s main route table, which is a common source of surprises when someone edits the main table and changes every unassociated subnet at once.

A route table is a list of entries, each a destination CIDR and a target:

• local for the VPC’s own range. It’s added automatically, can’t be deleted, and is why every subnet in a VPC can reach every other subnet by default (security groups and NACLs permitting).
• An internet gateway (igw-...) for internet traffic from public subnets.
• A NAT gateway (nat-...) for outbound internet from private subnets.
• A peering connection (pcx-...), transit gateway (tgw-...) or virtual private gateway (vgw-...) for other VPCs and on-premises networks.
• Gateway VPC endpoints for S3 and DynamoDB, which appear as prefix-list routes (pl-...).

The route table only decides where a packet goes next. It’s evaluated when traffic leaves an instance’s subnet; there’s no hop-by-hop path inside the VPC to reason about.

The most specific route wins

When several entries match a destination, the router picks the one with the longest prefix, the most specific. Order in the table doesn’t matter. Here’s that rule in a few lines of Python with the standard ipaddress module, using a private subnet’s route table:

# Longest-prefix match over a route table, the way a VPC router picks a target.
import ipaddress

private_rt = [
    ("10.0.0.0/16",    "local"),
    ("10.1.0.0/16",    "pcx-0a1b2c (peering to analytics VPC)"),
    ("10.1.8.0/24",    "tgw-07f3e1 (transit gateway)"),
    ("0.0.0.0/0",      "nat-0d4e5f (NAT gateway)"),
]

def lookup(table, dest):
    ip = ipaddress.ip_address(dest)
    matches = [(ipaddress.ip_network(c), t) for c, t in table if ip in ipaddress.ip_network(c)]
    if not matches:
        return "no route: dropped"
    net, target = max(matches, key=lambda m: m[0].prefixlen)
    return f"{str(net):<12} -> {target}"

for dest in ["10.0.42.7", "10.1.3.20", "10.1.8.99", "52.94.236.248", "172.31.5.5"]:
    print(f"{dest:<15} {lookup(private_rt, dest)}")

Output from python3 routes.py:

10.0.42.7       10.0.0.0/16  -> local
10.1.3.20       10.1.0.0/16  -> pcx-0a1b2c (peering to analytics VPC)
10.1.8.99       10.1.8.0/24  -> tgw-07f3e1 (transit gateway)
52.94.236.248   0.0.0.0/0    -> nat-0d4e5f (NAT gateway)
172.31.5.5      0.0.0.0/0    -> nat-0d4e5f (NAT gateway)

Walk through the rows. 10.1.8.99 matches both 10.1.0.0/16 and 10.1.8.0/24, and the /24 wins, so that one subnet goes via the transit gateway while the rest of 10.1.0.0/16 uses peering. That’s a legitimate way to steer a slice of traffic, and an easy way to break it by accident.

The last row is the sneaky one. 172.31.5.5 is a private address, maybe a database in another VPC that nobody added a route for. It still matches 0.0.0.0/0, so it’s sent to the NAT gateway, which forwards it to the internet gateway, where a private address goes nowhere. The connection times out. A missing specific route rarely looks like “no route”; with a default route present, it looks like a timeout.

A real route table has another rule the model skips: you can’t add a route more specific than the VPC’s local range that overrides it, except to send traffic through a middlebox such as a firewall endpoint.

Public and private are just routes

There’s no checkbox that makes a subnet public. A subnet is public because its route table has 0.0.0.0/0 -> igw-.... An instance in it can reach the internet only if it also has a public IPv4 address (or an Elastic IP), because the internet gateway translates between the instance’s private address and its public one. A private IP alone behind an IGW route can’t talk to the internet.

A private subnet has no route to an internet gateway. If it needs outbound access (pulling packages, calling a SaaS API), its default route points at a NAT gateway that lives in a public subnet. Inbound connections from the internet can’t start a conversation through NAT, which is the point.

A typical two-tier layout in one Availability Zone looks like this:

Public subnet 10.0.1.0/24 (route table rt-public)
  10.0.0.0/16  local
  0.0.0.0/0    igw-0123          <- load balancer, NAT gateway live here

Private subnet 10.0.11.0/24 (route table rt-private-a)
  10.0.0.0/16  local
  0.0.0.0/0    nat-0d4e5f        <- app servers live here

For each additional Availability Zone you’d add a public and private subnet and usually a NAT gateway per zone, with each zone’s private route table pointing at its own NAT gateway.

Reachability is a round trip

A route to the destination is half the job. The reply needs a route back. Suppose your app VPC (10.0.0.0/16) is peered with an analytics VPC (10.1.0.0/16), and you added 10.1.0.0/16 -> pcx-0a1b2c to your private route table. Connections still time out. The analytics subnet’s route table has no entry for 10.0.0.0/16, so replies go to its default route instead. Both sides need routes.

Then the filters: the destination’s security group must allow the inbound port from your source, and NACLs on both subnets must allow the traffic in both directions, including the ephemeral port range for replies because NACLs are stateless. That’s covered in the security groups versus NACLs note below.

And a few limits that catch people:

• Peering isn’t transitive. If A peers with B and B peers with C, A can’t reach C through B, even with routes. You’d peer A and C directly, or use a transit gateway, which does route between attachments.
• Overlapping CIDRs can’t be peered. Two VPCs both using 10.0.0.0/16 can’t be connected by peering. Plan address ranges early.
• A route can be a blackhole. If you delete a NAT gateway or peering connection, routes that pointed at it stay in the table with the status blackhole, and traffic matching them is dropped.

Debugging an instance that can’t connect

When an app server can’t reach something, go through it in this order:

1. Resolve the name. dig api.partner.example from the instance. Is the address what you expect, public or private?
2. Find the route. Look up the instance’s subnet, then its route table (aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=subnet-...), and apply longest prefix match by hand. Check the route’s state isn’t blackhole.
3. Check the way back. For anything inside AWS, check the destination subnet’s route table for a route to your range.
4. Check the filters. Security group on the destination, outbound rules on the source, NACLs on both subnets.
5. Let AWS check it. VPC Reachability Analyzer takes a source and destination and reports the hop where the path fails, naming the route table or rule. VPC Flow Logs show whether packets were ACCEPTed or REJECTed at an interface.

The CLI filter above is illustrative; the IDs are made up.

Interview practice: the private subnet that can’t reach S3

An app server in a private subnet uploads files to S3. It worked last week. Since then, the team deleted their NAT gateway to save money and added an S3 gateway endpoint instead. Uploads to the bucket now work, but pip install fails with a timeout. Then someone notices that uploads to a bucket in a different Region also fail. Explain all three behaviours.

Answer

A gateway endpoint for S3 adds a route to the subnet’s route table whose destination is the S3 prefix list for that Region and whose target is the endpoint. S3 addresses in that Region are a more specific match than 0.0.0.0/0, so S3 traffic goes through the endpoint without needing a NAT gateway. That’s why uploads work.

pip install goes to PyPI on the internet. Its addresses only match the default route, which still points at the deleted NAT gateway, so the route is now a blackhole and the packets are dropped. That’s the timeout.

The other Region’s bucket fails for the same reason: a gateway endpoint covers S3 in its own Region only, so the other Region’s S3 addresses aren’t in the prefix list and fall through to the blackholed default route. Fixes: keep a NAT gateway (or a proxy) for genuine internet access, or vendor dependencies into a private artifact store; for cross-Region S3, route through NAT or use S3 interface endpoints and replication. I’d also mention cleaning up the blackhole route so the failure is obvious.

Keep going

• AWS interview questions
• AWS MCQs
• AWS NAT gateways and private outbound access
• AWS security groups versus network ACLs
• Configure route tables (AWS docs)