Most networking bugs fail loudly. MTU bugs are different: the connection opens, the login page loads, and then a file download or a large API response sits there until it times out. Small packets work, big ones disappear, and nothing in the application logs explains why. Once you’ve seen the pattern you recognise it quickly, so it’s a favourite question for senior infrastructure roles.
Packet size limits, from the wire up
Every link has a maximum transmission unit: the largest IP packet it can carry in one frame. Ethernet’s standard MTU is 1500 bytes. You can see your interfaces’ values with ifconfig on macOS or ip link on Linux:
$ ifconfig en0 | head -1
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
$ ifconfig lo0 | head -1
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384Inside those 1500 bytes, IPv4 takes a 20-byte header and TCP another 20 (both without options). That leaves 1460 bytes of payload per segment, which is why TCP announces a maximum segment size (MSS) of 1460 in its SYN on a normal Ethernet network. For ping, ICMP’s header is 8 bytes, so the largest payload that fits in 1500 is 1472. Those numbers come up constantly, and an interviewer will expect you to derive them rather than recite them.
IPv6 has a 40-byte header, so the TCP MSS on a 1500-byte link drops to 1440. IPv6 also guarantees every link carries at least 1280 bytes, a floor that QUIC and DNS rely on.
What happens when a packet is too big
When a router has to forward a packet onto a link with a smaller MTU, IPv4 gives it two choices, controlled by the Don’t Fragment (DF) bit in the packet’s header.
If DF is clear, the router splits the packet into fragments that fit, and the destination reassembles them. That sounds helpful, and it’s mostly bad. Losing any one fragment loses the whole packet. Reassembly costs memory on the receiver and is a classic denial-of-service target. Many firewalls and NAT devices drop fragments because only the first one carries the TCP or UDP port numbers. And fragments of a large UDP datagram arriving at a load balancer may be sent to different backends.
If DF is set, the router drops the packet and sends back an ICMP “Destination Unreachable, fragmentation needed” message (type 3, code 4) that includes the next hop’s MTU. The sender lowers its packet size for that destination and retries. That feedback loop is path MTU discovery, defined for IPv4 in RFC 1191. Modern TCP stacks set DF on everything and use it. IPv6 removed router fragmentation entirely: routers always drop and send ICMPv6 “Packet Too Big”, and only the sending host may fragment.
Watching path MTU discovery on a real network
You can trigger this yourself with ping’s DF option (-D on macOS, -M do on Linux) and a destination from the reserved documentation range 192.0.2.0/24, which is never routed on the internet, so the packets go to your router and no further.
First, a payload one byte too big for the local interface. The kernel refuses before anything is sent:
$ ping -c1 -D -s 1473 192.0.2.1
ping: sendto: Message too long1473 + 8 + 20 = 1501 bytes, one over en0’s MTU. Now 1472, which fits the interface:
$ ping -c1 -D -s 1472 192.0.2.1
PING 192.0.2.1 (192.0.2.1): 1472 data bytes
556 bytes from 192.168.1.1: frag needed and DF set (MTU 1492)The packet left the laptop and the home router (192.168.1.1) bounced it with an ICMP message reporting a next-hop MTU of 1492. That’s an internet connection running PPPoE, which uses 8 bytes per packet for its own header: 1500 − 8 = 1492. The path is narrower than the first link, and the only way the laptop learns that is this ICMP reply. Checking the boundary:
$ ping -c1 -D -s 1465 192.0.2.1
556 bytes from 192.168.1.1: frag needed and DF set (MTU 1492)
$ ping -c1 -D -s 1464 192.0.2.1
1 packets transmitted, 0 packets received, 100.0% packet loss1464 + 28 = 1492 is accepted by the router and forwarded towards an address that doesn’t exist, so there’s no reply, which is the expected result here. 1465 is one byte over and gets the ICMP message. The same rejection happens for UDP: a Python socket with DF set gets OSError: [Errno 40] Message too long sending 1473 bytes to that address, and sends 1472 without complaint.
Black holes: when the ICMP message never arrives
Now imagine a firewall somewhere on the path that drops all ICMP because someone decided ping was a security risk. The router still drops the oversized DF packet, but the “fragmentation needed” message never reaches the sender. The sender keeps retransmitting a 1500-byte segment that can never get through.
The symptoms are distinctive because they depend on packet size:
- The TCP handshake works, since SYN packets are tiny.
- Small requests and responses work: a health check, a login form, an
lsin an SSH session. - Anything that fills full-size segments stalls: a file download, a large JSON response,
catof a big log over SSH, or sometimes the TLS handshake itself, because the server’s certificate chain often spans several full packets.
That’s a path MTU black hole. When someone says “HTTPS hangs but plain curl to the same host on port 80 works” or “SSH connects but freezes when I run git log”, MTU should be on your list.
Tunnels make the path smaller
Every encapsulation layer eats into the MTU, and it’s the most common cause of these problems inside companies:
- PPPoE: 8 bytes, as above.
- GRE: 24 bytes. VXLAN, used by many container and cloud overlays: 50 bytes.
- WireGuard: up to 80 bytes, which is why its default interface MTU is 1420.
- IPsec: varies with cipher and mode, often 50 to 70 bytes.
If a VPN or overlay network presents a 1500-byte interface to the host but carries traffic over a 1500-byte underlay, inner packets near full size can’t fit once the outer headers are added. Kubernetes networking plugins usually set pod MTUs (for example 1450 for VXLAN) for exactly this reason, and a mismatch between nodes is a known source of odd, size-dependent failures.
Fixing it
In order of preference:
- Let ICMP “fragmentation needed” and ICMPv6 “Packet Too Big” through. Blocking all ICMP breaks path MTU discovery. You can still filter echo requests if policy demands.
- Set the right MTU on tunnel interfaces so hosts never build packets the tunnel can’t carry.
- Clamp the TCP MSS on the tunnel or gateway router. The router rewrites the MSS option in passing SYN packets to a smaller value, so both ends agree to send smaller segments from the start. On Linux this is the iptables
TCPMSStarget with--clamp-mss-to-pmtu. It only helps TCP. - Use packetization-layer discovery (RFC 4821, and RFC 8899 for datagram protocols), which probes with real data and doesn’t depend on ICMP. Linux enables it for TCP with
net.ipv4.tcp_mtu_probing; QUIC implementations do their own probing above a 1200-byte minimum.
DNS is a good example of designing around the problem. Look at any dig output and you’ll see udp: 1232 in the EDNS line. Resolvers agreed in 2020 to advertise a 1232-byte UDP buffer so that responses fit inside IPv6’s 1280-byte minimum after headers, and larger answers fall back to TCP rather than fragmenting.
Practice question: the VPN that breaks downloads
Remote staff on the company VPN can browse the intranet, but downloading build artifacts over HTTPS from an internal server stalls after a few kilobytes. On the office network the same download works. How do you diagnose and fix it?
Answer and reasoning
The size dependence points at MTU. On the VPN, packets travel inside a tunnel, so the usable MTU is less than 1500. If the server sends 1460-byte segments and the “fragmentation needed” ICMP messages are being filtered somewhere (a common default on firewalls), the large segments vanish while small pages still load.
To confirm, from a VPN client I’d run ping -D -s 1472 <server> (or -M do on Linux) and step the size down until replies come back. If 1472 fails silently with no ICMP reply but something like 1380 succeeds, there’s a black hole with an effective MTU around 1408. Capturing on the server with tcpdump would show retransmissions of full-size segments with no matching ACKs.
The durable fix is to set the VPN interface MTU to what the tunnel can carry and clamp the TCP MSS on the VPN gateway, so connections negotiate smaller segments from the SYN onwards. I’d also ask the firewall team to allow ICMP type 3 code 4 (and ICMPv6 type 2) so path MTU discovery works for everything else. Lowering the server’s MTU would work too, but it penalises every non-VPN client, so it’s a last resort.