Network Debugging Along the Request Path

“The API is down” and “the site is slow” are where most network debugging starts, and the worst response is to guess: restart the service, flush DNS, roll back the last deploy, and hope. Interviewers ask debugging questions to see whether you have a method. The method that works is to follow the request along its path, find the first step that fails, and let everything before that step drop out of the investigation.

This note runs six different failures through the same curl command, shows which layer each one stopped at, then checks layers individually with dig, nc and openssl.

The path one request takes

Every HTTPS request goes through the same steps, and each one fails in its own recognisable way:

1. Name resolution. The hostname becomes an IP address, through the hosts file, OS cache and DNS resolvers.
2. TCP connect. A handshake with that address and port. Fails as refused (something answered “nothing here”) or timed out (nothing answered).
3. TLS. Protocol negotiation and certificate checks. Fails on trust, hostname or expiry.
4. HTTP. The request goes out and a response comes back, possibly through a CDN, load balancer and reverse proxy, each of which can answer on the app’s behalf.
5. The application. It does the work, which may mean waiting on databases and other services.

When something fails, the question is never “what’s wrong?” in general. It’s “how far did the request get?”

One command that shows where it stopped

curl can print a timer for each phase plus its exit code. Put the format in a file:

dns=%{time_namelookup}s tcp=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s total=%{time_total}s http=%{http_code} exit=%{exitcode}\n

The timers are cumulative from the start, and a phase that never completed shows zero. Here are six requests against a local HTTPS server built to fail in different ways (its certificate covers only localhost):

curl -sS -o /dev/null -w @fmt.txt https://api.shop.invalid/

curl: (6) Could not resolve host: api.shop.invalid
dns=0.000000s tcp=0.000000s tls=0.000000s ttfb=0.000000s total=0.010708s http=000 exit=6

curl -sS -o /dev/null -w @fmt.txt https://localhost:8448/

curl: (7) Failed to connect to localhost port 8448 after 0 ms: Couldn't connect to server
dns=0.000016s tcp=0.000000s tls=0.000000s ttfb=0.000000s total=0.000385s http=000 exit=7

curl -sS -o /dev/null -w @fmt.txt --cacert cert.pem --resolve api.shop.test:8447:127.0.0.1 https://api.shop.test:8447/

curl: (60) SSL: no alternative certificate subject name matches target host name 'api.shop.test'
dns=0.000014s tcp=0.000332s tls=0.000000s ttfb=0.000000s total=0.008894s http=000 exit=60

curl -sS -o /dev/null -w @fmt.txt --cacert cert.pem https://localhost:8447/crash

curl: (52) Empty reply from server
dns=0.000012s tcp=0.000285s tls=0.001737s ttfb=0.000000s total=0.003356s http=000 exit=52

curl -sS -o /dev/null -w @fmt.txt --cacert cert.pem https://localhost:8447/slow
curl -sS -o /dev/null -w @fmt.txt --cacert cert.pem https://localhost:8447/

dns=0.000010s tcp=0.000250s tls=0.001506s ttfb=1.516588s total=1.516782s http=200 exit=0
dns=0.000029s tcp=0.000521s tls=0.003716s ttfb=0.005113s total=0.005200s http=200 exit=0

Reading them:

The slow case is the one people misread most often. DNS, TCP and TLS were all done within 2 ms; the remaining 1.5 seconds passed between sending the request and receiving the first byte. That’s server time, so the network team can’t help. Keep a healthy baseline like the last line for comparison, ideally captured from the same machine before anything broke.

Other exit codes worth recognising: 28 is a timeout (check which timers are zero to see which phase it was in), 35 is a TLS handshake failure before certificate checks (often a protocol or cipher mismatch), and 56 means the connection broke while receiving.

Checking one layer at a time

When the summary points at a layer, use a tool that tests only that layer, so nothing above it can confuse the result.

DNS. dig shows exactly what a resolver returns, and which one you asked:

dig +short example.com A
dig +short example.com AAAA

172.66.147.243
104.20.23.154
2606:4700:8dd5:72db:f2c5:0:ef6b:ff98

Remember that applications resolve through the OS, which also reads the hosts file, so compare with what the app actually uses. If clients behave differently over IPv4 and IPv6, curl -4 and curl -6 force one or the other.

TCP. nc -z opens a connection and closes it, with no TLS or HTTP:

nc -vz -G 2 127.0.0.1 8447
nc -vz -G 2 127.0.0.1 8448

Connection to 127.0.0.1 port 8447 [tcp/*] succeeded!
nc: connectx to 127.0.0.1 port 8448 (tcp) failed: Connection refused

TLS. openssl s_client performs only the handshake and shows the certificate and the verification result. Always pass -servername, or a server hosting several sites may show you a default certificate:

openssl s_client -connect example.com:443 -servername example.com -brief </dev/null

CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Peer certificate: CN=example.com
Verification: OK

HTTP. curl -v shows every request and response header, including which proxy or CDN answered. --resolve host:port:ip pins a hostname to a specific address, which lets you test one backend directly or a new server before DNS points at it, while TLS still checks the real name.

A worked case: DNS fine, TLS broken

Suppose users report certificate warnings on api.shop.test after it moved to a new server. Following the path:

1. DNS: dig returns the new server’s address. DNS is doing its job, so changing records again won’t help.
2. TCP: nc -vz to that address on 443 succeeds. The firewall and the process are fine.
3. TLS: openssl s_client with the right server name:

openssl s_client -connect 127.0.0.1:8447 -servername api.shop.test -verify_hostname api.shop.test -CAfile cert.pem -brief </dev/null

depth=0 CN=localhost
verify error:num=62:hostname mismatch
CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Peer certificate: CN=localhost

The new server presents a certificate for a different name, probably a default one because the real certificate wasn’t installed or wasn’t mapped to this hostname. Three commands narrowed “the API is broken” to one configuration item, and each step ruled out a team: DNS, networking and the application all look fine.

Mistakes that waste hours

• Testing from the wrong place. Your laptop has different DNS, routes, proxies and firewall rules from the failing pod or the customer’s network. Run the commands from where the failure happens.
• Using ping as a health check. ICMP is often blocked, and a reply says nothing about the TCP port, the certificate or the application.
• Trusting the browser alone. It caches DNS, redirects, HSTS rules and sometimes whole responses through service workers. Reproduce with curl.
• Changing several things at once. If the problem goes away, you don’t know which change fixed it, or whether it just happened to stop.

Interview practice

Users in one region report intermittent timeouts calling your API, while other regions are fine. Requests that succeed are fast. How do you investigate?

Answer

Intermittent and region-specific points at something on the path that only that region uses: a resolver, a CDN edge, a load balancer node, a route, or one bad backend that only that region’s traffic reaches. I’d start by reproducing from inside the affected region, a cloud VM there or a customer’s network, rather than from my desk, and run the curl timing command in a loop, logging the remote IP of each attempt with %{remote_ip}.

The timers show which phase hangs. If failures sit at the connect phase and always involve the same one or two IPs, it’s a specific edge node or backend, and I’d pin each address with --resolve to confirm which one. If DNS sometimes returns an old or unexpected address, it’s resolution or a stale record. If connect and TLS finish but the first byte never comes, the request reached something that didn’t answer, so I’d check that region’s load balancer target health and the backend logs for those request times. If drops look random across IPs, I’d compare packet loss on the route with mtr from the region and check whether IPv6 behaves differently from IPv4. At each step I’d write down what’s been ruled out, so the next person doesn’t repeat it.

Next reading

• Networking interview questions
• Networking MCQs
• TLS encryption and server identity
• Network timeouts and operation deadlines
• Reference: MDN: HTTP