System Design Uploads and Safe Completion

A user taps “Upload” on a 2 GB video. The progress bar reaches 100%. Is the video ready? Not even close. The bytes may be sitting in storage, but nobody has checked that they’re actually a video, that they aren’t malware, that they’re within the size the user is allowed, or transcoded them into formats a phone can play. “Upload complete” and “ready to show other people” are different states, and a lot of upload bugs come from treating them as one.

An advanced interview answer treats an upload as a workflow with explicit states, and puts each state transition under the control of the component that can actually verify it.

Keep the bytes off your API servers

The naive design streams the file through the application: client to API server, API server to object storage. For small files it works. For video, it means an API server holds a connection open for the whole transfer, and at 20 Mbit/s a 2 GiB file takes about 14 minutes. A thousand concurrent uploads is 20 Gbit/s of traffic flowing through your fleet, which you’d otherwise size for small JSON requests.

The standard design is to let the client upload directly to object storage with a pre-signed URL:

1. The client asks the API: “I want to upload holiday.mp4, 2.1 GB, video/mp4.”
2. The API checks the user is allowed (quota, account status), creates an upload record with status pending_upload, chooses the object key itself (for example quarantine/u_8812/up_5f3a.mp4), and returns a URL signed for that key, valid for 15 minutes.
3. The client sends the bytes straight to storage.

Two details matter. The server, not the client, picks the key, so users can’t overwrite each other’s files or write outside their area. And the signature should constrain what can be sent: S3’s browser POST policies support a content-length-range condition, so a user granted a 2 GB upload can’t push 200 GB. A plain pre-signed PUT can’t enforce a size range, so you check the object’s real size after it arrives, which you’ll do anyway.

The upload as a state machine

Write the states down, and who moves each transition:

The client saying “I’m done” is a hint, not proof. A buggy or malicious client can call “complete” for an upload that never happened. The API should confirm the object actually exists, with the expected size, before moving to uploaded. Many designs skip the client call entirely and trigger on the storage service’s own notification (S3 event notifications to a queue, for example), which is authoritative about what landed.

Storage events, like most notifications, are delivered at least once and can arrive late. So the transition must be idempotent: UPDATE uploads SET status = 'uploaded' WHERE id = $1 AND status = 'pending_upload', where zero rows updated means “already handled” rather than an error.

Large files and flaky networks

A single PUT of a large file has a nasty failure mode on mobile networks: if the connection drops at 90%, the whole transfer starts over. Multipart upload splits the file into parts uploaded independently; a failed part is retried alone, and the client can resume after an app restart if it saved the upload id.

A simulation shows what that’s worth: a 2 GiB video over a 20 Mbit/s connection that drops on average once every ten minutes, with two seconds to reconnect after each drop:

if (need <= nextDrop) { total += need; nextDrop -= need; remaining -= chunk; }
else { total += nextDrop + 2; nextDrop = -Math.log(1 - rand()) * MEAN_GAP; }   // lose the partial chunk, 2 s to reconnect

2 GiB over 20 Mbit/s, a drop every ~10 min on average (clean transfer = 14.3 min):
  single PUT, restart from zero  median  24.8 min   p90  60.4 min
  multipart, 64 MiB parts        median  14.6 min   p90  15.2 min
  multipart, 16 MiB parts        median  14.4 min   p90  14.6 min

The single PUT needs a clean 14-minute window, which a connection that drops every ten minutes rarely provides. Half the users wait 25 minutes or more, and one in ten waits an hour. With 16 MiB parts, a drop costs at most a few seconds of re-sent data, and the upload finishes in close to the clean transfer time. Smaller parts lose less per drop, at the cost of more requests.

Part size has hard limits. S3 parts must be between 5 MiB and 5 GiB (the last part can be smaller), and an upload can have at most 10,000 parts. That sets a floor on part size for big files:

    2 GiB: smallest legal part 5.00 MiB; with 16 MiB parts -> 128 parts
   50 GiB: smallest legal part 5.12 MiB; with 16 MiB parts -> 3200 parts
 1024 GiB: smallest legal part 104.86 MiB; with 105 MiB parts -> 9987 parts

A fixed 16 MiB part size works up to about 156 GiB; beyond that, compute the part size from the file size. Incomplete multipart uploads also keep their parts in storage (and on the bill) until completed or aborted, so add a lifecycle rule that aborts them after a few days.

Nothing is served until it’s validated

The uploaded object sits in a quarantine area, a prefix or a separate bucket that no public URL or CDN can read. A processing worker then runs the checks that make it safe:

• Real type, not claimed type. The Content-Type header and the .mp4 extension are whatever the client said. Inspect the file’s actual bytes (magic numbers, a proper parser) and reject mismatches.
• Size and limits. Confirm the stored size matches what was authorised.
• Malware scanning for documents and archives that other users will download.
• Re-encoding. Transcode videos and re-encode images rather than serving originals. That normalises formats and strips metadata such as GPS coordinates in photo EXIF data, a real privacy leak if you publish originals.
• Content moderation if the product needs it, which may involve a review queue and can take minutes or hours.

Only after these pass does the worker write the derived files to the public location and mark the upload ready. Failures become rejected with a reason the user can see (“unsupported format”, “file too large”), not a silent disappearance.

Serve user content from a separate domain (for example usercontent.example.net rather than your main site). If something slips through, such as an HTML file disguised as an image, it can’t read your main site’s cookies.

Cleaning up what never finishes

Uploads get abandoned. Users close the app, URLs expire, processing crashes. Without cleanup, three kinds of debris accumulate: pending_upload records that will never complete, objects in quarantine that were uploaded but never processed or confirmed, and incomplete multipart parts. A scheduled job expires stale pending records; lifecycle rules delete quarantine objects older than a few days and abort old multipart uploads. Before deleting an object, make sure no ready record points at it, which is easier when published files live in a different location from quarantine.

Interview practice: profile photos at scale

“Design profile photo upload for a social app with 5 million uploads a day. Photos must appear on the user’s profile quickly, and other users must never see an unmoderated image.”

Answer

5 million a day is about 58 uploads a second on average, maybe 200 at peak. Photos are small, a few MB, so bandwidth isn’t the hard part; correctness and the publish step are. The client asks the API for an upload slot; the API creates an upload record (pending_upload), picks a quarantine key, and returns a pre-signed POST limited to 10 MB and image content types, valid for 10 minutes. The client uploads directly to object storage.

A storage event lands on a queue. A worker marks the record uploaded with a conditional update (so duplicate events are harmless), verifies the real image type by parsing it, re-encodes it into three sizes as WebP while stripping EXIF metadata, and runs automated moderation. If it passes, the worker writes the derived images to the public bucket behind the CDN with versioned keys and sets the profile’s photo_upload_id. If moderation is uncertain, the upload waits in a review queue.

For “appears quickly”, the uploader can see their own new photo immediately, served through a short-lived signed URL to the quarantine copy, while everyone else keeps seeing the old photo until the new one is ready. That keeps the experience fast for the uploader without ever showing other users unchecked content. Lifecycle rules clean quarantine after three days, and a metric on time from uploaded to ready tells us when processing falls behind.

Further reading

• System design interview questions
• System design MCQs
• S3 consistency and access control
• Queues and backlog management
• AWS docs: Uploading and copying objects using multipart upload in Amazon S3