File Upload APIs: Handling Binary Data in Web Services

How YouTube Handles 500 Hours of Video Uploaded Every Minute

Every minute, creators upload 500 hours of video to YouTube. That is approximately 720,000 videos per day, many of them multi-gigabyte 4K files uploaded over unreliable mobile connections. YouTube cannot route these massive files through their API servers - the servers would be overwhelmed. Instead, YouTube uses resumable uploads: the client uploads directly to Google Cloud Storage in chunks, resuming from the last successful chunk if the connection drops. The API server only handles metadata. This architecture separates the control plane (small, fast API) from the data plane (large, slow file transfer) - a pattern every file upload system should follow.

Three Upload Patterns

Pattern 1: Direct Upload (files < 10MB)

  Client ---multipart/form-data---> API Server ---> Storage
  Simple. Works for avatars, thumbnails, small documents.
  Problem: API server buffers entire file in memory.
  At 100 concurrent uploads of 10MB = 1GB RAM just for uploads.

Pattern 2: Pre-Signed URL (files 10MB - 5GB)

  Client ---GET /upload-url---> API Server
  API Server generates pre-signed S3 URL (expires in 15 min)
  Client <---signed URL---
  Client ---PUT file--------> S3 directly (bypasses API)
  Client ---POST /confirm---> API Server (file_key, metadata)
  API Server verifies file exists in S3, stores reference.

  Benefits:
  - API server handles 0 bytes of file data
  - S3 handles all the heavy lifting (built for this)
  - CDN integration is automatic
  - Works with any cloud: S3, GCS, Azure Blob

Pattern 3: Resumable Chunked Upload (files > 1GB)

  Client splits file: [Chunk 1: 5MB] [Chunk 2: 5MB] ... [Chunk N]
  Upload each chunk with position:
  PUT /upload/session123?offset=0      -> Chunk 1 OK
  PUT /upload/session123?offset=5242880 -> Chunk 2 OK
  PUT /upload/session123?offset=10485760 -> Connection lost!
  ...(reconnect)...
  GET /upload/session123/status -> { "received": 10485760 }
  PUT /upload/session123?offset=10485760 -> Resume Chunk 3
  ...
  Final chunk: server reassembles all chunks.

  Standards: tus protocol (open), Google Resumable Upload API

Pre-Signed URL Architecture

Pre-Signed URL Flow:

  Client                API Server              S3/Cloud Storage
    |                       |                        |
    |--POST /upload/init--->|                        |
    |  {filename, size,     |                        |
    |   content_type}       |                        |
    |                       |--Generate signed URL--->|
    |                       |  (PUT, 15 min expiry,  |
    |                       |   max-size: 100MB,     |
    |                       |   content-type: image/*)|
    |<--{upload_url,--------|                        |
    |    upload_id}         |                        |
    |                       |                        |
    |---PUT file (binary)---|----------------------->|
    |                       |                        |--Store file
    |<--200 OK (ETag)-------|------------------------||
    |                       |                        |
    |--POST /upload/complete|                        |
    |  {upload_id, etag}    |                        |
    |                       |--HEAD obj (verify)---->|
    |                       |<--200 (size, type)-----|
    |                       |  Validate size, type   |
    |                       |  Store metadata in DB  |
    |<--201 {file_id, url}--|                        |

Security Checklist

File Upload Security:

  Threat                  Prevention
  --------------------    ----------------------------------------
  Malicious file type     Validate MIME type server-side (libmagic)
                          Never trust Content-Type header
                          Never trust file extension

  Directory traversal     Generate random filenames (UUID)
                          Never use user-provided filenames in paths
                          Store in flat structure, not user directories

  XSS via uploaded HTML   Serve from separate domain (uploads.cdn.com)
                          Set Content-Disposition: attachment
                          Strip SVG script tags

  Virus/malware           Scan with ClamAV before making available
                          Quarantine until scan completes

  Resource exhaustion     Set max file size per upload (server-enforced)
                          Rate limit uploads per user
                          Pre-signed URL limits size and expiry

  Unauthorized access     Generate signed download URLs (15 min expiry)
                          Check authorization before generating URLs

Interview Tip

File uploads appear in designs for social media, cloud storage, video platforms, and document collaboration. Always separate the control plane (API server handles metadata) from the data plane (cloud storage handles binary data). For files over 10MB, use pre-signed URLs to S3 - never route large files through your API server. For very large files (videos, datasets), describe resumable chunked uploads with the tus protocol. Mention the security concerns proactively: validate MIME types server-side, use random filenames, serve uploads from a separate domain to prevent XSS, and scan for malware. The key architecture insight: your API server should never buffer entire files in memory.

<

/blockquote>

Key Takeaway

Small files (<10MB) can go directly through your API. Large files should upload directly to object storage via pre-signed URLs - never through your API server. Very large files need resumable chunked uploads. Always validate MIME types server-side, generate random filenames, serve uploads from a separate domain, and scan for malware before making files available.