Object Storage Design: How Amazon S3 Stores Exabytes of Data

300 Trillion Objects: How S3 Became the Internet's Hard Drive

When Amazon launched S3 in 2006, they stored a few thousand objects in the first month. By 2023, S3 stored over 300 trillion objects and processed over 100 million requests per second at peak. Dropbox, Netflix, Airbnb, Slack, Pinterest, and thousands of companies store their data on S3. The durability promise - 99.999999999% (11 nines) - means that if you store 10 million objects, you statistically lose one object every 10,000 years. This is not marketing - it is engineering. S3 achieves this through erasure coding across multiple data centers, continuous data integrity verification, and automatic repair of degraded data. Designing a system like S3 teaches fundamental lessons about durability, availability, and the tradeoffs of simplicity vs flexibility.

Object Storage Mental Model

Object Storage vs File System:

  File System:                    Object Storage:
  /home/                          Flat namespace with key-value API
    /user/
      /photos/                    PUT /bucket/photos/vacation.jpg
        vacation.jpg              GET /bucket/photos/vacation.jpg
        wedding.jpg               DELETE /bucket/photos/vacation.jpg
  
  Has directories: YES            Has directories: NO (/ in key is convention)
  Random writes: YES              Random writes: NO (immutable, rewrite whole)
  File locking: YES               File locking: NO
  Rename: O(1)                    Rename: copy + delete
  Max file size: varies           Max object size: 5TB (S3)
  Access: POSIX (mount)           Access: HTTP REST API
  Scale: single machine           Scale: unlimited (distributed)

  Each S3 object has:
  - Key: 'photos/2024/vacation.jpg' (up to 1024 bytes)
  - Data: the binary blob (up to 5TB)
  - Metadata: content-type, custom tags, encryption info
  - Version ID: (if versioning enabled)

Durability Through Erasure Coding

How S3 Achieves 11 Nines of Durability:

  Simple Replication (3 copies):
  Copy your 1 MB photo to 3 different disks.
  Storage overhead: 3x (3 MB stored for 1 MB file)
  Can survive: 2 disk failures

  Erasure Coding (6+3 scheme - what S3 uses):
  Split 1 MB into 6 data fragments + compute 3 parity fragments.
  Store 9 fragments across 9 different storage nodes.

  Original: [1 MB photo]
      |
  Encode (Reed-Solomon):
      |
  [D1][D2][D3][D4][D5][D6][P1][P2][P3]
   |    |   |   |   |   |   |   |   |
  Node Node Node Node Node Node Node Node Node
   1    2   3    4   5    6   7    8   9

  Recovery: ANY 6 of 9 fragments -> reconstruct original
  Can survive: 3 simultaneous node failures!
  Storage overhead: 1.5x (1.5 MB stored for 1 MB file)
  vs replication: 3x overhead

  S3 also spreads across 3+ Availability Zones:
  AZ-1: [D1, D4, P2]
  AZ-2: [D2, D5, P3]
  AZ-3: [D3, D6, P1]
  Entire AZ can fail -> data still recoverable.

S3 Internal Architecture

Simplified S3 Architecture:

  [Client]
     |
  PUT photos/vacation.jpg
     |
  [S3 Frontend (Load Balancer + Auth + Routing)]
     |
     +-- Authenticate request (IAM/SigV4)
     +-- Determine placement (hash bucket + key)
     |
  [Metadata Service (Index)]
     |
     +-- Store: key -> {chunk_locations, version, created_at}
     +-- Highly available: replicated across AZs
     +-- Handles: LIST, HEAD, versioning, lifecycle
     |
  [Storage Nodes (Data Plane)]
     |
     +-- Erasure-encode the object
     +-- Write fragments to storage nodes across AZs
     +-- Verify checksums (SHA-256 or MD5)
     +-- Return 200 OK when durability threshold met

  Background Processes:
  - Integrity auditor: continuously reads and verifies checksums
  - Auto-repair: detects degraded fragments, rebuilds from parity
  - Lifecycle manager: transitions objects between storage classes
  - Garbage collector: cleans up deleted objects and old versions

Storage Classes and Cost Optimization

S3 Storage Tier Comparison:

  Class              Cost/GB/mo  Retrieval   Min Duration  Best For
  -----------------  ----------  ----------  ------------  ----------------
  Standard           $0.023      Instant     None          Hot data
  Intelligent Tier   $0.023      Instant     30 days       Unknown access
  Infrequent Access  $0.0125     Instant     30 days       Monthly access
  One Zone IA        $0.010      Instant     30 days       Reproducible data
  Glacier Instant    $0.004      Instant     90 days       Quarterly access
  Glacier Flexible   $0.0036     1-12 hours  90 days       Archives
  Glacier Deep       $0.00099    12 hours    180 days      Compliance vaults

  Cost example - 100 TB stored for 1 year:
  Standard:      $27,600/year
  IA:            $15,000/year  (save 46%)
  Glacier Deep:  $1,188/year   (save 96%)

  Best practice: use S3 Intelligent Tiering
  to automatically move objects between tiers
  based on access patterns. Zero management overhead.

Interview Tip

When discussing storage in system design, say: 'For unstructured data (images, videos, logs, backups), I would use object storage like S3. It achieves 11 nines of durability through erasure coding - splitting each object into data and parity fragments spread across multiple availability zones. Any 6 of 9 fragments can reconstruct the original, with only 1.5x storage overhead compared to 3x for simple replication. The metadata layer maps keys to fragment locations and must handle millions of lookups per second. For cost optimization, I would use intelligent tiering to automatically move cold data to cheaper storage classes. S3 now provides strong read-after-write consistency, which eliminated the subtle bugs that existed when it was eventually consistent pre-2020.'

<

/blockquote>

Key Takeaway

Object storage trades file system semantics (directories, partial writes, locks) for unlimited scale and extreme durability. Erasure coding achieves 11 nines of durability at 1.5x storage overhead, far more efficient than 3x replication. The metadata layer is the hardest engineering challenge. Use storage tiering to reduce costs by 50-96% as data ages. S3's simplicity (PUT/GET/DELETE) is its greatest strength.