Bulk Operations: Designing APIs That Handle Thousands of Items Efficiently

How Shopify Imports 10 Million Products Per Day

When large enterprises migrate to Shopify, they bring catalogs with hundreds of thousands of products. Making individual API calls for each product would take days at 50ms per call. Shopify's bulk API accepts up to 50,000 operations in a single GraphQL mutation, processes them asynchronously, and provides a polling URL for results. Their bulk import system handles over 10 million product operations daily. The key insight: bulk APIs are not just about batching - they require fundamentally different error handling, progress tracking, and transaction semantics than individual endpoints.

Individual vs Bulk Performance

Importing 10,000 Products:

  Individual API Calls:
  10,000 requests x 50ms avg = 500 seconds (8.3 minutes)
  10,000 TCP connections
  10,000 JSON parses
  10,000 auth checks
  10,000 database transactions
  Network overhead: ~40 bytes headers per request = 400 KB

  Bulk API (single request):
  1 request x 2 seconds = 2 seconds
  1 TCP connection
  1 JSON parse
  1 auth check
  1 database transaction (batch INSERT)
  Network overhead: 40 bytes

  Speed improvement: 250x faster
  Resource savings: 99.99% fewer connections

Bulk API Design Patterns

Pattern 1: Synchronous Bulk (small batches, <1000 items):

  POST /api/products/bulk
  {
    "operations": [
      { "action": "create", "data": { "name": "Widget A", "price": 10 } },
      { "action": "create", "data": { "name": "Widget B", "price": 20 } },
      { "action": "update", "id": 42, "data": { "price": 15 } }
    ]
  }

  Response (per-item results):
  {
    "results": [
      { "index": 0, "status": "created", "id": 101 },
      { "index": 1, "status": "error", "error": "Duplicate SKU" },
      { "index": 2, "status": "updated", "id": 42 }
    ],
    "summary": { "total": 3, "succeeded": 2, "failed": 1 }
  }

Pattern 2: Async Bulk (large batches, >1000 items):

  POST /api/products/bulk-import
  { "file_url": "https://s3.../products.csv" }

  Response: 202 Accepted
  { "job_id": "import_abc", "status_url": "/jobs/import_abc" }

  GET /jobs/import_abc
  {
    "status": "processing",
    "progress": { "processed": 4500, "total": 10000, "errors": 12 },
    "started_at": "2024-01-15T10:00:00Z"
  }

  When complete: results available as downloadable file
  { "status": "completed", "results_url": "/jobs/import_abc/results.csv" }

Transaction Semantics: All-or-Nothing vs Partial

Design Decision: What happens when 1 item fails?

  All-or-Nothing (atomic):
  Items: [A: OK, B: OK, C: FAIL, D: OK]
  Result: ALL rolled back. Nothing saved.
  Use when: data must be consistent as a group
  Example: batch financial transactions

  Partial Success (best effort):
  Items: [A: created, B: created, C: FAILED, D: created]
  Result: A, B, D saved. C returns error.
  Use when: items are independent
  Example: product import, user invitations

  Recommendation: Default to partial success.
  Return per-item status codes so clients know exactly
  which items failed and can retry only those items.

Bulk API Best Practices

• Set and document max batch size (100-10,000 depending on item complexity). Reject oversized batches with 413 Payload Too Large.
• Return per-item results with index positions so clients can map results to their original items.
• Support mixed operations in one batch: creates, updates, and deletes together for efficiency.
• Use database batch operations internally: batch INSERT, COPY (PostgreSQL), or bulk_write (MongoDB) - not loops.
• Validate all items upfront before processing any: fail fast on schema errors rather than processing half and failing.
• For async bulk: provide progress updates, support cancellation, and set result expiration (7 days).

Interview Tip

Bulk operations come up when designing import/export features, admin tools, or data migration systems. The key trade-off is synchronous vs async: sync for small batches with immediate results, async for large batches with job tracking. Always mention per-item error reporting (clients need to know which items failed), the transaction semantics decision (atomic vs partial success), and max batch size limits. For the database layer, mention batch INSERT and COPY for PostgreSQL - never loop through items one by one. The advanced insight: async bulk import from a file URL (S3) is better than accepting the data in the request body, because it avoids request timeouts and allows the client to upload the file separately.

<

/blockquote>

Key Takeaway

Bulk endpoints reduce thousands of individual API calls into one efficient request. Return per-item results with index positions so clients know what succeeded and what failed. Use synchronous bulk for small batches (<1000) and asynchronous jobs for large imports. Use database batch operations internally for performance. Always document max batch size and transaction semantics.