Log Aggregation at Scale: Managing Billions of Log Lines

Cloudflare's 40 Million Log Lines Per Second

Cloudflare processes 40 million HTTP requests per second. Each request generates log data: URL, status code, latency, cache status, geographic origin, security verdict. That is 40 million log lines per second. You cannot SSH into a server and grep for that. You need a pipeline that ingests, indexes, and makes searchable billions of log entries per hour - and you need to keep costs under control while doing it.

The Log Aggregation Pipeline

Log Aggregation Architecture:

  [Server 1]  [Server 2]  ...  [Server 100]
  (app logs)  (app logs)       (app logs)
      |            |               |
  [Filebeat]  [Filebeat]      [Filebeat]    <- Log shippers
      |            |               |           (lightweight agents)
      +------------+---------------+
                   |
             [Kafka Buffer]                  <- Absorbs spikes
              (log topics)                      prevents data loss
                   |
            [Logstash / Fluentd]             <- Parse, transform,
             (parse, enrich)                    add metadata
                   |
            [Elasticsearch]                  <- Index and store
             (search engine)                    query in < 1s
                   |
            [Kibana / Grafana]               <- Visualize, search,
             (dashboard + alerts)               create alerts

Structured Logging: The Foundation

BAD (unstructured text log):
  2024-01-15 14:32:07 INFO User john logged in from 1.2.3.4

  Problems:
  - What is the user ID? Parse with regex... maybe.
  - What was the latency? Not recorded.
  - How do I find all logins with errors? Good luck.

GOOD (structured JSON log):
  {
    'timestamp': '2024-01-15T14:32:07Z',
    'level': 'INFO',
    'service': 'auth-service',
    'event': 'user.login',
    'user_id': 42,
    'ip': '1.2.3.4',
    'latency_ms': 23,
    'trace_id': 'abc-123',
    'environment': 'production',
    'version': 'v2.3.1'
  }

  Now you can query in Kibana:
  service:auth-service AND event:user.login AND latency_ms>1000
  Instant results across all 100 servers.

Cost Management: The #1 Challenge

Log Volume Reduction Strategies:

Strategy          Savings    Trade-off
----------------  ---------  --------------------------
Log levels        90%        No DEBUG in production
Sampling (1%)     99%        Lose rare events
Field filtering   30-50%     Remove verbose fields
Compression       60-70%     CPU cost for decompress
Tier storage      70%        Slower queries for old data

Retention tiering:
  Hot (Elasticsearch):  7 days  - full query speed
  Warm (cheaper nodes):  30 days - slower queries
  Cold (S3 archive):     1 year  - restore to query
  Delete:                > 1 year

Cost comparison (10 TB/day):
  Elasticsearch (self-hosted):  ~$15,000/month
  Datadog Logs:                 ~$30,000/month
  Grafana Loki (S3 backend):    ~$3,000/month
  CloudWatch Logs:              ~$5,000/month

ELK vs Loki vs Datadog

Tool             Storage     Query Speed   Cost      Best For
---------------  ----------  -----------   --------  ----------------
Elasticsearch    Full index  < 1 second    High      Full-text search
Grafana Loki     S3 chunks   2-10 seconds  Low       Cost-sensitive
Datadog          SaaS        < 1 second    Highest   Zero-ops teams
CloudWatch       AWS native  1-5 seconds   Medium    AWS-only stacks

Loki's approach:
  Does NOT index log content. Only indexes labels.
  Stores log chunks as compressed files in S3.
  You query by labels first, then grep within results.
  10x cheaper than Elasticsearch at the cost of
  slower ad-hoc queries.

Interview Tip

When discussing logging in a system design, say: 'I would use structured JSON logging with a correlation trace_id in every log line. Logs ship via lightweight agents (Filebeat) through a Kafka buffer to handle spikes, then to Elasticsearch for indexing. For cost management, I would use log level filtering (INFO+ in production), retention tiering (7 days hot, 30 days warm, 1 year cold in S3), and consider Grafana Loki for cost-sensitive environments where label-based querying is sufficient.'

<

/blockquote>

Key Takeaway

Log aggregation centralizes logs from all servers into one searchable system. Use structured JSON logging, ship via agents and a message bus, index in Elasticsearch. Manage costs with appropriate log levels, sampling, and tiered retention.