Time-Series Data: Ingestion, Downsampling & Windowed Aggregation
DatabasesSensors and metrics produce an endless, append-only stream of (time, value) points. Store them in time-ordered chunks, aggregate them in windows as they arrive, and roll old data up into coarser summaries.
A weather station writes a reading every second, but the almanac only prints the daily high and low. Recent data is kept in full detail; old data is kept as summaries.
What is special about time-series. Writes are almost all appends of new timestamps, reads are almost all 'a range of time for one device', and old data is rarely updated or read in detail. A general-purpose table with a random primary key fights all three properties.
Store it by time. Time-series databases (InfluxDB, TimescaleDB, Prometheus, Druid) partition data into chunks by time range. Recent chunks stay hot in memory/SSD, old chunks are compressed (timestamps as deltas, values as small diffs) and dropped when they expire. Deleting a month is dropping one chunk, not running a huge DELETE.
Ingest through a buffer. Millions of devices writing directly to a database is a spike you cannot control. Put a queue or log (Kafka, Kinesis) in front: devices publish, consumers batch-write. Batching 1,000 points per write is the single biggest ingest win. The queue also lets you replay if a consumer is down.
Windowed aggregation. Dashboards do not need every point; they need 'average temperature per minute'. A stream processor groups events into windows. Tumbling windows do not overlap (every minute). Sliding windows overlap (last 5 minutes, updated every minute). Session windows close after a gap of inactivity. Pick the one that matches the question.
Late and out-of-order events. Networks reorder data. Windows are defined by event time (when it happened), not arrival time, and a watermark says 'I will wait this long for stragglers before closing the window'. Later arrivals are either dropped or used to correct the result - decide which.
Downsampling (rollups). Keep raw data for a short period (say 7 days), 1-minute averages for 90 days, hourly for years. Store min, max, sum and count rather than only the average, so you can combine rollups correctly (the average of averages is wrong).
Cardinality is the hidden cost. Every unique combination of tags (device id x region x firmware) is a separate series. A tag with millions of values, such as a user id, can blow up the index. Keep identifiers as fields and only low-cardinality labels as tags.
Alerts. Evaluate rules on the stream ('temperature above 80 for 3 consecutive minutes') rather than polling the database, and de-duplicate so one outage sends one page, not one per sensor.
- Partition by time; drop old data by dropping whole chunks.
- Put a queue in front of ingestion and write in batches.
- Use event time with a watermark, not arrival time, for windows.
- Roll up with min/max/sum/count so rollups can be combined.
- Watch tag cardinality - it is what makes series databases fall over.
Curated Curation & Deep Insights
ScaleDojo CertifiedOur system architects are vetting high-quality, authorized video guides for Time-Series Data: Ingestion, Downsampling & Windowed Aggregation with zero third-party platform links.
We are preparing premium, zero-competitor deep-dives for Time-Series Data: Ingestion, Downsampling & Windowed Aggregation. Authorized reading references will appear automatically.