Why this matters
Time series data is critical in domains like IoT, monitoring, social media, and stock trading.
It consists of data points collected at regular intervals—seconds, minutes, hours, days.
This data type demands efficient storage and fast retrieval at scale.
What is time series data?
Time series data is a sequence of data points collected over time intervals.
Examples include:
- System health metrics
- Video view events on YouTube
- Stock price changes
- Messaging system messages
Storage pattern
A common pattern uses:
- Object storage (e.g., S3) for data log segments
- A low-latency database (e.g., DynamoDB) for indexes
Data is stored in segments, similar to message logs.
Data flow
When data arrives:
- Batch data and append to the currently active segment
- Update the index in DynamoDB
When a segment exceeds size limits:
- Create a new segment in object storage
- Update the index in DynamoDB
When a query is made:
- Retrieve index from DynamoDB for the time interval
- Identify start and end positions
- Fetch the byte range from object storage
Why this pattern?
Object storage handles large, immutable data logs. DynamoDB provides fast, low-latency index lookups.
This combination enables scalable, low-latency access.
Questions
- How does batching improve performance in time series storage?
- Why is object storage ideal for data logs?
- What role does DynamoDB play in time series retrieval?
- How do segment limits trigger new segment creation?
- Can this pattern support real-time queries?
- What are the trade-offs of using object storage over a dedicated time series database?
- How does indexing in DynamoDB support efficient range queries?
- What happens if a segment becomes too large?
Key takeaways
- Time series data is collected at regular intervals.
- Object storage stores data log segments.
- DynamoDB stores indexes for fast lookups.
- Data is batched and appended to active segments.
- Indexes are updated in DynamoDB on each write.
- Queries use indexes to locate data ranges in object storage.
- This pattern enables scalable, low-latency time series storage.
- Dedicated time series databases exist as alternatives.