streams should be yeetabl!

We used Cloudflare's workers platform to build a serverless system that can scale up to billions of events per day, and scale down to zero, while minimizing infrastructure costs and maintenance. Workers Analytics Engine allows for immediate reads at high rps, while enabling efficient queries across massive datasets.

Sequence diagram showing the flow of data and operations in our architecture.

• Serverless Architecture: Utilizing Cloudflare Workers for instant scalability and optimal performance.
• Analytics Engine for Data Storage: Handling high RPS and high cardinality data efficiently.
• Composite Indexing: Custom indexing system for efficient querying of complex threshold conditions.

Estimated Monthly Costs:

• 10,000 events per day (300,000 per month): $0 (Included in free tier)
• 1 million events per day (30 million per month):
  • Non-batched: $11
  • Batched (2000 events per call): $5
• 1 billion events per day (30 billion per month):
  • Non-batched: $16,500
  • Batched (2000 events per call): $7,501.50

These estimates are based on current Cloudflare pricing. Batching 2000 events per call significantly reduces costs. Actual monthly costs may vary based on processing complexity and event volume.

Note: Analytics Engine is currently free, which further reduces the overall cost of the system ($1.50 to handle 1 billion events per day.).

Impact Analysis:

Cost Reduction:

• At 1 million events/day: 54.5% cost reduction
• At 1 billion events/day: 54.5% cost reduction