Real-Time Analytics Architecture Design

You are a real-time analytics architect. Design a system.

Use case: [DESCRIBE WHAT NEEDS REAL-TIME]
Data sources: [LIST]
Latency requirement: [SECONDS/MINUTES]
Scale: [EVENTS PER SECOND]
Team: [SIZE]

1. Requirements:
   - Latency: end-to-end processing time target
   - Throughput: events per second, data volume
   - Accuracy: exactly-once vs at-least-once
   - Availability: uptime requirements

2. Architecture Patterns:
   - Lambda architecture: batch + speed layer
   - Kappa architecture: stream-only
   - Recommendation based on requirements

3. Components:
   - Ingestion: Kafka, Kinesis, Pub/Sub (comparison)
   - Processing: Flink, Spark Streaming, ksqlDB (comparison)
   - Storage: time-series DB, OLAP (Druid, ClickHouse, Pinot)
   - Serving: API layer, dashboard, alerts

4. Data Model:
   - Event schema design
   - Aggregation strategy: windows, tumbling, sliding, session
   - Dimension management: slowly changing dimensions

5. Implementation:
   - Kafka topics and partitioning strategy
   - Stream processing logic: filtering, enrichment, aggregation
   - State management: checkpointing, exactly-once semantics

6. Monitoring: pipeline lag, throughput, error rates, data quality
7. Cost: infrastructure sizing, optimization strategies
8. Migration: from batch to real-time transition plan