Monitoring and Observability Strategy

You are an observability architect. Design a monitoring strategy.

System: [DESCRIBE ARCHITECTURE]
Scale: [REQUESTS/SEC, SERVICES COUNT]
Current monitoring: [DESCRIBE]
SLA target: [UPTIME %]
Team: [SIZE]

1. Three Pillars:
   - Metrics: what to measure (RED method: Rate, Errors, Duration)
   - Logs: structured logging, log levels, aggregation
   - Traces: distributed tracing, span design, sampling

2. Infrastructure Monitoring:
   - Server/container: CPU, memory, disk, network
   - Database: connections, queries, replication lag, lock waits
   - Cache: hit rate, memory, evictions
   - Queue: depth, processing rate, consumer lag

3. Application Monitoring:
   - API endpoints: latency (p50, p95, p99), error rate, throughput
   - Business metrics: signups, transactions, revenue
   - Dependencies: third-party API health, circuit breaker states

4. Alerting Strategy:
   - Alert levels: critical (page), warning (notify), info (log)
   - Alert design: actionable, not noisy, runbook-linked
   - On-call: rotation, escalation, fatigue prevention
   - SLO-based alerting: error budgets, burn rate

5. Dashboards: service overview, golden signals, business metrics, on-call
6. Tool Stack: Datadog, Grafana, PagerDuty, ELK comparison
7. Incident Correlation: connecting metrics, logs, traces for fast debugging
8. Cost Management: data retention, sampling, aggregation