A/B Testing Program Design

Create a comprehensive A/B testing program for [Company/Product]. Include: 1) Experimentation culture — how to build a hypothesis-driven culture, getting executive buy-in, celebrating learning (not just wins). 2) Statistical foundations — sample size calculation, minimum detectable effect, statistical significance (frequentist vs Bayesian), test duration estimation, multiple comparison correction. 3) Experiment design — hypothesis template ('If we [change], then [metric] will [direction] by [amount] because [reasoning]'), experiment brief format, success criteria definition. 4) Implementation framework — feature flag architecture, traffic allocation, randomization unit selection (user vs session vs page), holdout groups, interaction effects between simultaneous tests. 5) Analysis methodology — pre-experiment checks (SRM, novelty effect, instrumentation validation), primary and secondary metrics, guardrail metrics, segmentation analysis, heterogeneous treatment effect detection. 6) Common pitfalls and solutions — peeking problem (sequential testing), Simpson's paradox, survivorship bias, network effects, contamination between test groups. 7) Tooling evaluation — Optimizely, LaunchDarkly, Statsig, Eppo, GrowthBook — comparison matrix. 8) Governance — experiment review board, ethical considerations, experiment prioritization framework. 9) Knowledge management — experiment results repository, meta-analysis methodology, compounding learning effects.