A/B Test Statistical Analysis Framework

Create a framework for designing and analyzing A/B tests.

📊 Data & AnalyticsadvancedData Scientist✓ Free

The Prompt

You are an experimentation statistician. Create an A/B testing framework.

Business: [COMPANY]
Product: [PRODUCT]
Typical test volume: [DAILY USERS/EVENTS]
Current process: [DESCRIBE or new]
Tools: [ANALYTICS/EXPERIMENTATION PLATFORM]

1. Test Design:
   - Hypothesis template
   - Sample size calculator methodology
   - Duration estimation
   - Randomization strategy
   - Guardrail metrics selection
2. Statistical Methods:
   - Frequentist approach: z-test, t-test, chi-square (when to use each)
   - Confidence intervals and p-values interpretation
   - Multiple comparison corrections (Bonferroni)
   - Sequential testing: when and how
   - Bayesian alternative: when to prefer
3. Analysis Template:
   - Pre-analysis checklist (SRM, novelty effects, instrumentation)
   - Results summary format
   - Segment analysis approach
   - Practical vs statistical significance
4. Decision Framework: ship/iterate/kill criteria, minimum detectable effect
5. Common Pitfalls: 10 experimentation mistakes with solutions
6. Documentation: test registry template, results library, knowledge sharing

💡 Tip: Replace all [bracketed text] with your specific details before pasting into your AI model.