a-b-test-design
owl-listener/designer-skills
Design statistically rigorous A/B experiments with clear hypotheses, metrics, and sample sizing.
What is a-b-test-design?
A/B Test Design helps you structure quantitative experiments with controlled variants, measurable outcomes, and statistical rigor. Use this when you need to validate a change through measurable behavior at scale; for qualitative user observation, use test-scenario instead.
- Define structured hypotheses linking changes to expected outcomes with clear rationale
- Set up isolated control (A) and treatment (B) variants, testing one variable at a time
- Select primary metrics that are measurable, relevant, and sensitive to the proposed change
- Identify secondary and guardrail metrics to detect unintended consequences
- Calculate required sample size based on effect size, baseline rate, significance level, and statistical power
- Plan test duration accounting for weekly cycles and minimum run time
How to install a-b-test-design
npx skills add https://github.com/owl-listener/designer-skills --skill a-b-test-designHow to use a-b-test-design
- 1.Write your hypothesis in the form: 'If we [change], then [outcome] will [improve/decrease] because [rationale]'
- 2.Define your control (A) variant as the current state and treatment (B) as the single proposed change
- 3.Select one primary metric that directly measures success and is sensitive to your change
- 4.Identify 2–3 secondary or guardrail metrics to catch unintended side effects
- 5.Calculate sample size using your baseline conversion rate, minimum detectable effect, 95% significance level, and 80% power
- 6.Plan to run the test for complete weeks until sample size is reached, typically 1–2 weeks minimum
- 7.Document all decisions before launching; avoid peeking at results mid-test
- 8.Analyze overall results first, then segment results by user cohorts or conditions
Use cases
- Validate UI/UX changes (button color, copy, layout) by measuring conversion or engagement impact
- Test pricing or offer changes to determine revenue and retention effects
- Evaluate feature additions or removals against baseline behavior
- Optimize onboarding flows by measuring completion rates and downstream engagement
- Assess messaging or positioning changes on sign-up or purchase intent
- Product managers designing experiments
- Data analysts planning statistical tests
- UX/UI designers validating design changes
- Growth and marketing teams optimizing conversion funnels
- Anyone running quantitative feature validation
a-b-test-design FAQ
Use A/B testing when you can measure behavior quantitatively at scale and need statistical confidence. Use qualitative testing (test-scenario) when you need to understand *why* users behave a certain way or when sample sizes are too small.
Sample size depends on your baseline conversion rate, the minimum effect you want to detect, your desired significance level (typically 95%), and statistical power (typically 80%). Use an online calculator or consult a statistician with these parameters.
Stopping the test early when results look positive. This inflates false-positive rates. Always run until your pre-calculated sample size is reached and account for weekly cycles.
No. Test one variable at a time to isolate the effect. Testing multiple changes simultaneously makes it impossible to know which change drove the result.
Run until you reach your target sample size. Account for weekly cycles (run in full weeks) to avoid day-of-week effects. Most tests run 1–2 weeks, but high-traffic products may finish faster.
Full instructions (SKILL.md)
Source of truth, from owl-listener/designer-skills.
name: a-b-test-design
description: Design an A/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use test-scenario.
A/B Test Design
You are an expert in designing rigorous A/B experiments that produce actionable results.
What You Do
You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor.
Test Structure
1. Hypothesis
Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].'
2. Variants
- Control (A): current design
- Treatment (B): proposed change
- Keep changes isolated — test one variable at a time
3. Primary Metric
The single most important measure of success. Must be measurable, relevant, and sensitive to the change.
4. Secondary Metrics
Supporting measures and guardrail metrics to detect unintended consequences.
5. Sample Size
Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%).
6. Duration
Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically.
Common Pitfalls
- Peeking at results before completion
- Too many variants at once
- Metric not sensitive enough to detect change
- Sample size too small
- Not accounting for novelty effects
- Ignoring segmentation effects
When Not to A/B Test
- Very low traffic (insufficient sample)
- Ethical concerns with withholding improvement
- Foundational changes that affect everything
- When qualitative insight is more valuable
Best Practices
- One hypothesis per test
- Document everything before starting
- Don't stop early on positive results
- Analyze segments after overall results
- Share learnings broadly regardless of outcome
Related skills
More from owl-listener/designer-skills and the wider catalog.
accessibility-audit
Audit interfaces against WCAG 2.2 with severity ratings and remediation steps.
accessibility-test-plan
Plan comprehensive accessibility testing with assistive technologies, WCAG criteria, and real user participation.
aesthetic-usability
Apply the Aesthetic-Usability Effect to build trust through visual consistency and polish.
affinity-diagram
Cluster qualitative research data into themed insights across multiple interviews and sources.
animation-principles
Apply animation principles—easing, staging, follow-through—to tune how a single UI motion feels.
card-sort-analysis
Analyze card sort results into proposed groupings and labels for information architecture.