content-experimentation-best-practices
sanity-io/agent-toolkit
A/B testing and experimentation guidance for content-driven products and CMS workflows.
What is content-experimentation-best-practices?
Comprehensive reference for designing, executing, and analyzing content experiments. Covers hypothesis framing, statistical foundations, metrics selection, sample sizing, CMS integration patterns, and common pitfalls. Use when planning experiments, setting up variants, choosing success metrics, or interpreting results.
- Provides hypothesis framework and experiment design patterns
- Guides metric selection and sample size calculation
- Explains statistical significance, p-values, and confidence intervals
- Documents CMS-managed variant strategies and field-level testing
- Catalogs 17 common experimentation mistakes and how to avoid them
- Supports multivariate testing and Bayesian analysis approaches
How to install content-experimentation-best-practices
npx skills add https://github.com/sanity-io/agent-toolkit --skill content-experimentation-best-practicesHow to use content-experimentation-best-practices
- 1.Identify the experimentation problem (design, statistics, CMS integration, or pitfalls)
- 2.Reference the matching guide in references/ (experiment-design.md, statistical-foundations.md, cms-integration.md, or common-pitfalls.md)
- 3.Apply the relevant principles to your experiment setup or analysis
- 4.Validate your approach against the documented best practices and common mistakes
Use cases
- Setting up A/B testing infrastructure for a content-driven product
- Designing an experiment to test headline or copy variations
- Calculating required sample size and statistical power for a planned test
- Integrating experimentation capabilities into a headless CMS workflow
- Interpreting test results and determining statistical significance
- Product managers planning experiments
- Data analysts designing and analyzing tests
- Frontend engineers building experimentation systems
- CMS developers integrating variant management
- Content teams evaluating content changes
content-experimentation-best-practices FAQ
A/B testing compares two variants (A vs B) to determine which performs better. Multivariate testing tests multiple variables simultaneously to find optimal combinations.
Results are statistically significant when the p-value is below your chosen threshold (typically 0.05), meaning there's less than a 5% chance the results occurred by random chance. See statistical-foundations.md for confidence intervals and power analysis.
Start with high-impact, low-effort changes like headlines, calls-to-action, or key messaging. Reference experiment-design.md for hypothesis framing and metric selection guidance.
No—stopping early (peeking) introduces bias and inflates false positive rates. Calculate required sample size upfront and run the full test duration. See common-pitfalls.md for details.
Reference cms-integration.md for patterns on CMS-managed variants, field-level variants, and external platform integration strategies.
Full instructions (SKILL.md)
Source of truth, from sanity-io/agent-toolkit.
name: content-experimentation-best-practices description: Content experimentation and A/B testing guidance covering experiment design, hypotheses, metrics, sample size, statistical foundations, CMS-managed variants, and common analysis pitfalls. Use this skill when planning experiments, setting up variants, choosing success metrics, interpreting statistical results, or building experimentation workflows in a CMS or frontend stack.
Content Experimentation Best Practices
Principles and patterns for running effective content experiments to improve conversion rates, engagement, and user experience.
When to Apply
Reference these guidelines when:
- Setting up A/B or multivariate testing infrastructure
- Designing experiments for content changes
- Analyzing and interpreting test results
- Building CMS integrations for experimentation
- Deciding what to test and how
Core Concepts
A/B Testing
Comparing two variants (A vs B) to determine which performs better.
Multivariate Testing
Testing multiple variables simultaneously to find optimal combinations.
Statistical Significance
The confidence level that results aren't due to random chance.
Experimentation Culture
Making decisions based on data rather than opinions (HiPPO avoidance).
References
Start with the reference that matches the current problem, such as design, statistics, CMS integration, or pitfalls. See references/ for detailed guidance:
references/experiment-design.md— Hypothesis framework, metrics, sample size, and what to testreferences/statistical-foundations.md— p-values, confidence intervals, power analysis, Bayesian methodsreferences/cms-integration.md— CMS-managed variants, field-level variants, external platformsreferences/common-pitfalls.md— 17 common mistakes across statistics, design, execution, and interpretation
Related skills
More from sanity-io/agent-toolkit and the wider catalog.
content-modeling-best-practices
Structured content modeling guidance for schema design, reusability, and multi-channel delivery in headless CMSes.
portable-text-conversion
Convert HTML and Markdown into Portable Text blocks for Sanity content migration and pipelines.
portable-text-serialization
Render and serialize Portable Text across React, Vue, Svelte, Astro, HTML, Markdown, and plain text.
sanity-best-practices
Sanity development best practices for schema design, GROQ, TypeGen, Visual Editing, and framework integrations.
sanity-migration
Plan, implement, and validate CMS-to-Sanity migrations with structured ETL workflows.
seo-aeo-best-practices
SEO and AEO best practices for metadata, structured data, sitemaps, and AI answer engine optimization.