ads-test
agricidaniel/claude-ads
Design and evaluate paid-ad experiments with rigorous hypothesis testing, sample sizing, and decision rules.
What is ads-test?
A framework for planning and analyzing A/B tests and split tests in paid advertising. Use this when designing experiments, calculating required sample sizes and duration, setting up guardrails, or reading out results with statistical rigor.
- Define causal hypotheses, treatment/control groups, and randomization units
- Calculate sample size and test duration from statistical assumptions
- Identify platform constraints, seasonality, interference, and measurement risks
- Pre-register exclusions, quality checks, and decision thresholds
- Verify assignment integrity and data completeness before effect estimation
- Output experiment setup or readout in versioned JSON with plain-language decisions
How to install ads-test
npx skills add https://github.com/agricidaniel/claude-ads --skill ads-testHow to use ads-test
- 1.State the decision you need to make, your causal hypothesis, and the treatment vs. control
- 2.Specify the randomization unit (user, session, account), population, and primary metric
- 3.Declare your assumptions: minimum detectable effect, statistical power, baseline conversion rate, and test duration
- 4.Check for platform constraints, overlapping experiments, conversion lag, seasonality, and measurement quality issues
- 5.Calculate required sample size and duration; disclose any approximations or simplifications
- 6.Pre-register all exclusions, quality checks, analysis methods, and decision thresholds in writing
- 7.For readout: verify assignment integrity and data completeness, then estimate effect size and confidence intervals
- 8.Return the experiment design or results in JSON format with a clear plain-language recommendation
Use cases
- Plan an A/B test for a new ad creative before launching on Facebook or Google Ads
- Calculate minimum sample size and duration needed to detect a 10% lift in conversion rate
- Review an experiment readout to confirm statistical significance and decide whether to ship a change
- Identify potential confounds (seasonality, overlapping experiments) that could bias results
- Document experiment design and decision rules for stakeholder alignment before running the test
- Performance marketers designing paid-ad experiments
- Product managers planning A/B tests for campaign optimization
- Data analysts setting up experiment infrastructure and analysis
- Growth teams evaluating ad creative, targeting, or bidding changes
ads-test FAQ
The randomization unit is what you randomly assign to treatment or control (e.g., user, session, account). It must match your causal question and avoid interference—if one user sees both variants, you have a problem. Choose the smallest unit that prevents spillover.
Sample size depends on baseline conversion rate, minimum detectable effect, statistical power (usually 80%), and significance level (usually 5%). Use a power calculator with your assumptions. Larger effects and higher baselines require fewer samples; smaller effects require more.
Guardrails are secondary metrics you monitor to avoid harming the business (e.g., revenue per user, unsubscribe rate). If a guardrail drops significantly, you stop the test even if the primary metric wins. Set thresholds before the test starts.
No. Repeatedly checking results and stopping when favorable inflates false-positive rates. Pre-register your stopping rule (fixed duration or pre-specified sample size) and stick to it.
Check statistical significance (p-value < 0.05 or 95% confidence interval excludes zero), effect size (is it practically meaningful?), and assignment integrity (were users actually randomized?). Underpowered tests produce noisy results; don't call them winners.
Full instructions (SKILL.md)
Source of truth, from agricidaniel/claude-ads.
name: ads-test description: "Design and evaluate paid-ad experiments with hypotheses, randomization units, sample-size and duration assumptions, guardrails, platform experiment tools, analysis, and decision rules. Use for A/B test, split test, experiment design, hypothesis, statistical significance, sample size, test duration, or experiment readout."
Paid Media Experiment
- State the decision, causal hypothesis, treatment, control, randomization unit, population, primary metric, guardrails, minimum effect, and stopping rule.
- Check platform constraints, overlapping experiments, conversion lag, seasonality, interference, and measurement quality.
- Calculate sample and duration from declared assumptions; disclose approximations.
- Change one decision surface unless the design explicitly estimates interactions.
- Pre-register exclusions, quality checks, analysis, and decision thresholds.
- For readout, verify assignment integrity and data completeness before estimating effect and uncertainty.
- Return setup or readout in versioned JSON with a plain-language decision.
Do not repeatedly peek and stop on a favorable result, call underpowered noise a winner, or generalize beyond the tested population.
Related skills
More from agricidaniel/claude-ads and the wider catalog.
ads-tiktok
Audit TikTok Ads campaigns, measurement, creative, audiences, and policy compliance.
ads-youtube
Audit YouTube Ads campaigns across inventory, creative, audiences, bidding, and measurement.
ads
Operate professional paid advertising across 12 major platforms with source-grounded audits, strategy, and campaign management.
ads-amazon
Audit Amazon Ads profiles, campaigns, targeting, budgets, ACOS/TACOS metrics, and policy compliance.
blog
Full-lifecycle blog engine with 31 sub-skills for writing, optimization, SEO, and AI citation readiness.
blog-analyze
Audit blog posts on a 5-category 100-point scale with SEO, E-E-A-T, and AI readiness scoring.