Claude Skill MartBrowse skillsWhat is a Skill?
 
← Back to list

A/B Test Setup

Designs statistically valid A/B tests end to end — hypothesis, sample size, guardrails, and result analysis.

Marketing & SEOIntermediate★ 24,151⑂ 3,405AI score 8/10Last updated: Aug 9, 2026

What it does

Turns "let's just try it" into a disciplined experiment.

  • Hypothesis framework: forces the "Because [data], we believe [change] will cause [outcome] for [audience]" structure, with weak vs. strong examples.
  • Sample sizing: runs the bundled sample_size_calculator.py (two-proportion z-test, α=0.05, 80% power) to produce per-variant sample size and an estimated test duration, plus a quick-reference lift table.
  • Metric design: separates primary, secondary, and guardrail metrics so a "win" doesn't quietly hurt refunds or support load.
  • Execution discipline: pre-launch QA checklist, traffic allocation options (50/50, 90/10, ramping), and an explicit warning about the peeking problem.
  • Analysis: a six-step checklist and a decision table for winners, losers, and inconclusive results.
  • Artifacts: experiment brief, QA checklist, results report, and a prioritized test backlog.

Who it's for

  • Growth marketers optimizing landing or pricing page conversion
  • PMs who struggle to call experiments confidently
  • CRO teams that need a repeatable experiment backlog
  • Teams using PostHog, Optimizely, VWO, LaunchDarkly, or Split

Examples

  1. Pricing page test: "Plan selection is 3% with 60k monthly visits; I want to push annual billing" → you get a brief with hypothesis, ~14k per variant, estimated runtime, and guardrails like refund rate and support tickets.
  2. Settling a copy debate: when the team splits between "Start free" and "Try 14 days", the skill proposes a test design and QA checklist instead of opinions.
  3. Reading results: "7 days in, variant is +12% but p=0.09" → it checks whether you hit the pre-committed sample size, flags early-stopping risk, and estimates how many more days are needed.

· · · Install guide · · ·

Install in the Claude app (no terminal)
  1. Download the ZIP with the button below.
  2. In Claude, open Settings → Capabilities and turn on 'Code execution and file creation'. (one time)
  3. Go to Customize → Skills → + → 'Upload a skill' and upload the ZIP.
↓ Download ZIP
Install in Claude Code

Let Claude do it — paste this into Claude Code

Install the skill I found on Claude Skill Mart.
Copy the .gemini/skills/ab-test-setup folder from the GitHub repo alirezarezvani/claude-skills into my ~/.claude/skills/ab-test-setup/.
When it's done, tell me in one line what this skill can do.

Install with a command instead

git clone https://github.com/alirezarezvani/claude-skills.git /tmp/claude-skills && mkdir -p ~/.claude/skills && cp -r /tmp/claude-skills/.gemini/skills/ab-test-setup ~/.claude/skills/

⚠ This is a third-party skill. Check the source repository before installing.

  1. Open a terminal (Terminal on macOS, Git Bash or WSL on Windows).
  2. Create the skills directory if needed: mkdir -p ~/.claude/skills
  3. Clone the repository into a temp folder: git clone https://github.com/alirezarezvani/claude-skills.git /tmp/claude-skills
  4. Copy the entire skill folder so scripts/ and references/ come along: cp -r /tmp/claude-skills/.gemini/skills/ab-test-setup ~/.claude/skills/
  5. Verify Python 3 is available: python3 --version (install from python.org or via brew install python3 if missing).
  6. Restart Claude Code and try a prompt like "Design an A/B test for my pricing page."
  7. When sizing is needed the skill runs python3 scripts/sample_size_calculator.py --baseline 0.05 --mde 0.20. If it errors with a missing file, re-check step 4.
View source on GitHub ↗License: MIT