Back to Skills
Data/ab-test-analyzer

A/B Test Analyzer

Interpret experiment results with statistical rigor and clear ship/no-ship recommendations.

Time Saved

2-3 hrs → 20 min

Compared to doing it manually

Slash Command

/ab-test-analyzer

Type this in Claude to run the skill

The Problem

A/B test results sit in dashboards, but interpreting them requires statistical knowledge most PMs don't have. Bad interpretation leads to shipping losers or killing winners.

What You Get

  • ✓Data quality checks (SRM, novelty effects)
  • ✓Statistical analysis with confidence intervals
  • ✓Segment breakdowns
  • ✓Clear ship/kill/iterate recommendation
  • ✓Guardrail metric monitoring
  • ✓Stakeholder-ready summary

How to use this skill

  1. 1Download the skill file using the button on this page
  2. 2Add the file to your .claude/skills/ folder in your project
  3. 3Type /ab-test-analyzer in Claude to run the skill

Best For

Experiment reviewsGrowth meetingsDecision documentationLearning logs

Related Skills

Metric Framework Builder

/metric-framework-builder

Build comprehensive metrics frameworks using the AARRR pirate metrics or input/output methodology.

Funnel Analyzer

/funnel-analyzer

Diagnose conversion funnel problems and generate data-backed improvement hypotheses.

Experiment Designer

/experiment-designer

Design A/B tests with proper methodology, sample sizes, and success criteria.

A/B Test Designer

/ab-test-designer

Design statistically sound experiments with clear hypotheses and sample size calculations.

Frequently Asked Questions

Check statistical significance first (usually 95% confidence). Look at the primary metric, but also secondary metrics for unintended effects. Segment results by user type — averages can hide important patterns.

Inconclusive results are still results. Options: run longer for more data, test a bigger change, or accept that this variable doesn't matter much. Document learnings either way.

Don't peek at results early, ensure adequate sample size before starting, test one variable at a time, and account for novelty effects. Run for full business cycles to capture weekly patterns.