Что продакт должен знать про А/Б-тесты?

What A/B tests are (and what they are not)

• A/B test is a measurement framework, not a tool that automatically increases metrics.
• Its purpose is to determine whether a change caused a real effect or if results are random noise / sampling coincidence.

Why businesses need A/B tests

• Validate ideas early: confirm whether a proposed product change works or fails.
• De-risk rollouts in mature products: detect whether small metric movements (e.g., +0.5%) are real or just fluctuation.
• Prevent losses by testing on a subset before full release.

Why “before vs after” is insufficient

• External factors (seasonality, marketing changes, geopolitics/news, other team releases) make periods incomparable.
• Example: a mortgage calculator on a bank site during a period when key interest rates rose, affecting conversions—so the drop may be unrelated to the calculator.

MVP vs A/B testing

• MVP: launching from “near zero” / new product to establish baseline (no meaningful comparator like A vs B).
• A/B: comparing version A vs B when at least one baseline exists.
• After MVP proves the product concept, A/B tests can optimize parts of it.

Statistical foundations (key terms used)

Sampling error

Tests rely on sampling error: different samples from the same population can produce different observed metric values even when there’s no real effect.

Type I / Type II errors

• Type I error (“false positive”): you find significance but there is no real effect.
• Type II error (“false negative”): the real effect exists but is missed.

Planning controls / thresholds / decision rules (framework)

• Alpha (α): significance threshold, commonly 5% by default.
• p-value (“Pv indicator” in subtitles): compared against α to decide significance.
• Power / beta (β): set during experiment design to control Type II error.
• MDE (Minimal Detectable Effect) is central for sample size:
  • Smaller MDE ⇒ larger required sample
  • Larger expected effect (e.g., major growth) ⇒ smaller sample needed

Key metrics & KPI-related concepts mentioned

• Primary metric examples:
  • conversion rate (CR)
  • retention
  • treatment/engagement-like metrics
• Secondary metric examples:
  • “warning/invariant/guardrail” metrics (e.g., ensure nothing breaks)
• Long-term metrics caution:
  • LTV over ~6 months may require long observation windows; propose using shorter proxy/early metrics when possible.

Concrete guidance: minimum audience / duration (rules of thumb)

• There’s no universal “minimum user count” answer; it depends on:
  • MDE
  • metric variance/dispersion
  • α and power settings
• Examples:
  • Large lift (e.g., CR growth from 10% to 47%): could be detectable with hundreds of users.
  • Small lift (e.g., +0.5% or +1%): may require tens of thousands of users.

When A/B tests are NOT worth running

• When the hypothesis is already disproven/obvious from analysis (they cite that only ~20% of tests typically show significant improvements; others do nothing or go negative).
• When targeting is wrong / sample size is too small:
  • Example: testing with 10,000 visits, but only 5% reach the relevant screen ⇒ effectively ~500 users/month, making the test last ~6 months.
• When change cost is high and expected value is negligible compared to rollout cost (research/design → rollout overhead outweighs incremental profit).

Step-by-step A/B test process (playbook)

1) Hypothesis formulation + prioritization

Convert ideas into a formal hypothesis:

• what changes in the product
• who is targeted
• what should be affected
• how it will be measured

Prioritize hypotheses that align with the business growth point and likely impact.

2) Experimental design (documentation)

A design document includes:

• Primary metric(s) + secondary metrics
• Guardrail (“protective”) metrics: shouldn’t degrade
• Warning metrics / stop thresholds:
  • If metrics fall beyond a certain level (example: CR drops sharply from the normal 2–3% daily/weekly volatility band), treat it as a potential broken experiment.
• Invariant metrics: should not change under the tested hypothesis
  • Example: page load speed if the experiment isn’t about performance. If it changes materially, likely indicates a technical regression.
• Sample size calculation and statistical parameters:
  • α, power, β
• Metric statistical compatibility validation:
  • some metrics don’t work cleanly with standard tests (needs validation)

3) Pre-launch validation / simulations

• Run ~1000 simulated historical tests to validate:
  • Type I error rate behavior
  • whether the chosen metric + statistical criterion works as expected

4) Implementation + QA

• Ensure splitter / routing works correctly (developers or experimentation platform).
• QA checks:
  • tracking and event instrumentation
  • correct experiment identifiers and user assignment
  • no missing/duplicate data

5) Launch monitoring + data validation

• Monitor that data flows and metrics update as expected.
• Key validation mentioned:
  • SRM (Sample Ratio Mismatch):
    • if allocation planned 50/50 but observed becomes 60/40, the experiment is likely invalid; investigate and restart.
• Also validate segmentation integrity (example: operator capturing orders incorrectly can create “false winners”).

6) Analysis + decision rule

Only after stopping by the planned deadline:

• evaluate statistical significance
• ensure the planned metric configuration defines success/failure (to avoid “metric shopping”)
• provide results to stakeholders; notify other teams to avoid conflicting experiments

Common mistakes (top issues called out)

1. Testing something unnecessary (already obvious failure/success from prior data).
2. Lack of proper experimental design (just “roll out and observe”).
3. Peeking too early (checking daily for significance and stopping early).
4. Manipulating interpretation (“metric shopping”, adding unplanned metrics/segments to make a variant look better).
5. Wrong calculator/statistical setup:
   • e.g., using conversion-by-sessions when the independence unit should be users
   • sessions are not independent because one user can have multiple sessions → dispersion assumptions break

Additional mistake discussed:

• Stopping the test the moment p < 0.05 repeatedly while monitoring daily:
  • inflates Type I error risk (alpha becomes overstated)
  • mentions sequential testing as the proper technique if you truly want interim looks (but it requires correct design)

SRM and “50/50 expected but 5/95 observed” rule

• If allocation doesn’t match plan (extreme case mentioned: 5% instead of expected split):
  • treat A/B as invalid
  • find and fix splitter/routing problem
  • restart experiment

How to choose duration (timeline playbook)

Duration is determined by:

• traffic / DAU/MAU
• planned sample size
• weekly seasonality / representativeness

Suggested practical baseline:

• most A/B tests: 2–4 weeks
• aim to include at least one week (to cover weekday/weekend effects)

Very fast traffic systems can deliver results quickly, but still must avoid non-representative sampling (e.g., only Monday users).

Organizational / ownership model (product vs analytics)

• Core principle: Product is accountable for the test; Analytics supports the process.
• Suggested collaboration structure:
  • agree on success/failure definitions, stop conditions, constraints
  • analysts typically handle:
    • metric selection, statistical parameters, metric validation, analysis setup
  • product typically handles:
    • hypothesis, implementation coordination, stakeholder communication
• Noted case: at a bank, an analyst guided product step-by-step when products lacked A/B testing expertise.

High-level investing/market content

• Minimal direct market/investing discussion; focus remained on business execution and experimentation methodology.

---

Presenters / sources mentioned

• Vladislav Naskavets
• Yuriy Borzilo (spelled “Yuri” in subtitles; also referenced as running an A/B tests course and channel)