Ozon Design Talks — Хороший дизайн по мнению аналитика

Main ideas / concepts / lessons

• “Good design” is defined operationally: A design is “good” if it positively impacts product metrics (and “bad” if it harms them).

• Naive before/after reasoning is insufficient: Simply checking whether a metric goes up or down after a release is unreliable because other factors can change metrics too, such as:

  • competitor actions
  • releases by other teams
  • seasonality / external events (e.g., “fishing season opening”)

• A/B testing (A-tests) isolates the effect of one change: Users are split into groups that differ only in one element (e.g., whether they see the new design/button).

  • If metrics differ between groups, the change may have an effect.
  • Differences might still be due to random noise, so statistical testing is required.

• Statistical tests produce a “p-value” (referred to as “Pvue/pivelio”):

  • The speaker explains p-values with an analogy: trying to decide whether your dog is guilty given chaotic evidence.
  • Key takeaway: p-values are mathematically convenient, but not necessarily intuitive.

• Two types of errors are possible:

  • Type I error (α level): real effect is absent, but you conclude there is an effect.
    • often set around 5%
  • Type II error (β level): real effect is present, but you conclude there is not.
    • often around 20%
  • The exact comfort levels are chosen by the company, not “fixed by nature.”

• Decision rule using p-value and α:

  • Run an experiment and compute a p-value for metric comparisons between groups.
  • If p-value > α → the observed change is likely random / not statistically significant.
  • If p-value < α → the change is statistically significant / not random.

• What effect can you detect? Introduce MDE (Minimal Detectable Effect):

  • MDE is the smallest percent change in a metric that the experiment can reliably detect while keeping error rates at the chosen levels.
  • Detectability depends mainly on:
    • sample size
    • variance (dispersion) / noise level of the metric
  • Larger effect sizes are easier to detect.

• Intuition via examples:

  • Toothbrush reviews example:
    • A product with fewer ratings may show a higher/lower approval rate, but it’s less trustworthy due to small sample size.
    • A product with more ratings is more believable even if the percentage difference is not huge.
  • Variance/dispersion example:
    • If group distributions overlap heavily (high dispersion), it’s hard to distinguish them.
    • Lower dispersion makes group differences easier to see.

• Experiment design emphasis: the experiment description matters most

  • Clearly state:
    • what changes for the user
    • why it should change behavior
    • which metrics will indicate success

• Example experiment: red “Cancel” button on an active orders page

  • The hypothesis includes both:
    • Users who don’t truly need to cancel shouldn’t click cancel more just because it looks “repulsive/dangerous.”
    • Users who do need to cancel should still find the button quickly and not need to contact support (e.g., “cancel in chat”).

• Metric mapping in the example: Confirm hypotheses with three metric-related checks:

  1. Number of orders submitted should not change
  2. Requests for “cancel my order” should decrease
  3. Cancel button click rate (CR) should increase (a control to confirm users are actually finding/clicking it when needed)

• Multiple hypotheses increase risk of Type I error: Testing multiple hypotheses/metrics increases the probability of at least one Type I error.

• How long to run the experiment (duration strategy using MDE):

  • Experiments are usually multiple weeks, often to account for weekend behavior.
  • Two duration strategies:
    1. If you know the minimum effect you must detect: compute MDE by week and run long enough to achieve that detectability.
    2. If you don’t know expected effect size:
       • compute MDE for several week lengths (e.g., 3–6 weeks)
       • find the inflection point: the week after which MDE stops improving meaningfully (no additional power gained), then stop.

• Outcome of the example experiment (interpretation):

  • Cancelled orders share increased in group B, but p-value was above α → interpreted as random, not a real positive/negative impact.
  • Requests for cancellation dropped, with a p-value below α → interpreted as real and statistically significant.
  • Cancel button CR increased → used as a control sign that users truly clicked the button more.

• Wrap-up summary:

  • Covered:
    • why A/B tests matter for isolating design impact
    • how “good vs bad” design relates to metric effects
    • what p-values and MDE mean
    • how to design/run an experiment (as shown in the example)

---

Methodology / instruction-style steps

A) Determine whether design changes improve the product

1. Define the product’s goals via metrics.
2. Treat “good design” as:
   • positive impact on metrics (and “bad design” as negative impact)
3. Avoid simplistic approach: Don’t rely only on “metric changed after release,” since other factors may be responsible.

B) Isolate the design change effect using A/B testing

1. Identify the one variable to change (e.g., new button color/design).
2. Split users into two groups:
   • Group A (control): sees old version / no change
   • Group B (treatment): sees the changed version
3. Measure chosen metrics in each group after exposure.
4. Use statistical tests to evaluate whether differences are random or real.

C) Run statistical tests and make a decision

1. Compute a p-value (speaker’s “Pvue/pivelio”).
2. Set an α threshold (commonly ~5%).
3. Apply the decision rule:
   • If p-value > α → difference is not statistically significant (likely chance).
   • If p-value < α → difference is statistically significant (likely due to the change).

D) Plan experiment sensitivity using MDE

1. For each metric, compute MDE.
2. Understand MDE is driven mainly by:
   • sample size
   • variance/dispersion/noise
3. Remember: larger real effects are easier to detect.

E) Design the experiment description + metrics carefully

1. Write a clear user-facing change (e.g., “make the cancel button red”).
2. Provide a hypothesis explaining why it should affect behavior.
3. Select metrics that correspond to each hypothesis (example):
   • include safety/neutrality checks (e.g., orders submitted shouldn’t worsen)
   • include behavioral outcomes (e.g., cancellation requests)
   • include a control metric (e.g., cancel button CR to verify the mechanism)

F) Choose experiment duration using MDE over time

• Strategy 1 (known target effect):
  1. Specify an effect size you must detect (e.g., 3%)
  2. Compute MDE by week
  3. Run until you can reliably detect that effect
• Strategy 2 (unknown expected effect):
  1. Compute MDE for several week lengths (e.g., 3–6 weeks)
  2. Pick the inflection point where MDE stops improving meaningfully
  3. Stop there to avoid unnecessary runtime

G) Interpret results after the experiment

1. Compare metrics between group A and B.
2. For each metric, check statistical significance vs α:
   • not significant → treat as random/no real effect
   • significant → treat as real effect likely caused by the design change
3. Use control metrics to validate mechanism (e.g., higher CR means users actually interacted with the button).

---

Speakers / sources featured

• Nikita Manannikov — at Ozon, manages the wholesale buyer’s product analytics department; speaker of the video.
• No other named speakers or external sources are clearly identified beyond a general mention of “colleagues” knowledgeable in the topic.