Jethro's Braindump

Experimental Data Science

A/B Testing

- When should A/B Testing not be Trusted to Make Decisions - Quora


  • generally proportional to number of users
  • Statistical power and Minimum Detectable Effect (MDE)
    • MDE treated as smallest effect size (signals) you can find in your experiment with certain probability

TODO Power, minimal detectable effect, and bucket size estimation in A/B tests