Statistics & Significance

Understanding the statistical concepts behind experimentation is crucial for running trustworthy tests and making reliable decisions.

Statistical Foundations

Mtrix uses rigorous statistical methods to ensure experiment validity:

Add alt text and captions to your images

Key Statistical Concepts

• Null Hypothesis: The assumption that there is no difference between variants

• Alternative Hypothesis: The assumption that there is a real difference

• p-value: Probability of seeing the observed results if the null hypothesis is true

• Confidence Level: Degree of certainty in rejecting the null hypothesis (e.g., 95%)

• Statistical Power: Probability of detecting a true effect (typically 80%)

• Effect Size: Magnitude of the difference between variants

Statistical Tests Used

Mtrix employs appropriate statistical tests based on metric types:

• Binary Metrics (for conversion, yes/no outcomes):

  • Chi-Square Test

  • ANOVA Test (for multi-variate testing)

• Continuous Metrics (revenue, time on site):

  • Student's t-test

  • Mann-Whitney U test (for non-normal distributions)

• Count Metrics (items purchased, pages viewed, or any other custom event you want to validate):

  • Poisson test

  • Negative binomial test (for overdispersed data)

By default Mtrix dashboard will generate statistical rigor and a confidence score for Conversions and Average Revenue metrics; for any other custom metrics you want to receive a score, please get in touch with your customer manager.