## What is a Type II Error?

Aliases: *false negative*

A type II error a.k.a. an error of the second kind is committed when we **fail to reject a false null hypothesis** (erroneously accept the alternative hypothesis). For example, assuming the null hypothesis is that of no difference, even though there is in fact a difference of magnitude (μ) between the means of the Control Gropu and Test Group(s) we fail to observe a statistically significant difference between them after performing an online controlled experiment.

After an A/B Test is completed we have either committed a type II error, or we have not. **The type II error rate** is thus, inevitably, a characteristic of the testing procedure, not of the tested hypothesis. A properly designed and executed significance test offers conservative guarantees regarding the probability of committing a type II error. In fact, we know that with many different statistics we can devise an α-Uniformly Most Powerful Test so that we have optimal power given a certain sample size. However, the type II error guarantees only hold if the test actually performed conforms to the test as it was planned.

The type II error of a test is at odds with the type I error: increasing one leads to decreasing the other, and vice versa, assuming fixed variance, sample size and minimum effect of interest.

## Articles on Type II Error

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.