/ A/B Testing

Type I and Type II Errors | Smoke Detector and the Boy Who Cried Wolf

Why Type I and Type II Errors Matter

A/B testing is an essential component of large scale online services today. So essential, that every worth mentioning online business is already doing it. A/B testing is also used in email marketing by all major online retailers. The Obama for America data science team received a lot of press coverage for leverage data science, especially A/B testing during the presidential campaign.

Type 1 and Type 2 Errors image

Here is an interesting article on this topic

If you have been involved in anything related A/B testing (online experimentation) on UI, relevance or email marketing, chances are that you have heard of Type I and Type II errors are. The usage of these terms is very common but a good understanding of these terms is not as common.

I have seen illustrations as simple as this.

Examples of Type I and Type II Errors

I intend to share two great examples I recently read that will help you remember this very important concept in hypothesis testing.

TYPE I ERROR: An alarm without a fire. **TYPE II ERROR: **A fire without an alarm.

Every cook knows how to avoid Type I Error – just remove the batteries. Unfortunately, this increases the incidences of Type II error. 🙂

Reducing the chances of Type II error would mean making the alarm hypersensitive, which in turn would increase the chances of Type I error.

Another way to remember this is by recalling the story of the Boy Who Cried Wolf.

Image of the boy who cried wolf

Null Hypothesis: There is no wolf.
Alternate Hypothesis: There is a wolf.

Villagers believing the boy when there was no wolf (Rejecting null hypothesis incorrectly): Type I Error
Villagers not believing the boy when there actually was a wolf (Rejecting alternate hypothesis incorrectly): Type II Error


The purpose of the post is not to explain type I and type II errors. If this is the first time you are hearing about these terms, here is the Wikipedia entry: Type I and Type II Error.

Raja Iqbal

Raja Iqbal

Raja is the CEO and Chief Data Scientist at Data Science Dojo. He has worked at Microsoft Bing and Bing Ads in various research and development roles in data science and machine learning.

Read More