Type I and Type II Errors

13.7. Type I and Type II Errors#

Further Reading: §6.12 and §6.13 in Navidi (2015)

# load libraries
import scipy.stats as stats
import numpy as np
import math
import matplotlib.pyplot as plt

13.7.1. Learning Objectives#

After studying this notebook, completing the activities, participating in class, and reviewing your notes, should be able to:

  • Formulate null and alternative hypotheses from a problem description

  • Draw conclusions by interpreting a calculated P-value

  • Explain Type I and Type II errors in the context of an application

13.7.2. Example: Law and Order#

A few classes ago, we said the significance level \(\alpha\) is often chosen at 0.05. But, this choice impacts the rate of wrong conclusions (errors). We will dive into this more today.

Consider a criminal trial in the American justice system. For simplicity, we’ll assume a defendant is either innocent or guilty. Likewise, the jury can either convict or acquit. Let’s express these options using the language of hypothesis testing:

Here is another table (same information, different formatting).

As we can see, there are four outcomes:

  1. Correct Inference / True Negative / Probability \(1 - \alpha\)

  2. Correct Inference / True Positive / Probability \(1 - \beta\)

  3. Type I Error / False Positive / Probability \(\alpha\)

  4. Type II Error / False Negative / Probability \(\beta\)

As we can see, the false positive error rate is \(\alpha\). Thus changing the significance level \(\alpha\) gives us direct control of how frequently we make a type I error. We will see late how to compute \(\beta\), the type II error rate.

13.7.3. Type I Errors#

Summary: Null hypothesis is true, but we reject it.

Other names: “asserting something that is absent”, “false hit”, “False Positive”

Examples:

  • Concluding a new drug is more effective than a placebo when it is not.

  • Concluding a manufacturing process is out of calibration when it is not.

  • Peter crying wolf when there is no wolf.

The choice of the significant level \(\alpha\) directly controls the Type I error rate.

13.7.4. Type II Errors#

Summary: Null hypothesis is false, but we erroneously fail to reject

Other names: “failing to assert what is present”, “miss”, “False Negative”

Examples:

  • Failing to conclude a new drug is more effective than a placebo when it actually is.

  • Failing to detect the ozone hole when it is there. good side tangent

  • The villagers ignoring Peter when the wolf is present.

The Type II error rate, denoted \(\beta\), is related to the power of a statistical test (\(1 - \beta\)).

Class Activity

With a partner, think of a science or engineering example of hypothesis testing.

Contents