Consequences of errors and significance

Significance tests often use a significance level of $\alpha =0.05$‍, but in some cases it makes sense to use a different significance level. Changing $\alpha$‍ impacts the probabilities of Type I and Type II errors. In some tests, one kind of error has more serious consequences than the other. We may want to choose different values for $\alpha$‍ in those cases.

A Type I error is when we reject a true null hypothesis. Lower values of $\alpha$‍ make it harder to reject the null hypothesis, so choosing lower values for $\alpha$‍ can reduce the probability of a Type I error. The consequence here is that if the null hypothesis is false, it may be more difficult to reject using a low value for $\alpha$‍. So using lower values of $\alpha$‍ can increase the probability of a Type II error.

A Type II error is when we fail to reject a false null hypothesis. Higher values of $\alpha$‍ make it easier to reject the null hypothesis, so choosing higher values for $\alpha$‍ can reduce the probability of a Type II error. The consequence here is that if the null hypothesis is true, increasing $\alpha$‍ makes it more likely that we commit a Type I error (rejecting a true null hypothesis).

Let's look at a few examples to see why it might make sense to use a higher or lower significance level.

Employees at a health club do a daily water quality test in the club's swimming pool. If the level of contaminants are too high, then they temporarily close the pool to perform a water treatment.

We can state the hypotheses for their test as $H_0:$‍ The water quality is acceptable vs. $H_{\text{a}}:$‍ The water quality is not acceptable.

Since one error involves greater safety concerns, the club is considering using a value for $\alpha$‍ other than $0.05$‍ for the water quality significance test.

Seth is starting his own food truck business, and he's choosing cities where he'll run his business. He wants to survey residents and test whether or not the demand is high enough to support his business before he applies for the necessary permits to operate in a given city. He'll only choose a city if there's strong evidence that the demand there is high enough.

We can state the hypotheses for his test as $H_0:$‍ The demand is not high enough vs. $H_{\text{a}}:$‍ The demand is high enough.

Seth has determined that a Type I error is more costly to his business than a Type II error. He wants to use a significance level other than $\alpha =0.05$‍ to reduce the likelihood of a Type I error.