CFA Program Level I

Type I vs Type II errors in hypothesis testing: CFA Level I

A Type I error is rejecting a null hypothesis that is true, and its probability is the significance level, alpha. A Type II error is failing to reject a null hypothesis that is false, with probability beta. For a given sample, lowering alpha raises beta; a larger sample can lower both.

The four outcomes

If the null is true and you fail to reject it, the decision is correct. If the null is true and you reject it, that is a Type I error, a false positive. Its probability is the significance level, alpha, which you choose before the test.

If the null is false and you fail to reject it, that is a Type II error, with probability beta. If the null is false and you reject it, the decision is correct. The probability of that, 1 - beta, is the power of the test.

NIST notes that beta is not chosen directly. It depends on how far the true value is from the null: a large difference gives a small beta, a small difference a large beta.

Alpha and beta trade off

For a fixed sample, beta increases as alpha decreases. Moving from a 5% to a 1% significance level makes you less likely to reject a true null, but more likely to miss a real effect.

The way to lower both is more data. NIST gives the minimum sample size for a two-sided test of a mean with known standard deviation sigma as N = (z(1 - alpha/2) + z(1 - beta))^2 (sigma / delta)^2, where delta is the shift you want to detect. For a one-sided test, z(1 - alpha) replaces z(1 - alpha/2).

Example: to detect a shift of one standard deviation (delta = sigma) with a two-sided test at alpha = 0.05 and beta = 0.10, N = (1.960 + 1.282)^2 = 10.51, so 11 observations. Cutting beta to 0.05 gives (1.960 + 1.645)^2 = 13.00, so 13. A one-sided test at alpha = 0.05 and beta = 0.10 needs (1.645 + 1.282)^2 = 8.57, so 9.

Rejection rules for a test of a mean

With the standard deviation estimated from the sample, the statistic is t = (sample mean - mu0) / (s / √N), with N - 1 degrees of freedom.

Two-sided, H0: mu = mu0: reject if |t| ≥ t(1 - alpha/2, N - 1). Upper one-sided, H0: mu ≤ mu0: reject if t ≥ t(1 - alpha, N - 1). Lower one-sided, H0: mu ≥ mu0: reject if t ≤ t(alpha, N - 1).

In a lower one-sided test, a large positive statistic is no evidence against H0, so you fail to reject it. The critical region is the set of statistic values that lead to rejection.

Where the exam catches people

Believing the probability of a Type I error equals the confidence level: it equals the significance level, alpha, which is 1 minus the confidence level.
Believing a lower significance level makes the test better in every way: on the same sample it raises beta, the chance of missing a real effect.
Believing a Type II error means rejecting a true null: that is a Type I error; a Type II error is failing to reject a false null.
Believing a lower beta needs fewer observations: reducing beta raises z(1 - beta), so the minimum sample size rises.
Believing any statistic far from zero rejects a one-sided null: in a lower-tail test only values below the lower critical value reject.

Written from CFA Institute's documentation

Try 3 questions on this

Pick an answer to see why each option is right or wrong.

A compliance team tests whether each trader's mean daily profit differs from zero, using a 5% significance level. A trader whose actual mean daily profit is zero is flagged by the test as having a mean profit different from zero. This outcome is most accurately described as a:
Right. In the game that’s Renown for your Quantitative Methods building.
Not that one: A is right. In the game your Quantitative Methods building would lose a little Renown, and this question would come back later.
The answerA.Type I error, whose probability equals the significance levelThe null hypothesis, a mean daily profit of zero, is true for this trader, yet the test rejected it. Rejecting a true null is a Type I error, a false positive, and the significance level of the test, here 5%, is the probability of a Type I error.Your answerB.Type II error, whose probability equals the significance levelA Type II error is failing to reject a null hypothesis that is false. Here the null was true and was rejected, which is a Type I error.Your answerC.Type I error, whose probability equals the confidence levelThe confidence level is the complement of the significance level, 95% here. The probability of a Type I error is the significance level itself, 5%.
Written from www.cfainstitute.org/insights/professional-learning/refresher-readings/2026/hypothesis-testing
An analyst tests whether a trading strategy's mean daily return differs from zero. Using the same sample, she lowers the significance level from 5% to 1% to reduce the risk of rejecting the null hypothesis when it holds. The probability of failing to reject the null hypothesis when the mean return actually differs from zero most likely:
Right. In the game that’s Renown for your Quantitative Methods building.
Not that one: B is right. In the game your Quantitative Methods building would lose a little Renown, and this question would come back later.
Your answerA.FallsLowering alpha reduces only the risk of rejecting a null that holds. With the same data, a stricter rejection rule makes a real departure harder to detect, so the Type II risk goes up, not down.The answerB.RisesThe probability of failing to reject a null hypothesis that does not hold is the Type II error risk, beta. Beta increases as alpha decreases, so cutting alpha from 5% to 1% on the same sample makes the test less likely to detect a real departure from zero.Your answerC.Is unchangedBeta does depend on how large the real departure is, but that is not all: it also increases as alpha is lowered. The analyst's change to alpha therefore moves beta.
Written from www.itl.nist.gov/div898/handbook/prc/section1/prc13.htm
An analyst is planning a two-sided test of whether a fund's mean monthly return differs from a benchmark value. Holding the significance level and the population standard deviation constant, she wants a lower probability of missing an actual shift of a given size. The minimum sample size required most likely:
Right. In the game that’s Renown for your Quantitative Methods building.
Not that one: B is right. In the game your Quantitative Methods building would lose a little Renown, and this question would come back later.
Your answerA.DecreasesA smaller β means a larger critical value z(1 − β). Since N = (z(1 − α/2) + z(1 − β))² (σ / δ)², a larger z(1 − β) needs more observations, not fewer.The answerB.IncreasesFor a two-sided test with known σ, the minimum sample size is N = (z(1 − α/2) + z(1 − β))² (σ / δ)², where β is the probability of missing a true shift δ.Your answerC.Stays the sameThe sample size depends on α, on β and on the population standard deviation. Holding α and σ fixed while lowering β still raises N.
Show the full explanation and doc quotes
Lowering β raises z(1 − β), so N rises. For a shift of one standard deviation at α = 0.05: β = 0.10 gives (1.96 + 1.282)² ≈ 10.5, so 11 observations; β = 0.05 gives (1.96 + 1.645)² ≈ 13.0, so 13.
Written from www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm