Week 13 · Chapter

Chapter 13 — Hypothesis Testing: Foundations

Introduction to Statistics Free sample edition

The module's primary reading · ~40–50 minutes · Objective 7 (carry out and correctly interpret hypothesis tests for means and proportions, including one- and two-sample settings, p-values, significance levels, and Type I and Type II errors) · Read it first this week; the lecture, the tutorial, the quiz, and the Data Lab all build on it.

In plain language

A student hands in a 10-question quiz with 4 choices per question, scores 7, and tells you they guessed on every question. Should you believe them? This week you learn a hypothesis test (a way to check a claim by asking how surprising the data would be if the claim were true) and its main output, the p-value (the chance of a result at least this extreme if the boring explanation were true). You will learn to say exactly what that number means and what it does not mean. You will also learn the two ways a test can go wrong, and why a "significant" result is not always an important one.

Why you already care about this

You run small tests all the time without the vocabulary. A friend flips a coin and gets heads 10 times in a row. A store says its new checkout line saves 2 minutes, yet your wait feels the same. A teammate swears a lucky pair of socks lifted their free throws from 7 in 10 to 8 in 10. Each time there is a boring explanation, plain luck or no real change, and a more interesting claim.

You decide which to believe by asking one question: how surprising would this be if nothing special were going on? That is the whole logic of this week. You write the boring explanation down and assume it is true for a moment. Then you measure how surprising the data would be. Very surprising data make you doubt the boring explanation. Ordinary data let it stand, without proving it.

The guessing quiz from Week 7 is the perfect place to start. You already know how a pure guesser does on 10 questions with four options each: about 2.5 correct on average, with a probability for every score from 0 to 10. So when a student who claims to have guessed scores 7, you can work out how often a real guesser would do that well.

By the end of this chapter you will be able to:

write a null and an alternative hypothesis about a parameter, and choose a one-sided or two-sided alternative from the question;
explain the logic of a significance test in one or two sentences;
describe what a test statistic measures, and compute one for the guessing quiz;
compute a p-value from the guessing-quiz table by hand and with a spreadsheet, and interpret it in one correct sentence;
name three things a p-value is not;
compare a p-value with a significance level α and state the decision and the conclusion in context;
define Type I and Type II errors and power, and explain the trade-off between the two errors;
tell statistical significance from practical significance, and explain why "fail to reject" is not "accept".

Carry this sentence with you: a p-value measures surprise — not truth.

Where we are

Already covered: Weeks 1 to 3 described one variable, and Week 4 related two. Weeks 5 to 7 built probability and the binomial, Week 8 added the z-score, Week 9 the normal model, and Week 10 sampling distributions. Weeks 11 and 12 built confidence intervals for a mean and a proportion.
This week: the foundations of hypothesis testing: hypotheses, the logic of a test, test statistics and p-values, significance levels and decisions, Type I and Type II errors, and statistical versus practical significance.
How they connect: a confidence interval estimates a parameter; a test checks one claimed value of a parameter. Both rest on the fact that a statistic wobbles from sample to sample in a predictable way.
Where we're going next: this week opens the testing half of the inference unit. Next week you run full tests for means and proportions.

Three earlier weeks matter most here. Week 5 gave you the complement rule: the chance something does not happen is 1 minus the chance it does. Week 7 gave you the binomial setting and the guessing-quiz table. With n (the number of questions) equal to 10 and p (the chance of a correct guess) equal to 0.25, the mean is 2.5 correct answers and the standard deviation is about 1.37. Week 8 gave you the z-score, the number of standard deviations a value sits above or below the mean. This week all three come back in one argument.

1. Two hypotheses: name the claim you doubt

Go back to the student with the 7. You can tell two stories about that score. In the first, the student really guessed and got lucky. In the second, the student knew some answers. A hypothesis test puts those two stories into precise words.

Each story is a claim about a parameter, a fixed number that describes the whole process, not the one quiz you saw. Here the parameter is p (the student's true chance of answering any one question correctly). A pure guesser has p = 1 ⁄ 4 = 0.25, because each question has four options and one is right.

The null hypothesis is the claim of no effect or no difference that a test assumes is true for the sake of argument, written H₀ and read "H naught". It always names a value for a parameter, such as p = 0.25 for a pure guesser.
The alternative hypothesis, the rival claim, is the claim a test looks for evidence of, written Hₐ. It states that the parameter is greater than, less than, or different from the value the null hypothesis names.

Worked example — the guessing claim. The question: did the student do better than guessing?

Name the parameter. p is the student's true chance of a correct answer on each question.
Write the null hypothesis. H₀: p = 0.25. The student is guessing.
Write the alternative. Hₐ: p > 0.25. The student does better than guessing.
Check the direction. Only a high score would cast doubt on the claim. A very low score would not suggest the student knew the material. So the alternative points one way.

In words: you will ask whether a score of 7 is surprising enough to doubt that p is 0.25. A teacher who must decide whether to accept the student's story needs this pair of sentences first.

An alternative that points one way, like p > 0.25, is a one-sided alternative (it looks in one direction only). Sometimes either direction would matter. Suppose a field biologist asks whether female and male penguins are equally common. A share far above one half or far below it would both cast doubt on an even split. Then H₀: p = 0.5 and Hₐ: p ≠ 0.5, a two-sided alternative (it looks in both directions). You will run that penguin test next week; this week you only need to write it.

Two more quick examples show the pattern. A board-game player suspects a die lands on six too often. For a fair die the chance of a six is 1 ⁄ 6, about 0.1667, so H₀: p = 1 ⁄ 6 and Hₐ: p > 1 ⁄ 6. A pollster wants to know whether a majority of voters back a measure. Then H₀: p = 0.5 and Hₐ: p > 0.5, where p is the share of all voters who back it.

Misconception check — writing hypotheses about the sample. Students often write "H₀: the student got 7 right" or put p̂ (the sample share) in a hypothesis, because the sample is the number on the page. But you already know the sample result; there is nothing to test about it. Hypotheses are always about a parameter, never about the sample. The cure: before you write H₀, say out loud which unknown number the question is about.

Memory hook: write the claim about the process — never about the sample.

2. The logic of a test: assume the null, then measure surprise

You already know how a trial works, and a courtroom runs on the same logic as a hypothesis test. The defendant is presumed innocent. The prosecution shows evidence. If the evidence would be very hard to explain for an innocent person, the jury rejects the presumption and says "guilty". If the evidence could easily happen to an innocent person, the jury says "not guilty".

Notice what the jury never does. It never declares the defendant innocent. "Not guilty" means the evidence was not strong enough, nothing more.

A significance test makes the same three moves:

Presume. Assume H₀ is true for the sake of argument.
Weigh the evidence. Ask how surprising the data would be if H₀ were true.
Decide. If the data would be very surprising under H₀, reject H₀. If not, fail to reject H₀.

A hypothesis test is a procedure, also called a significance test, that weighs data against a claim about a parameter by assuming the claim of no effect is true and asking how surprising the data would be under it.

Worked example — the logic, before any arithmetic. Take the student's claim of guessing.

Presume. Suppose the student really guessed, so p = 0.25.
Weigh the evidence. Under guessing, the average score is 10 × 0.25 = 2.5 correct answers. A score of 7 sits far above that.
Decide, for now, in words. If guessers almost never score 7, the story is hard to believe. If guessers often score 7, the story stands.

In words: the whole test turns on how often a real guesser would score 7 or more. The next two sections turn that question into a number a teacher can act on.

Misconception check — "the test tries to prove H₀". Students often think a test sets out to show the null hypothesis is true, because the test starts by assuming it. It is tempting, since a "hypothesis" sounds like the thing being proved. But you assume H₀ only to see whether the data contradict it, the way a jury presumes innocence only to see whether the evidence overturns it. The cure: think of H₀ as the presumption, and of the data as the evidence against it.

Memory hook: presume the null — then weigh the evidence.

3. Test statistics: measure how far the data sit from H₀

Surprise needs a ruler. A score of 7 is far from 2.5, but "far" means something only when you know how much scores usually vary. That is the job of a test statistic.

A test statistic is a number computed from the sample that measures how far the data sit from what the null hypothesis predicts, such as a count of successes or a statistic's distance from the null value in standard errors.

For the guessing quiz the simplest test statistic is the count itself: X (the number of correct answers) is 7. H₀ predicts a count near 2.5, with a standard deviation of about 1.37 correct answers. Week 8's z-score turns the gap into standard deviations.

Worked example — how far is 7 from guessing? The question: how many standard deviations above the guessing average is a score of 7?

The null mean. Under H₀ the mean count is np = 10 × 0.25 = 2.5 correct answers.
The null spread. The standard deviation of the count is the given input 1.3693, about 1.37 correct answers.
The gap. The score is 7 − 2.5 = 4.5 correct answers above the null mean.
In standard deviations. (7 − 2.5) ⁄ 1.3693 ≈ 3.29.

In words: your student's score sits about 3.29 standard deviations above what guessing predicts. For comparison, the Week 8 Gentoo penguin that seemed unusually light sat about 2.13 standard deviations below its species mean. A teacher who sees a score more than three standard deviations above the guessing average has good reason to look closer.

Many test statistics you will meet work the same way: they count how many standard errors a statistic sits from the value H₀ names. The farther away, the more surprising your data would be under H₀.

One honesty note. The guessing count does not follow a normal model. Week 7 showed that np = 2.5 is below 10, so the large-counts rule of thumb fails. That is why the p-value in the next section comes from the exact binomial table, not from a normal table.

Misconception check — "a big test statistic is the answer". Students often stop at 3.29 and call the result significant, because a big number feels decisive. But a test statistic is only a distance. You still have to turn it into a probability before you can judge how surprising it is, and the right probability depends on the null model. The cure: treat the test statistic as the ruler reading, and always finish with the p-value.

Memory hook: read the ruler first — the verdict comes later.

4. p-values: turn surprise into a probability

Now for the number that does the real work.

A p-value is the probability, computed assuming the null hypothesis is true, of getting data at least as extreme as the data observed. A small p-value means the data would be surprising if the null hypothesis were true, and it is not the probability that the null hypothesis is true.

Three parts of that definition matter. Computed assuming H₀ means you use the guessing model, p = 0.25. At least as extreme means you count the observed score and every score that would be even more surprising; here Hₐ is p > 0.25, so "more extreme" means higher. Data, not hypotheses: the p-value is a statement about scores, not about the student.

Write a p-value with the word "value" every time. In this course a bare p is a population proportion, such as the student's true chance of a correct answer.

Here is the guessing-quiz table from Week 7, which the whole week uses. Each entry is the chance a pure guesser gets exactly k questions right, where k is one particular count from 0 to 10.

Chance of exactly k correct answers for a pure guesser on 10 four-option questions, rounded to four decimals
Correct answers (k)	0	1	2	3	4	5	6	7	8	9	10
Chance of exactly k	0.0563	0.1877	0.2816	0.2503	0.1460	0.0584	0.0162	0.0031	0.0004	0.0000	0.0000

Worked example — the p-value for a score of 7. The question: if the student were guessing, how often would they score 7 or more?

Name the tail. "7 or more" means the scores 7, 8, 9, and 10.
Read the table. Those chances are 0.0031, 0.0004, 0.0000, and 0.0000.
Add them. P(X ≥ 7) = 0.0031 + 0.0004 + 0.0000 + 0.0000 = 0.0035, rounded to four decimals.
Say it. If the student were purely guessing, a score of 7 or more would occur on about 0.35% of quizzes, about 35 quizzes in 10,000.

In words: a real guesser almost never does this well. You, as the teacher, now have a precise reason to doubt the story.

The course's template for every p-value sentence is "if the null were true, data this extreme would occur about … of the time". You fill in the blank with the p-value, and you say what "the null" and "this extreme" mean in the setting.

Compare two other scores to calibrate your sense of surprise.

A score of 5. The p-value is P(X ≥ 5), the given input 0.0781. If the student were guessing, a score of 5 or more would occur on about 7.8% of quizzes, about 8 in 100. A class of guessers would produce that every few quizzes.
A score of 3. By the complement rule, P(X ≥ 3) = 1 − P(X ≤ 2) = 1 − 0.5256 ≈ 0.47. About 47 guessers in 100 score 3 or more, so a 3 gives you no evidence against guessing at all.

In practice (your tools). In Google Sheets or Excel, type =1-BINOM.DIST(6,10,0.25,TRUE) in any cell. BINOM.DIST with TRUE returns the running total up to 6 correct, and subtracting it from 1 leaves the tail of 7 or more. The cell shows 0.0035 to four decimals. For a score of 5, change the 6 to a 4; for a score of 3, change it to a 2. The pattern is always "1 minus the running total up to one less than the score".

Misconception check — counting only the exact score. Students often report the table entry for exactly 7, the given input 0.0031, as the p-value, because 7 is the score that happened. But a p-value counts the observed result and everything more extreme in the direction of Hₐ. A score of 8 would have been even stronger evidence, so it belongs in the tail. The cure: before you add, write the tail in words, "7 or more", and list every score in it.

Memory hook: at least as extreme — count the whole tail.

5. What a p-value is not: three wrong readings

The p-value is the most misread number in statistics. In 2016 the American Statistical Association published a statement (Wasserstein and Lazar) listing six principles about p-values, and this course checks every p-value sentence against them. Three of those principles name the wrong readings you will hear most often.

A p-value indicates how incompatible the data are with a specified statistical model. It does not measure the probability that the studied hypothesis is true. It does not measure the probability that the data were produced by random chance alone. And it does not measure the size or importance of an effect.

Worked example — fixing three sentences about 0.0035. Each sentence below misreads the student's p-value. Read the wrong version, then the repair.

Wrong: "There is a 0.35% chance the student guessed." Repair: "If the student were guessing, a score of 7 or more would occur on about 0.35% of quizzes." The p-value was computed by assuming guessing, so it cannot also be the chance of guessing.
Wrong: "There is a 0.35% chance the score was due to chance." Repair: the same sentence as above. "Due to chance" sounds like a probability about the cause of this one score, and the p-value is not that. It describes how often guessing produces scores like this one.
Wrong: "A p-value of 0.0035 means the student knows a lot." Repair: "The score is strong evidence that the student did better than guessing; it does not say how much better." A student who knows a little and one who knows a lot could both produce small p-values.

In words: a p-value is a statement about your data, computed as if H₀ were true. It is not the chance that H₀ is true, and it is not a measure of size. You can avoid all three mistakes by sticking to the template.

Misconception check — "the p-value is the probability the null is true". Students often read 0.0035 as "a 0.35% chance the student was guessing", because it is the only probability in sight. It is tempting: it sounds like exactly what you want to know. But the p-value is computed inside the guessing story. It answers "how surprising are these data if guessing is true?", not "how likely is guessing, given these data?" The cure: start every p-value sentence with "If the null were true, …".

Memory hook: a p-value measures surprise — not truth.

6. Decisions: compare the p-value with α

A test ends in a decision, and a decision needs a cutoff.

The significance level is the cutoff α, chosen before the data are seen, that a p-value is compared with. It is also the probability of rejecting a null hypothesis that is true, and 0.05 is the most common choice.

Read α as "alpha". You will also see 0.01 and 0.10. The decision rule has two branches:

If the p-value is at most α, you reject the null hypothesis (decide the data are too surprising for it). That decision means the data give convincing evidence against H₀ and in favor of Hₐ, though they do not prove Hₐ. The result is statistically significant (its p-value fell at or below the cutoff).
If the p-value is above α, you fail to reject the null hypothesis (let it stand for now). That decision means the data are consistent with H₀, which is not the same as showing that H₀ is true.

Worked example — two students, one rule. The rule: α = 0.05, fixed before anyone saw the scores.

The student who scored 7. The p-value 0.0035 is below 0.05, so you reject H₀. It is also below the stricter α = 0.01.
Conclusion in context. The score gives strong evidence that the student was not purely guessing. It does not prove they studied. It does not say how much they know, either.
A second student who scored 5. The p-value is 0.0781, which is above 0.05, so you fail to reject H₀.
Conclusion in context. A score of 5 is consistent with guessing. The data do not give convincing evidence that this student did better than guessing.

In words: one rule treats the two students differently because their scores carry different amounts of evidence. A teacher who must decide whether to follow up with each student has a clear, pre-stated reason for each choice.

There is a caveat worth stating plainly. At α = 0.10, the p-value 0.0781 would reject H₀. The second student's decision depends on α, and that is exactly why you choose α before you see the data. If you pick α after you see the p-value, you can reach whichever decision you wanted. The ASA statement makes a related point: a conclusion should not rest only on whether a p-value crosses a threshold. A p-value of 0.049 and one of 0.051 carry nearly the same evidence.

Misconception check — "statistically significant means important". Students often hear "significant" in its everyday sense and assume the result matters a lot. It is tempting, because in daily life "significant" means "big". But in statistics it only means the p-value fell at or below α. Section 8 shows a result that is significant and still small. The cure: whenever you write "statistically significant", add one sentence about how big the effect is.

Memory hook: α is set first — the data come second.

7. Type I and Type II errors: two ways to be wrong

Your smoke alarm runs a test every second, and it can be wrong in two directions. Its null hypothesis is "there is no fire". If it shrieks at burnt toast, it rejected a true null: a false alarm. If it stays silent during a real fire, it failed to reject a false null: a miss. Both are mistakes, and they are different mistakes.

The four outcomes of a hypothesis test, by the decision and by whether the null hypothesis is actually true
Decision	H₀ is actually true	H₀ is actually false
Reject H₀	Type I error (probability α)	Correct decision (probability 1 − β, the power)
Fail to reject H₀	Correct decision	Type II error (probability β)

A Type I error is rejecting a null hypothesis that is actually true, such as concluding a pure guesser knew the material. Its probability is the significance level α.
A Type II error is failing to reject a null hypothesis that is actually false, such as missing a student who really does better than guessing. Its probability is written β, read "beta".
The power of a test (its knack for catching what is really there) is the probability that it rejects the null hypothesis when the null hypothesis is false, equal to 1 − β. It is the chance the test catches a real effect.

Worked example — two cutoffs for the guessing quiz. Suppose a teacher flags any student whose score is too high to be a likely guess. Compare two rules.

Rule A: flag a score of 5 or more. A pure guesser is then wrongly flagged with probability P(X ≥ 5), the given input 0.0781. That is the Type I error rate, the α of Rule A.
Rule B: flag a score of 7 or more. A pure guesser is wrongly flagged with probability 0.0035.
Compare. Rule A's Type I error rate versus Rule B's is about a 22-fold difference, because 0.0781 ⁄ 0.0035 ≈ 22.
The cost. A student who truly knows some answers can still score 5 or 6. Rule A flags that student; Rule B misses them. So Rule B makes more Type II errors: β rises as α falls.

In words: Rule B wrongly flags about one honest guesser in 286 instead of about one in 13, but it lets more students who know some material slip through. You, as the teacher, have to choose which mistake costs more.

That choice is a judgment, not a calculation. If a wrong flag leads to an unfair accusation, a small α matters most. If a missed student loses a chance at extra help, power matters most. One limitation to state honestly: this chapter never gives β a number, because β depends on how much the student really knows, and that is the unknown the test is about.

Misconception check — "α is the chance my decision is wrong". Students often read α = 0.05 as a 5% chance that any decision is a mistake. It is tempting, because α is the error rate everyone quotes. But α is the chance of rejecting H₀ when H₀ is true. The chance of a miss when H₀ is false is β, a different number. The cure: name the error first, "a false alarm" or "a miss", then name its probability.

Pause and reflect — choose one. Pick either of the 2 prompts; they count the same.

(a) Think of a screening decision from a field you care about, such as a medical test, an airport bag scan, or a spam filter. Write its null hypothesis in words, describe its Type I and Type II errors, and explain in two or three sentences which error is worse there and why.
(b) A coach flags any runner whose sprint time looks too slow to be an off day. Describe a Type I and a Type II error for that rule, and explain in two or three sentences how a stricter rule would change each one.

Memory hook: false alarm versus miss — shrink one and the other grows.

8. Statistical versus practical significance: size matters, and silence is not proof

A small p-value says your data would be surprising if H₀ were true. It does not say the effect is large. To see the difference, keep the same 40% score and change only the length of the quiz.

Worked example — the same 40%, two quiz lengths. The question: how surprising is a score of 40% for a pure guesser?

Ten questions. A score of 4 out of 10 is 40%. Its p-value is P(X ≥ 4) = 1 − (0.5256 + 0.2503) = 0.2241, rounded to four decimals.
Say it. About 22 guessers in 100 do that well, so 40% on 10 questions should not surprise you.
One hundred questions. On a 100-question version, guessing has mean 100 × 0.25 = 25, and its standard deviation is the given input 4.3301, about 4.33.
Check the model. Both np = 25 and n(1 − p) = 75 are at least 10, so a normal model for the count is reasonable.
Measure the distance. A score of 40 is (40 − 25) ⁄ 4.3301 ≈ 3.46 standard deviations above 25, beyond 3 standard deviations.
Turn it into a tail. The empirical rule leaves about 0.3% of a normal model beyond 3 standard deviations on both sides combined, so fewer than about 2 guessers in 1,000 would score 40 or more. The p-value is well below 0.01.

In words: the same 40% share is unremarkable on 10 questions and highly significant on 100 questions. The p-value shrank because the quiz got longer, not because the edge grew. A student who scores 40% has still beaten guessing by only 15 questions out of 100.

That gap has a name. Practical significance is whether an effect is large enough to matter in its real-world setting, a judgment about size and consequences that a p-value does not measure. A tiny, unimportant effect can be statistically significant in a large enough sample.

The ASA statement says it directly: a p-value does not measure the size of an effect or the importance of a result. So when you report a test, report the size too.

Fail to reject is not accept. Return to the second student, who scored 5 on the 10-question quiz. "Fail to reject H₀" means a score this high is consistent with guessing. A student who knows a little could also score 5, so the test cannot tell the two apart. The data are silent, not supportive. A jury that says "not guilty" has not declared the defendant innocent, and a test that fails to reject has not shown that H₀ is true. Write "the data do not give convincing evidence that …", never "the data prove …" or "we accept H₀".

Misconception check — "a smaller p-value means a bigger effect". Students often compare two p-values and conclude that the smaller one belongs to the bigger effect, because "more significant" sounds like "more". But the 40% example shows one effect giving a p-value of 0.2241 on a short quiz and one well below 0.01 on a long quiz. The cure: beside every p-value, report the effect's size in the units of the problem, here the share of questions answered.

Misconception check — "failing to reject proves the null". Students often read "fail to reject H₀" as "H₀ is true", because a test has two outcomes and this is the other one. But a test can only gather evidence against H₀. Weak evidence against a claim is not strong evidence for it. The cure: say "consistent with", never "proves", and remember the jury.

Memory hook: significant is not big — silent is not true.

Key terms

null hypothesis — The claim of no effect or no difference that a test assumes is true for the sake of argument, written H₀ and read "H naught". It always names a value for a parameter, such as p = 0.25 for a pure guesser. (first used: 1. Two hypotheses: name the claim you doubt)
alternative hypothesis — The claim a test looks for evidence of, written Hₐ. It states that the parameter is greater than, less than, or different from the value the null hypothesis names. (first used: 1. Two hypotheses: name the claim you doubt)
one-sided alternative — An alternative hypothesis that points in one direction, such as p > 0.25, used when only results on one side of the null value would cast doubt on the null hypothesis. (first used: 1. Two hypotheses: name the claim you doubt)
two-sided alternative — An alternative hypothesis written with a not-equal sign, such as p ≠ 0.5, used when results far from the null value in either direction would cast doubt on the null hypothesis. (first used: 1. Two hypotheses: name the claim you doubt)
hypothesis test — A procedure, also called a significance test, that weighs data against a claim about a parameter by assuming the claim of no effect is true and asking how surprising the data would be under it. (first used: 2. The logic of a test: assume the null, then measure surprise)
test statistic — A number computed from the sample that measures how far the data sit from what the null hypothesis predicts, such as a count of successes or a statistic's distance from the null value in standard errors. (first used: 3. Test statistics: measure how far the data sit from H₀)
p-value — The probability, computed assuming the null hypothesis is true, of getting data at least as extreme as the data observed. A small p-value means the data would be surprising if the null hypothesis were true, and it is not the probability that the null hypothesis is true. (first used: 4. p-values: turn surprise into a probability)
significance level — The cutoff α, chosen before the data are seen, that a p-value is compared with. It is also the probability of rejecting a null hypothesis that is true, and 0.05 is the most common choice. (first used: 6. Decisions: compare the p-value with α)
reject the null hypothesis — The decision made when the p-value is at or below α: the data give convincing evidence against H₀ and in favor of Hₐ, though they do not prove Hₐ. (first used: 6. Decisions: compare the p-value with α)
statistically significant — Describes a result whose p-value is at or below the chosen significance level α, so the data are judged too surprising to square comfortably with the null hypothesis. It says nothing on its own about how large or important the effect is. (first used: 6. Decisions: compare the p-value with α)
fail to reject the null hypothesis — The decision made when the p-value is above α: the data are consistent with H₀, which is not the same as showing that H₀ is true. (first used: 6. Decisions: compare the p-value with α)
Type I error — Rejecting a null hypothesis that is actually true, such as concluding a pure guesser knew the material. Its probability is the significance level α. (first used: 7. Type I and Type II errors: two ways to be wrong)
Type II error — Failing to reject a null hypothesis that is actually false, such as missing a student who really does better than guessing. Its probability is written β. (first used: 7. Type I and Type II errors: two ways to be wrong)
power — The probability that a test rejects the null hypothesis when the null hypothesis is false, equal to 1 − β. It is the chance the test catches a real effect. (first used: 7. Type I and Type II errors: two ways to be wrong)
practical significance — Whether an effect is large enough to matter in its real-world setting, a judgment about size and consequences that a p-value does not measure. A tiny, unimportant effect can be statistically significant in a large enough sample. (first used: 8. Statistical versus practical significance: size matters, and silence is not proof)

Summary

A hypothesis test checks a claim about a parameter: assume the null hypothesis, then ask how surprising the data would be.
The null hypothesis names a value for the parameter; the alternative says greater than, less than, or different from it.
A test statistic measures how far the data sit from what the null hypothesis predicts.
A p-value is the chance of data at least this extreme if the null were true; it is not the chance the null is true.
Reject the null when the p-value is at or below the significance level, chosen in advance; otherwise fail to reject it.
A Type I error is a false alarm; a Type II error is a miss; power is the chance of catching a real effect.
Statistical significance is not practical significance, and failing to reject the null never proves it.

Check your understanding

Work these before looking at the answers — they are ungraded, and the answers (with reasoning) are at the bottom of the page.

A bakery says its sourdough loaves weigh 500 grams on average. A regular customer suspects they are lighter. Write the null and alternative hypotheses, say what the parameter is, and say whether the alternative is one-sided or two-sided.
A friend flips a coin 5 times, gets 5 heads, and insists the coin is fair. For H₀: the chance of heads is 0.5, against Hₐ: the chance of heads is greater than 0.5, find the p-value and interpret it in one sentence.
A study of a sleep app reports a p-value of 0.03 for the claim that the app increases sleep, tested at α = 0.05. A headline says, "There is only a 3% chance the app does nothing." State the decision, and rewrite the headline's sentence correctly.
An airport bag scanner works with the null hypothesis "this bag is safe". Describe a Type I error and a Type II error for the scanner, and say what happens to Type II errors if the airport makes the scanner flag fewer bags.

Answers (worked reasoning — read after attempting)

The parameter is μ, the mean weight of all the bakery's sourdough loaves, in grams. The null hypothesis is the bakery's claim of no difference: H₀: μ = 500. The customer suspects lighter loaves, so Hₐ: μ < 500. Only loaves lighter than claimed would support the customer, so the alternative is one-sided. A customer deciding whether to complain needs this pair of sentences before collecting any data.
If the coin is fair, each flip lands heads with chance 0.5, and the flips are independent. So the chance of 5 heads in 5 flips is 0.5 × 0.5 × 0.5 × 0.5 × 0.5 = 1 ⁄ 32, about 0.03. Five heads is the most extreme result possible, so the tail holds only that one result. If the coin were fair, 5 heads in 5 flips would occur about 1 time in 32. At α = 0.05 you would reject H₀, but at α = 0.01 you would not, which is a reminder to choose α before you flip.
The p-value 0.03 is below α = 0.05, so the study rejects H₀, and the result is statistically significant. The headline misreads the p-value as the chance that H₀ is true. A correct version: "If the app had no effect on sleep, results this extreme would occur in about 3% of studies like this one." The p-value also says nothing about how much extra sleep the app gives, so a reader deciding whether to pay for it still needs the size of the effect.
A Type I error rejects a true null: the scanner flags a safe bag, and a traveler gets an extra search for nothing. A Type II error fails to reject a false null: the scanner passes a bag that is not safe. If the scanner flags fewer bags, Type I errors fall, so fewer travelers are searched for nothing, but Type II errors rise, so more unsafe bags get through. A miss costs far more than a false alarm here, which is why an airport chooses to flag many bags.

Next up in this module: the Readings & Resources page, the lecture tutorial, the practice set, and the quiz — and next week the same logic powers the one-sample t test and the test for a proportion.

Built with The Course Maker · samples.thecoursemaker.com