Lecture: Hypothesis Testing: Foundations
Lecture script · about 15–20 minutes spoken · Week 13
Opening
A clinical team wants to know whether a new treatment beats the old one. A quality inspector wants to know whether a machine has drifted off its setting. A teacher wants to know whether a quiz score is too high to be a lucky guess. All three start the same way. They assume nothing has changed, and then they ask how surprising the data would be if that were true. That move is today's whole lecture.
Here's where we are. In Weeks 1 to 3 you learned to describe one variable, and Week 4 related two. Weeks 5 to 7 built probability and the binomial. Week 8 gave you the z-score, Week 9 the normal model, and Week 10 sampling distributions. Then Weeks 11 and 12 used all of that to build confidence intervals, an estimate with an honest margin of error. Today we ask a different kind of question. Not "what is the true value?" but "is this one claimed value believable?" Next week, in Week 14, you'll run full tests for means and proportions.
I'll take eight steps in order: two hypotheses, the logic of a test, the test statistic, the p-value, what a p-value is not, the decision, the two kinds of error, and the difference between significant and important. Almost everything runs on one story, a guessing quiz you already know from Week 7.
1. Two hypotheses: name the claim you doubt
Picture this. A student hands in a 10-question quiz with four options per question. They score 7, and they tell you they guessed on every single question. Do you believe them?
You can tell two stories about that 7. In the first, the student really guessed and got lucky. In the second, the student knew some answers. A hypothesis test writes those two stories down precisely, and it writes them about a parameter, a fixed number that describes the whole process. Here the parameter is p, the student's true chance of getting any one question right. A pure guesser has p equal to 0.25, one right option out of four.
The null hypothesis is the claim of no effect or no difference that a test assumes is true for the sake of argument, written H₀, said "H naught". Here it says p equals 0.25: the student is guessing. The alternative hypothesis is the claim a test looks for evidence of, written Hₐ, said "H sub a". Here it says p is greater than 0.25: the student does better than guessing.
Why "greater than" and not "different from"? Because only a high score would make you doubt the guessing story. A very low score wouldn't suggest the student knew the material. When only one direction matters, you have a one-sided alternative. When both directions matter, you have a two-sided alternative, written with a not-equal sign. A biologist asking whether female and male penguins are equally common would use one, because a share far above one half or far below it would both be news.
Pause and try this. A board-game player suspects a die lands on six too often. Is that alternative one-sided or two-sided?
Think about which direction would worry the player.
Here's the answer. It's one-sided. Only too many sixes would back up the suspicion, so the alternative says the chance of a six is greater than one in six.
2. The logic of a test: assume the null, then measure surprise
You already know how a hypothesis test thinks, because you know how a trial works. The defendant is presumed innocent. Then the evidence comes in. If the evidence would be very hard to explain for an innocent person, the jury rejects the presumption and says "guilty". If it could easily happen to an innocent person, the jury says "not guilty".
Notice what a jury never says. It never says "innocent". "Not guilty" only means the evidence wasn't strong enough.
A hypothesis test is a procedure, also called a significance test, that weighs data against a claim about a parameter by assuming the claim of no effect is true and asking how surprising the data would be under it. So it makes three moves. First, presume the null is true. Second, ask how surprising the data would be under that presumption. Third, decide: very surprising data lead you to reject the null, and ordinary data leave it standing.
Apply that to our student. Presume they guessed. Under guessing, the average score is 2.5 correct answers out of 10. A score of 7 sits far above that. So the whole case comes down to one question: how often would a real guesser score 7 or more?
Pause and try this. In the trial picture, what plays the role of the null hypothesis?
Give it a second; it's the thing the jury starts out assuming.
Here's the answer. The presumption of innocence. The jury assumes it only to see whether the evidence overturns it. A test assumes the null in the same way.
3. Test statistics: measure how far the data sit from H-naught
Surprise needs a ruler. A score of 7 is far from 2.5, but "far" only means something once you know how much guessers' scores usually spread out.
A test statistic is a number computed from the sample that measures how far the data sit from what the null hypothesis predicts. For the quiz, the simplest one is the count itself: 7 correct answers. To see how far that is, borrow Week 8's z-score. Under guessing, the scores have a standard deviation of about 1.37 correct answers. Take the gap, 7 minus 2.5, which is 4.5, and divide by 1.37. You get about 3.29. So the student's score sits about 3.29 standard deviations above what guessing predicts.
How big is that? Remember the Gentoo penguin from Week 8 that seemed unusually light. It sat about 2.13 standard deviations below its species average. Our student sits even farther from the guessing average, and on the high side.
One honest caution. The guessing scores don't follow a bell curve. In Week 7 you saw that the expected count, 2.5, is below 10, so the large-counts rule of thumb fails. That means you shouldn't turn 3.29 into a probability with a normal table. You'll use the exact guessing-quiz table instead.
Pause and try this. Does a test statistic of 3.29 settle the question by itself?
Hold that thought for one breath.
Here's the answer. No. It's a distance, a ruler reading. You still have to turn it into a probability, and that probability comes from the null model.
4. p-values: turn surprise into a probability
Here's the number that does the real work. A p-value is the probability, computed assuming the null hypothesis is true, of getting data at least as extreme as the data observed.
Three pieces matter. "Assuming the null is true" means you work inside the guessing story, with p equal to 0.25. "At least as extreme" means you count the score that happened and every score that would be even more surprising. Here that means 7, 8, 9, and 10. And "data" means the p-value is about scores, not about the student.
From the Week 7 table, a guesser gets exactly 7 right with chance 0.0031 and exactly 8 right with chance 0.0004. Nine and ten both round to zero. Add them up and the p-value is 0.0035. In a spreadsheet you'd type one minus BINOM.DIST of 6, 10, 0.25, TRUE, and you'd see the same 0.0035.
Now say it in a sentence, using the course's template. If the student were purely guessing, a score of 7 or more would happen on about 0.35% of quizzes, about 35 quizzes in 10,000. That's rare.
Compare two other scores so you get a feel for it. A score of 5 or more has a p-value of 0.0781, which happens on about 8 quizzes in 100. A score of 3 or more has a p-value of about 0.47, so about 47 guessers in 100 do that well. A 3 tells you nothing.
Pause and try this. Someone reports the p-value for a score of 7 as 0.0031. What did they leave out?
Look back at what "at least as extreme" covers.
Here's the answer. They counted only a score of exactly 7. The p-value also counts 8, 9, and 10, which brings it to 0.0035.
5. What a p-value is not: three wrong readings
The p-value is the most misread number in statistics. In 2016 the American Statistical Association put out a statement with six principles about p-values, and this course checks every p-value sentence against them. Three of those principles warn about the mistakes you'll hear most.
A p-value tells you how incompatible the data are with a specified model. It is not the probability that the null hypothesis is true. It is not the probability that the result was produced by chance alone. And it is not the size or importance of an effect.
Let's try them on our 0.0035. "There's a 0.35% chance the student guessed" is wrong, because the p-value was computed by assuming the student guessed. It can't also be the chance of guessing. "There's a 0.35% chance the score was a fluke" is wrong for the same reason. And "the student must know a lot" is wrong too. A student who knows a little and one who knows a lot could both produce small p-values.
Here's a picture that helps. A smoke detector chirps. The chirp tells you something about the air. It doesn't tell you how big the fire is, and it doesn't hand you a probability that the house is safe. You still have to go and look.
Pause and try this. Fix this sentence: "The p-value of 0.0035 means a 0.35% chance the null is true."
Try starting your sentence with the word "if".
Here's the answer. If the student were guessing, a score of 7 or more would occur on about 0.35% of quizzes. The p-value describes the data under the null; it isn't the chance the null is true.
6. Decisions: compare the p-value with alpha
A test ends in a decision, and a decision needs a cutoff. The significance level is the cutoff alpha, chosen before the data are seen, that a p-value is compared with. The most common choice is 0.05, and you'll also see 0.01 and 0.10.
The rule has two branches. If the p-value is at or below alpha, you reject the null hypothesis: the data give convincing evidence against it, though they don't prove the alternative. That result is called statistically significant. If the p-value is above alpha, you fail to reject the null hypothesis: the data are consistent with it.
Run two students through one rule, with alpha set at 0.05 before anyone sees a score. The student who scored 7 has a p-value of 0.0035. That's below 0.05, and below 0.01 too, so you reject. The score is strong evidence the student wasn't purely guessing. A second student scored 5. Their p-value is 0.0781. That's above 0.05, so you fail to reject. A score of 5 is consistent with guessing.
Here's the catch. At an alpha of 0.10, that same 0.0781 would reject. The second student's verdict depends on alpha. That's exactly why you choose alpha first. If you pick it after seeing the p-value, you can reach whatever decision you wanted.
Pause and try this. With alpha at 0.05, what's the decision for a p-value of 0.0781?
Compare the two numbers in your head.
Here's the answer. Fail to reject the null. The p-value 0.0781 is above 0.05, so a score of 5 is consistent with guessing.
7. Type I and Type II errors: two ways to be wrong
Every decision under uncertainty can go wrong in two directions. Think of a smoke alarm again. Its null hypothesis is "no fire". If it shrieks at burnt toast, it rejected a true null. That's a false alarm. If it stays quiet during a real fire, it failed to reject a false null. That's a miss.
Those mistakes have names. A Type I error is rejecting a null hypothesis that is actually true, and its probability is alpha. A Type II error is failing to reject a null hypothesis that is actually false, and its probability is called beta. And the power of a test is the probability that it rejects the null when the null is false, which is 1 minus beta. Power is the chance you catch a real effect.
Now make it concrete. Suppose a teacher flags any student whose score looks too high to be a guess. Rule A flags 5 or more. Then a pure guesser gets flagged with probability 0.0781. That's Rule A's Type I error rate. Rule B flags 7 or more, so a pure guesser gets flagged with probability 0.0035. Rule A versus Rule B is about a 22-fold difference in false alarms: roughly one honest guesser in 13 under Rule A, and one in 286 under Rule B.
But Rule B pays for that. A student who really knows a few answers might score 5 or 6. Rule A catches them; Rule B misses them. So the stricter rule makes more Type II errors. Push alpha down and beta goes up. Which mistake is worse depends on what each one costs.
Pause and try this. Which rule has more power to catch a student who knows a little: Rule A or Rule B?
Think about which rule flags more scores.
Here's the answer. Rule A. It flags more scores, so it catches more real effects, at the price of more false alarms.
8. Statistical versus practical significance: size matters, and silence is not proof
Here's a puzzle. Take one score, 40%, and change only the length of the quiz.
On 10 questions, 40% means 4 right. The p-value for 4 or more is 0.2241, so about 22 guessers in 100 do that well. Nothing surprising.
Now a 100-question quiz. Guessing averages 25 right, with a standard deviation of about 4.33, and here a bell curve is a fair model. A score of 40 sits about 3.46 standard deviations above 25. The empirical rule says fewer than about 2 guessers in 1,000 land that far out, so the p-value is well below 0.01.
Same 40%. Wildly different p-values. The p-value shrank because the quiz got longer, not because the student got better. And 40% still means beating guessing by only 15 questions out of 100.
That's practical significance: whether an effect is large enough to matter in its real-world setting. A p-value doesn't measure it. A tiny, unimportant effect can be statistically significant in a large enough sample. So whenever you report a p-value, report the size of the effect beside it.
One more thing. Go back to the student who scored 5, where we failed to reject. That does not mean we accept the guessing story. A student who knows a little could also score 5, and the test can't tell the two apart. The data are silent, not supportive. It's the jury again: "not guilty" never means "innocent".
Pause and try this. True or false: a smaller p-value always means a bigger effect.
Remember the two quizzes.
Here's the answer. False. The same 40% gave a p-value of 0.2241 on 10 questions and one well below 0.01 on 100 questions. Sample size moves the p-value too.
Closing
Let's gather it up. A test starts with two hypotheses about a parameter: the null, which names the boring value, and the alternative, which points one way or both ways. You presume the null and ask how surprising the data would be. A test statistic measures the distance, and the p-value turns it into a probability. For our student, 7 or more correct happens on about 0.35% of guessed quizzes. That p-value is not the chance the null is true, not the chance of a fluke, and not the size of the effect. You compare it with an alpha you chose in advance, and you either reject or fail to reject. A Type I error is a false alarm and a Type II error is a miss, and lowering one raises the other. Significant doesn't mean big, and failing to reject never proves the null.
Next in this module come the Readings & Resources page, the lecture tutorial, the practice set, and the quiz. Then come the discussion, the assignment, and the Data Lab, where you'll rebuild the guessing model in a spreadsheet and find this p-value yourself. Next week, in Week 14, Tests for Means & Proportions, you'll put this same logic to work on penguin weights and shares.
Built with The Course Maker · samples.thecoursemaker.com