Week 13 · Chapter

Chapter 13 — Hypothesis Testing: Foundations

Introduction to Statistics Generic evergreen edition

Course: Introduction to Statistics (18-week generic edition)
The module's primary reading · ~25–35 minutes · Objective 7 · Read this before (or alongside) the week's lecture — everything else in the module builds on it.

Why you already care about this

The world makes claims at you all day long. Charges in 90 minutes. Our new formula works better. Lasts twice as long. Behind every one of those bold-type sentences, somebody measured something — and you already know, from Week 10, the uncomfortable truth about measurements: samples wobble. Even when a claim is perfectly true, your own sample will almost never match it exactly. So when your data disagree with a claim, there are always two suspects: the claim is wrong, or chance did it.

Hypothesis testing is the machinery for choosing between those two suspects — with a number attached. It is the single most used (and most misused) tool in all of applied statistics: it decides which medicines reach pharmacies, which headlines say "studies show," and which product claims survive a lawsuit. This week is deliberately light on calculation — every p-value in this chapter is handed to you by software or by a story's simple arithmetic — because the hard part of hypothesis testing was never the computing. It's the judgment.

By the end of this chapter you will be able to: write the two rival hypotheses (H₀ and Hₐ) for any claim, with the right direction; say in one sentence what a p-value means — and what it absolutely does not mean; make the reject / fail to reject decision against a significance level α and state the verdict in plain English; describe the two ways the verdict can be wrong (Type I and Type II errors); and explain why "statistically significant" is not the same as "actually important."

Here is the one-line version of the whole week, worth memorizing now: a hypothesis test never asks "is the claim true?" It asks: "if the claim were true, how surprising would my data be?"

Where we are

Weeks 11 and 12 built confidence intervals — a range of plausible values for an unknown parameter, powered by Week 10's sampling distributions. A hypothesis test asks the matching question from the other side: someone hands you one specific claimed value — is it plausible? Same engine, new verdict. Everything this week is logic; the t-test machinery for actually computing p-values arrives in Week 14, and it will feel easy once this week's ideas are solid.

1. The claim on trial: H₀ and Hₐ

Start with the picture that runs the whole week: every significance test is a tiny courtroom. The claim goes on trial. It gets the benefit of the doubt. The data are the evidence. And the verdict — like a real verdict — is either "guilty beyond reasonable doubt" or "not enough evidence," never "proven innocent."

Every test begins by naming two rival explanations:

The null hypothesis (H₀) is the dull explanation: nothing's going on. The claim is as stated; there is no change, no effect, no difference; anything odd-looking in the data is just sampling chance. H₀ is the defendant, and it keeps the benefit of the doubt until the evidence is strong.
The alternative hypothesis (Hₐ) is the suspicion: something IS going on — the conclusion we are not allowed to reach without convincing evidence.

Three rules govern how the pair is written:

Hypotheses are about parameters, never statistics. They describe the population truth — μ (a mean) or p (a proportion) — not your sample's x̄ or p̂. You don't put your own sample on trial; you already know exactly what it says. A quick self-test: a legal hypothesis could have been written down before any data existed.
H₀ always holds the equals sign. H₀: μ = 90 or H₀: p = 0.5 — the specific, testable "nothing's going on" value.
Hₐ points where the suspicion points, and the question — never a peek at the data — chooses the direction: < if you suspect the true value is too low, > if too high, or ≠ if you suspect it's simply different, either way (the two-sided alternative).

Worked example (this one runs all week). A phone-charger maker advertises: "Charges your phone from empty to full in 90 minutes, on average." A tech-review site suspects real-world charging runs slower than advertised and plans to time 25 full charges.

The parameter: μ = the true mean charge time, in minutes, across all charges of this model.
H₀: μ = 90 — the claim is as stated; a slow-looking sample would be ordinary wobble.
Hₐ: μ > 90 — charging genuinely takes longer than advertised.

Watch the direction carefully, because it's the classic slip: "slower" sounds like "less," but slower charging means more minutes, so the suspicion points right, toward larger μ. Always translate the suspicion into the measurement's direction before writing anything.

And a two-sided contrast: an oven's display claims the true oven temperature matches its 350° setting. A repair technician has no directional suspicion — the sensor could run hot or cold. H₀: μ = 350 vs. Hₐ: μ ≠ 350. No suspected direction → two-sided.

Misconception check. "We're testing whether x̄ = 90." No — hypotheses are about the population. Your sample mean is the evidence, not the defendant. Writing H₀: x̄ = 90 is like putting the eyewitness on trial.

2. The logic of a test: the tea experiment that started it all

The cleanest way to see how the courtroom actually works is the story of the first modern significance test.

Cambridge, England, in the 1920s. At an afternoon tea, a scientist named Muriel Bristol announces that she can taste whether the milk or the tea was poured into the cup first. The statistician Ronald Fisher, sitting at the same table, doesn't believe her — and instead of arguing, he designs an experiment on the spot. Eight cups: four poured milk-first, four tea-first, presented in random order. She knows there are four of each; her job is to say which four are which.

Now the logic, in four steps — this is the whole week in miniature:

Assume the dull explanation. H₀: she's guessing — no tasting ability at all.
Ask what the dull explanation would produce. A guesser is choosing 4 cups out of 8 to call "milk-first." There are exactly 70 equally likely ways to choose 4 of 8, and only one of them is completely correct. So P(a guesser sorts all eight cups correctly) = 1/70 ≈ 0.014.
Look at what actually happened. As the story is told, she got all eight right.
Deliver the verdict. If she were guessing, a performance this good would happen about 1.4% of the time. That is too surprising to shrug off. We reject "she's guessing" — the evidence says she really can taste the difference.

Notice what the argument did not do: it never computed the probability that she has the ability. It computed how well chance alone could imitate her — and found chance to be a lousy imitator. That is the entire logic of hypothesis testing: give the dull explanation every opportunity to account for the data, and reject it only when it fails badly.

One piece of vocabulary rides along. In real tests, software first computes a test statistic — a standardized score measuring how far the sample landed from what H₀ expected (the z-scores of Weeks 8 and 10 are exactly this idea). A sample far from expectation gives a big test statistic, which gives a small p-value. This week you only read the results; in Week 14 you'll compute a test statistic yourself.

In practice (your tools). You can build Fisher's kind of reasoning in a spreadsheet in two minutes. =RANDBETWEEN(0,1) is a fair coin flip; put 20 of them in a row, =SUM(...) the row to count heads, fill down 200 rows, and you've built a world where "nothing's going on" is true by construction. Then =COUNTIF(...) answers questions like "how often did chance alone produce 15 or more heads?" That is precisely this week's Data Lab — and =1 − BINOM.DIST(14, 20, 0.5, TRUE) gives the exact answer (0.0207) without simulating at all.

3. The p-value: what it means — and what it does not

The 0.014 in the tea story has a name, and its definition is the most important sentence of the week. Read it twice.

The p-value is the probability, computed assuming H₀ is true, of getting data at least as extreme as what was actually observed.

Every phrase is load-bearing. Assuming H₀ is true: the p-value lives inside a what-if world — it is a fact about what chance can do, not a fact about whether the claim is right. At least as extreme: it's the probability of a whole tail of outcomes (all eight cups, or anything as impressive), not of one exact outcome.

Reading the number is a one-line skill:

Small p-value → data this extreme are rare if H₀ is true → chance is a bad explanation → evidence against H₀.
Large p-value → data like this are ordinary if H₀ is true → chance explains them fine → no case against H₀.

Misconception check. "p = 0.014 means there's a 1.4% chance she was guessing" — the single most common misreading in statistics, and it's wrong. The p-value was computed by assuming she was guessing; it cannot turn around and report the probability of the thing it assumed. It is P(data at least this extreme | H₀ true), never P(H₀ true | data). The tea test's 0.014 is a statement about guessers — "guessing looks this good 1.4% of the time" — not a statement about her.

Two more misreadings to retire on sight: a p-value is not "the probability the results are due to chance" (close, but sloppy — it's the probability chance would produce results this extreme, a subtly different sentence that keeps the what-if where it belongs), and a large p-value is not proof that H₀ is true (Section 4 finishes that one).

4. α, the decision, and how to say the verdict

How small is "small enough"? Before running a test, we draw the line: the significance level α (alpha) — the "surprising enough" threshold, chosen before the data are examined. The near-universal default is α = 0.05. Then the decision is mechanical:

p ≤ α → reject H₀. The result is called statistically significant — the evidence cleared the bar.
p > α → fail to reject H₀. The evidence didn't clear the bar.

Why 0.05? Convention and history — not physics. α is best understood as a risk you accept: even when H₀ is true, p-values land at or below 0.05 about 5% of the time (that's just what p-values do), so α = 0.05 means "I'm willing to wrongly convict a true H₀ about 1 time in 20." Because the line is chosen, not discovered, two habits follow: report the p-value itself, not just the verdict; and treat p = 0.049 vs. p = 0.051 as nearly identical evidence, not a discovery versus nothing.

Worked example — the full walk. Back to the charger. (1) H₀: μ = 90 vs. Hₐ: μ > 90. (2) α = 0.05, set in advance. (3) The evidence: 25 timed charges average x̄ = 96 minutes. (4) The site's software reports p = 0.021 — meaning: if the true average really were 90 minutes, only about 2.1% of samples of 25 charges would average this high or higher. (5) The verdict: 0.021 ≤ 0.05 → reject H₀: "There is convincing evidence that this charger's true mean charge time exceeds the advertised 90 minutes."

That last sentence is a template. Memorize both endings:

Reject: "We reject H₀. There is convincing evidence that [Hₐ, in context]."
Fail to reject: "We fail to reject H₀. There is not convincing evidence that [Hₐ, in context]."

The companion case — and the trap. The same site tests the maker's travel charger, advertised at 150 minutes: 25 charges, x̄ = 153, and the software reports p = 0.38. Since 0.38 > 0.05, we fail to reject H₀ — there is not convincing evidence that the travel charger's true mean exceeds 150 minutes. Now the question that separates careful analysts from careless ones: did we just prove the 150-minute claim is accurate? No. Maybe the claim is fine — or maybe the true mean is 152 and a sample of 25 simply couldn't see a difference that small. A large p-value means the data are compatible with H₀, nothing more.

Misconception check. "p was large, so we accept H₀ — the claim is confirmed." Banned. "Not guilty" is not "innocent"; it means the evidence didn't clear the bar. We say fail to reject, never accept, because absence of evidence is not evidence of absence. Also banned: "H₀ is true" and "the claim is proven."

In practice (your tools). In real work you'll rarely compute a p-value by hand: spreadsheet functions (like =T.TEST next week) and every stats package print the test statistic and p-value for you. Technology's job ends there. The hypotheses, the α, the verdict sentence, and the judgment about what it all means — those never stop being your job.

5. The two ways to be wrong: Type I and Type II errors

Even a perfectly run test can deliver a wrong verdict, because evidence is probabilistic. There are exactly two ways, and they are not symmetric:

A Type I error is rejecting a true H₀ — convicting the innocent. The false alarm. Its probability is exactly α (that's what α is: the false-alarm risk you agreed to).
A Type II error is failing to reject a false H₀ — letting the guilty walk. The miss.

Worked example (a modern courtroom). An email spam filter runs a tiny hypothesis test on every message, with H₀: "this message is legitimate."

Type I: a real message from your instructor lands in the junk folder. The filter convicted an innocent email — a false alarm.
Type II: a scam email sails into your inbox. The filter missed a guilty one.

Which error is worse? That's a design decision, and here is the trade-off nobody escapes: make the filter less trigger-happy (lower α) and fewer real emails get junked — but more spam slips through. At a fixed sample size, tightening one error loosens the other. The only way to shrink both at once is better evidence — usually a bigger sample.

Consequences, not habit, should pick α. Screening aircraft parts for hidden cracks? A miss is catastrophic — tolerate more false alarms. Deciding whether a snack flavor needs reformulating? A false alarm merely wastes a taste test — the default 0.05 is fine.

The memory hook: "Type I cries wolf when there's no wolf. Type II sleeps through the real one."

6. Statistically significant vs. actually important

One last piece of judgment, and it's the one that makes you a sharp reader of headlines forever. In a hypothesis test, "significant" means "hard to explain by chance" — it does not mean "big" or "important."

Worked example. A running-shoe brand tests a new insole on 80,000 runners and finds marathon times improve by 4 seconds on average, with p = 0.001.

Statistically significant? Absolutely. With 80,000 runners, the test can detect whisper-thin differences, and chance is a terrible explanation for even a 4-second gap.
Practically important? Four seconds, in a race that takes hours. Would you pay extra for that insole?

With a huge sample, a trivial true effect reliably earns a tiny p-value. So a complete reading of any result asks two questions, always in this order: "Is it significant?" and "How big is the effect?" The p-value answers only the first.

The mirror image matters just as much: a small study can miss a real, important effect — a large Type II risk — so "we found no significant difference" from a tiny sample is weak information, not a debunking. "No evidence of effect" is not "evidence of no effect."

Misconception check. "p = 0.001 — the effect must be huge." The p-value measures surprise, not size. A tiny p can come from a big effect in a small sample or a microscopic effect in an enormous one. Ask for the effect size before you care.

Key terms

Null hypothesis (H₀) — the "nothing's going on" explanation: the claim as stated, no change, no effect; holds the equals sign; gets the benefit of the doubt.
Alternative hypothesis (Hₐ) — the suspicion requiring convincing evidence; points <, >, or ≠, with the direction chosen from the question before seeing data.
One-sided / two-sided alternative — an Hₐ pointing in one suspected direction (< or >) versus either direction (≠).
Test statistic — a standardized score measuring how far the sample landed from what H₀ expects; big distance → small p-value.
p-value — the probability, computed assuming H₀ is true, of data at least as extreme as what was observed.
Significance level (α) — the pre-chosen "surprising enough" line; equals the Type I (false-alarm) risk you accept; 0.05 by convention.
Statistically significant — p ≤ α; the data are hard to explain by chance alone. Says nothing about the size or importance of the effect.
Reject H₀ / fail to reject H₀ — the only two verdicts; "fail to reject" is never "accept" or "prove."
Type I error — rejecting a true H₀ (false alarm); probability α.
Type II error — failing to reject a false H₀ (miss); shrinks with more data, grows when α is tightened.
Practical significance — whether the effect is big enough to matter in the real world; judged by the effect's size, not by the p-value.

Summary

Every test is a courtroom: H₀ (nothing's going on) gets the benefit of the doubt; Hₐ (the suspicion) needs convincing evidence. Hypotheses describe parameters (μ, p) — never x̄ or p̂ — H₀ holds the equals sign, and the question picks Hₐ's direction before the data arrive.
The logic, since the tea test: assume H₀, ask how well chance imitates your data. The p-value = P(data at least this extreme | H₀ true). Small p → chance is a bad explanation → evidence against H₀. It is never the probability that H₀ is true.
Decide by comparing p to a pre-chosen α (usually 0.05): p ≤ α → reject (statistically significant); p > α → fail to reject. Deliver the verdict in context, with the template sentences — and never say "accept H₀."
Two possible wrong verdicts: Type I (false alarm; probability α) and Type II (miss). Tightening one loosens the other; consequences should choose α. Type I cries wolf; Type II sleeps through the real one.
Significant ≠ important. Huge samples make trivial effects significant; tiny samples miss real ones. Ask "how big?" right after "how surprising?"

Check your understanding

Work these before looking at the answers — they're ungraded, and the answers (with reasoning) are at the bottom of the page.

An insulated-bottle maker claims its bottle keeps drinks cold for 12 hours on average. A consumer group suspects the true average is less. Define the parameter and write H₀ and Hₐ in symbols. Is this test one-sided or two-sided — and what, specifically, in the story decides that?
A car wash advertises a 7-minute average wash. A skeptical customer times a sample of washes and runs a test of H₀: μ = 7 vs. Hₐ: μ > 7; the software reports a p-value of 0.32. (a) Write one sentence saying what 0.32 means here. (b) State the decision at α = 0.05 and the conclusion in context. (c) A friend says, "Great — so the 7-minute claim is proven right." What's wrong with that?
In a sport's anti-doping program, each athlete's test is a hypothesis test with H₀: "this athlete is clean." Describe a Type I error and a Type II error in this context, name one consequence of each, and explain why the lab can't drive both risks to zero at once just by moving its decision threshold.
A toothpaste company tests a reformulated paste on 60,000 people and reports a whiteness improvement of 0.2 points on a 100-point scale, with p = 0.001. The ad says "clinically proven whiter." Using this week's ideas, give the two-sentence assessment you'd offer a friend deciding whether to switch.

Answers (worked reasoning — read after attempting)

Parameter: μ = the true mean time (in hours) the bottle keeps drinks cold, across all bottles of this model. H₀: μ = 12 (the claim as stated, holding the equals sign) vs. Hₐ: μ < 12 (the suspicion: less than claimed). It is one-sided, because the consumer group suspects a specific direction — shorter cold time. Had they merely wondered whether the claim was off in either direction, Hₐ: μ ≠ 12 (two-sided) would be right. The direction comes from the question, decided before the data are seen.
(a) "If the true average wash really were 7 minutes, about 32% of samples like this one would average as high as ours or higher" — a statement made assuming H₀, about what chance produces. (b) Since 0.32 > 0.05, fail to reject H₀: there is not convincing evidence that the true average wash time exceeds 7 minutes. (c) The friend converted "no conviction" into "proven innocent." Failing to reject doesn't confirm the claim — the true average could be, say, 7.3 minutes with a sample too small to show it. The data are compatible with the claim; they don't prove it.
Type I: the test flags a clean athlete as doping — a false alarm; consequence: an innocent athlete is suspended, with reputation and livelihood damaged. Type II: the test passes an athlete who is doping — a miss; consequence: cheating goes unpunished and the field is unfair. Moving the threshold only trades one risk for the other: make the test harder to trigger and false alarms fall while misses rise; make it more sensitive and the reverse. At a fixed amount of evidence, the two risks see-saw — only better evidence (more or better testing) shrinks both.
Something like: "With 60,000 people, even a microscopic real difference becomes statistically significant — p = 0.001 says the 0.2-point gap is hard to blame on chance, not that it's big. Two-tenths of a point on a 100-point whiteness scale is invisible in a mirror, so 'proven whiter' is technically defensible and practically empty — I'd switch for price, not for this." (Any answer that separates surprise from size and questions the practical importance of 0.2/100 earns full marks.)

Next up in this module: the slide deck if you want the lecture's visual walk-through, then the Readings & Resources page for optional deeper dives — and when you're ready to practice, Lecture Tutorial 13 turns your chatbot into your personal Week 13 tutor.