← Back to the Introduction to Statistics outline The Course Maker
← Introduction to Statistics outline
Week 13 · Data Lab

Week 13 — Data Lab · "Could Chance Do That? Putting 15-of-20 on Trial"

Introduction to Statistics Generic evergreen edition

Course: Introduction to Statistics (18-week generic edition)
Objective: Objective 7 — the logic of significance testing: null worlds, p-values, and verdicts · SLO A (reason quantitatively from data)
Worth 50 points · Data labs group = 15% of the grade · Data Lab 13 · Due: end of Week 13
Format: a hands-on data lab — this week you build the null hypothesis with your own hands: 200 sets of 20 fair coin flips in a spreadsheet, then you locate a suspicious result inside that world and deliver the verdict; then you catch the AI's mistakes when it interprets your work.

This is the course's signature weekly component. Every instructional week has a Data Lab: real, free, linked public data — nothing to buy, download, or sign up for — worked in your spreadsheet, finished with an AI-critique step. Plan on 60–90 minutes. (This week the "dataset" is one your own spreadsheet generates — an entire world where the null hypothesis is true by construction.)

Part 1 — The Big Picture

This week's chapter said every significance test asks one question: if nothing were going on, how surprising would my data be? Today you stop imagining that "if" and build it. Your spreadsheet will manufacture 200 honest sets of 20 fair coin flips — a complete world where H₀ ("the coin is fair, heads is no more likely than tails") is true by construction — and then you'll ask that world a pointed question: how often does plain chance produce 15 or more heads in 20 flips?

The setup: at a weekend market, a performer charges a dollar to demonstrate "heads on demand — I can make it land heads." You watch 20 flips and count 15 heads. Impressive… or exactly the kind of streak fair coins produce all the time? By the end of the lab you'll have an empirical p-value (from your simulation), the theoretical p-value (from Week 7's binomial, supplied by a spreadsheet function), a verdict at α = 0.05 — and, just as important, a clear view of what that verdict does and doesn't prove.

Optional 7-minute background read before you start: The Binomial Distribution (Math is Fun) 🔗 https://www.mathsisfun.com/data/binomial-distribution.html

Part 2 — The Task

The guiding question: Where does 15-of-20 sit in a world where the coin is fair — ordinary wobble, or out in the tail where chance almost never goes — and what verdict does that location earn?

The hypotheses on trial (write these into your submission): with p = the true probability the performer's flip lands heads — H₀: p = 0.5 (fair; "heads on demand" is theater) vs. Hₐ: p > 0.5 (heads really is more likely on their flips). The suspicion points up, so the extreme results are high head-counts.

The tools (nothing to buy or download): a spreadsheet — Google Sheets (free) or Excel. Its =RANDBETWEEN(0,1) function is a fair coin: each call produces 0 or 1 (call 1 "heads"), each with probability 0.5, with no memory of other calls.

Part 3 — Materials & Procedure

You need: a spreadsheet and about 60–90 minutes. Nothing else.

Warm-up (5 min, no spreadsheet needed). Three small computations by hand or calculator — write all three down; they're your benchmarks for reading the simulation:

  1. The expected number of heads in 20 fair flips: 20 × 0.5 = ______.
  2. The standard deviation of that head-count (Week 7's formula √(np(1−p))): √(20 × 0.5 × 0.5) = √5 = ______ (round to two decimals).
  3. Using answers 1 and 2: the observed 15 heads sits (15 − 10) ÷ 2.24 ≈ ______ standard deviations above expected (round to one decimal). Keep that number in mind as you watch the simulation.

Now build the null world:

  1. Label the columns. In row 1, type flip1 in A1, then fill the label across through T1 (flip20); in U1 type heads.
  2. Flip one set of 20. In A2 type =RANDBETWEEN(0,1) and copy it across A2:T2 — twenty fair flips, where 1 = heads.
  3. Count the set's heads. In U2 type =SUM(A2:T2).
  4. Make 200 sets. Select A2:U2, copy, and paste into rows 3 through 201. Column U now holds the head-counts of 200 honest sets. (They re-flip every time the sheet recalculates — expected; see step 8.)
  5. Ask the tail question. In W2 type =COUNTIF($U$2:$U$201, ">=15") — how many of your 200 fair sets produced 15 or more heads? In W3 type =W2/200 — your empirical p-value for the performer's 15.
  6. Get the exact answer too. In W4 type =1 - BINOM.DIST(14, 20, 0.5, TRUE) — the theoretical P(X ≥ 15) for a fair coin. It returns 0.0207; your W3 estimates this same tail by brute force.
  7. Picture the world. Select U2:U201 → Insert ▸ Chart → histogram (Excel: Insert ▸ Statistic Chart ▸ Histogram). You should see a mound centered near 10, thinning fast toward the edges. Give it a title like "200 sets of 20 fair flips."
  8. Freeze your run. Select columns A:U → Copy → Paste special ▸ Values only onto the same cells (Excel: Paste Values). Your 200 sets are now frozen data.
  9. Profile the null world. In empty cells: =MIN($U$2:$U$201), =MAX($U$2:$U$201), and =COUNTIFS($U$2:$U$201, ">=6", $U$2:$U$201, "<=14") — the count of "ordinary" sets (6–14 heads).
  10. Run the companion case. A classmate watched a different performer flip 12 heads in 20. In X2: =COUNTIF($U$2:$U$201, ">=12"), in X3: =X2/200 (empirical p-value for 12), and in X4: =1 - BINOM.DIST(11, 20, 0.5, TRUE) — the theoretical P(X ≥ 12) = 0.2517.

Plausibility note (read before judging your run): with 200 sets, the count reaching 15+ heads is usually 1 to 8 sets (empirical p-value roughly 0.005–0.04), and about 1 run in 65 shows zero such sets — if that's yours, your empirical p-value is 0, read as "rarer than 1-in-200," and nothing is broken. The 12+ count typically lands around 38–63 sets. Your histogram's mound should sit near 10 with almost every set between 6 and 14.

Fallback (no spreadsheet available): flip a real coin 20 times and tally heads; repeat for as many sets as you reasonably can (aim for 10) and say you used the physical fallback. For the 200-set questions, use this model run (a frozen run of the same simulation; counts sum to 200): 3 heads × 1 set · 5 × 1 · 6 × 5 · 7 × 8 · 8 × 22 · 9 × 44 · 10 × 33 · 11 × 24 · 12 × 35 · 13 × 14 · 14 × 7 · 15 × 6. From it: sets with ≥ 15 heads = 6 (empirical p-value 0.03); sets with ≥ 12 heads = 62 (empirical p-value 0.31); sets with 6–14 heads = 192; min 3, max 15.

Part 4 — The Scaffold (complete both tables)

Table 1 records the trial of the performer's 15-of-20, from hypotheses to verdict.

Item Your entry
H₀ (symbols and words) ______
Hₐ (symbols and words, with direction) ______
Observed result on trial 15 heads in 20 flips
Sets (of your 200) with ≥ 15 heads ______
Empirical p-value (count ÷ 200) ______
Theoretical p-value (from =1-BINOM.DIST(14,20,0.5,TRUE)) ______
Decision at α = 0.05 (with the template sentence) ______

Table 2 profiles your null world and runs the companion case.

Quantity Value
Expected heads per set (warm-up 1) ______
Smallest head-count in your 200 sets (=MIN) ______
Largest head-count in your 200 sets (=MAX) ______
Sets with 6–14 heads ("ordinary" sets, =COUNTIFS) ______
Sets with ≥ 12 heads ______
Empirical p-value for the companion's 12-of-20 ______
Theoretical P(X ≥ 12) (from =1-BINOM.DIST(11,20,0.5,TRUE)) ______
Decision for the companion performer at α = 0.05 ______

Part 5 — Analysis Questions

Answer in 1–3 sentences each:

  1. Describe your histogram — where's the mound, and how far out do the edges reach? Explain why this picture deserves the name "the null world": what, exactly, is true in it by construction?
  2. Your empirical p-value and the theoretical 0.0207 are close but (almost certainly) not equal. Are they estimates of the same thing or different things — and why don't they match exactly? (Your Week 5 long-run ideas apply.)
  3. Deliver the verdict on "heads on demand" at α = 0.05 using the course template sentence. Then the careful part: does rejecting H₀ prove the performer has a skill? Name at least one explanation other than skill or plain chance that your test cannot rule out (think about the coin itself, or how the flip is thrown).
  4. The companion performer's 12-of-20 gives a p-value around 0.25. State that verdict with the template sentence — and explain why this result does not prove that performer's flips are fair.
  5. (Connect forward.) Suppose the market ran this same α = 0.05 test on 100 honest performers whose coins are all perfectly fair. About how many would be wrongly "convicted," what is that error called — and why does this number explain the course rule that α is chosen before the data arrive? (Next week this exact logic runs on means, with the t-test doing the tail arithmetic.)

Part 6 — AI-Critique Moment (required — the BYOAI step)

Now bring in your chatbot and be the analyst who checks its work.

  1. Paste your completed Table 1 into your chatbot, along with this claim: "My empirical p-value came out a bit different from the theoretical 0.0207, so I think my simulation is broken. Also, since the p-value is small, there's only about a 2% chance the coin is fair — right? Please explain my results."
  2. Check everything it says against your own work. This week's three named AI failure modes:
    - Endorsing the misread — agreeing (or half-agreeing, to be agreeable) that a small p-value is "the chance the coin is fair." It isn't: the p-value assumes fairness and reports how often chance looks this extreme — P(data | H₀), never P(H₀ | data).
    - Noise read as a bug — calling your simulation "broken" because the empirical and theoretical p-values differ. With only 200 sets, empirical counts of ≥ 15 anywhere from about 1 to 8 are ordinary; both numbers estimate the same tail.
    - Arithmetic slips — a COUNTIF miscount, or an empirical p-value not equal to (count ÷ 200). Re-do the division yourself.
  3. Write 2–3 sentences reporting what the AI got right and at least one thing you had to correct, question, or verify. If it happened to get everything right, say how you verified each claim — that's the skill.

The habit all term: the tool drafts, you judge. A chatbot will confidently call your honest simulation broken — or soothe you with the p-value misread. Catching it is the point.

Part 7 — What to Submit

One document or text entry containing: your three warm-up values, your completed Table 1 and Table 2, a screenshot or one-sentence description of your histogram, your Part 5 answers (1–5), and your Part 6 AI-critique paragraph. Due at the end of Week 13 · 50 points (rubric below the key).

Instructor answer key & model values — REMOVE BEFORE PUBLISHING TO STUDENTS

Every student's 200 sets are their own, so simulated counts and empirical p-values are graded on method and interpretation, not on matching any printed value. The theoretical values and plausibility bands below are pre-computed and verified in the week's math-check script; no claim here depends on an unverified number.

Warm-up (fixed — every student should match): (1) 20 × 0.5 = 10. (2) √5 = 2.24 (2.236…; accept 2.24 or 2.236). (3) (15 − 10) ÷ 2.24 ≈ 2.2 standard deviations above expected (accept 2.2–2.24) — already whispering "tail."

Theoretical values (verified in script): P(X ≥ 15 | n = 20, p = 0.5) = 21,700 ⁄ 1,048,576 = 0.0207 (the =1-BINOM.DIST(14,20,0.5,TRUE) value). P(X ≥ 12) = 263,950 ⁄ 1,048,576 = 0.2517. P(6 ≤ X ≤ 14) = 0.9586, so about 192 of 200 sets are "ordinary" (anywhere ~185–198 is unremarkable). A two-sided version of the 15-heads test would double the tail to ≈ 0.0414 — not used this week, but a useful margin note if a student asks about ≠.

Simulation plausibility bands (theory, verified in script): the count of sets with ≥ 15 heads is binomial with n = 200, p = 0.0207: mean 4.1, SD 2.01 — so 1 to 8 sets is the ordinary range (empirical p-value 0.005–0.04), and P(zero such sets) ≈ 0.015 (~1 run in 65; that student's empirical p-value is 0 — grade it as correct method, read as "rarer than 1-in-200"). The count with ≥ 12 heads has mean 200 × 0.2517 = 50.3, SD 6.1 — roughly 38–63 sets. Flag only formula errors (unanchored COUNTIF ranges, dividing by the wrong denominator), never luck.

Table 1 grading: H₀: p = 0.5 with "the coin/flips are fair — heads no more likely" in words; Hₐ: p > 0.5 with the direction justified (the claim is heads on demand → high head-counts are the extreme ones). Empirical p-value must equal their own count ÷ 200 (re-do this division — it's the arithmetic actually graded). Theoretical cell = 0.0207. Decision: 0.0207 (and almost every empirical value) ≤ 0.05 → reject H₀ with the template sentence — "convincing evidence that heads is more likely than 0.5 on this performer's flips."

Table 2 grading: expected heads 10; min/max/counts from their own frozen run (typical: min 3–5, max 15–17, ordinary count near 192); companion cells: their count ÷ 200, theoretical 0.2517, and fail to reject for 12-of-20, with the "not convincing evidence" template.

Part 5 expected answers:
1. Mound near 10, edges thinning by 5–6 and 14–15; it's "the null world" because H₀ — a fair coin, p = 0.5 — is true by construction: every number in it was manufactured under the null, so it shows exactly what chance alone can do.
2. The same thing: both estimate P(X ≥ 15 | fair coin). The empirical version is a 200-run long-run relative frequency, so it wobbles around the exact 0.0207 (Week 5's law of large numbers: more sets → closer).
3. "We reject H₀: there is convincing evidence that heads is more likely than 0.5 on these flips." Then: rejecting does not prove skill — the test discredits chance with a fair coin, not every rival story: a weighted/two-headed coin, a practiced biased toss, or even watching only the performer's best streak (selective viewing) all survive. Statistical evidence rules out luck, not alternative mechanisms — that's this lab's limitation question.
4. "We fail to reject H₀: there is not convincing evidence that heads is more likely than 0.5 for the companion performer." Not proof of fairness: 12-of-20 is compatible with a fair coin and with, say, a mildly loaded one — "not guilty" ≠ "innocent"; absence of evidence is not evidence of absence.
5. About 5 of 100 honest performers get wrongly convicted — Type I errors — because α = 0.05 is the false-alarm rate against true nulls. That's precisely why α must be fixed before the data arrive: it's the amount of injustice the tester agreed to tolerate, not a knob to turn after seeing a tempting result. (Forward link: Week 14 runs this identical logic on means, with the t-statistic doing the tail work.)

AI-critique full credit: names a specific claim checked or corrected — most commonly the chatbot endorsing "2% chance the coin is fair," agreeing the simulation is "broken" over an ordinary empirical-vs-theoretical gap, or a count ÷ 200 slip — OR documents verification of each claim (definition restated correctly, gap checked against the 1–8 band, division re-done). Zero credit for "the AI agreed with everything" with no evidence of checking.

Grading rubric — 50 points

The rubric below maps each part to its points; bands are full / partial / none.

Criterion Full Partial None
Warm-up + build — 10, 2.24, and ≈2.2 correct; the 20-column × 200-row null world constructed with the specified formulas (8) 8 4–6 0–3
The trial (Table 1) — hypotheses correct in symbols and words with direction justified; empirical p-value = own count ÷ 200; theoretical 0.0207 recorded; correct verdict in template language (12) 12 6–10 0–5
Histogram + null-world profile (Table 2) — histogram present and read correctly; min/max/ordinary-count filled from own run; companion case (12-of-20) completed with correct fail-to-reject verdict (10) 10 5–8 0–4
Analysis questions — Q1–Q5 hit the expected ideas, especially the null-world meaning (Q1), empirical vs. theoretical (Q2), and the two "doesn't prove" traps (Q3–Q4) (15) 15 8–12 0–7
AI-critique — specific catch or verification reported in 2–3 sentences (5) 5 3 0–2

Quality gate (self-checked): warm-up arithmetic re-verified (10; √5 ≈ 2.24; ≈2.2 SDs); every theoretical value and band in this key (0.0207; 0.2517; 0.9586 → ≈192/200; count bands 1–8 and 38–63; P(zero) ≈ 0.015; about 5 of 100 Type I; two-sided 0.0414) re-computed in the week's math script; the fallback model run's tally (sums to 200; 6 sets ≥ 15 → 0.03; 62 sets ≥ 12 → 0.31; 192 ordinary; min 3, max 15) regenerated and verified in the same script; no assertion about any student's random results beyond method and plausibility bands; simulation uses =RANDBETWEEN per the course's simulation-lab convention with expected ranges, not exact values; no graded-item leakage (the quiz, practice, assignment, and chapter self-checks use different surfaces and numbers — the coin world lives only here and in the lecture's technology demo); optional background link verified live; rubric sums 8 + 12 + 10 + 15 + 5 = 50. ✓