← Back to the Introduction to Statistics outline The Course Maker
← Introduction to Statistics outline
Week 15 · Data Lab

Week 15 — Data Lab · "Split the Traffic: Your Own A/B Test"

Introduction to Statistics Generic evergreen edition

Course: Introduction to Statistics (18-week generic edition)
Objective: Objective 7 — the two-proportion z-test, run end to end; power and Type II error experienced firsthand · SLO A (reason quantitatively from data)
Worth 50 points · Data labs group = 15% of the grade · Data Lab 15 · Due: end of Week 15
Format: a hands-on data lab — this week you play both roles in a tech company's most common experiment: you rig the truth (two checkout pages with genuinely different conversion rates), simulate 400 shoppers, and then see whether the two-proportion z-test catches the difference you know is there; then you catch the AI's mistakes when it interprets your runs.

This is the course's signature weekly component. Every instructional week has a Data Lab: real, free, linked public data — nothing to buy, download, or sign up for — worked in your spreadsheet, finished with an AI-critique step. Plan on 60–90 minutes. (This week the "dataset" is one your own spreadsheet generates — 400 simulated shoppers who exist nowhere else in the world.)

Part 1 — The Big Picture

Every A/B test in industry has a dirty secret: the analyst never knows the truth. They see two conversion rates wobble in, run the test, and hope the verdict matches a reality they can't check. Today you get the one thing they never have — omniscience. You will build the truth: an old checkout page that converts exactly 10% of shoppers in the long run, and a redesigned page that converts exactly 20%. Then you'll simulate 200 shoppers on each, run the week's two-proportion z-test, and find out whether the test catches a difference you know is real.

Here is the spoiler worth the whole lab: it won't always. Run the experiment several times and most runs will reject H₀ loudly — but about one honest run in five will shrug and fail to find anything, even though the difference is baked in. That miss is Week 13's Type II error made personal, and the catch-rate has a name — power. After today, "the study found no significant difference" will never again sound to you like "there is no difference."

Optional 6-minute background read before you start: An Easy Introduction to Statistical Significance (Scribbr) 🔗 https://www.scribbr.com/statistics/statistical-significance/

Part 2 — The Task

The guiding question: The redesigned page really is better — you built it that way. Will YOUR 400 simulated shoppers reveal it, how loudly, and how often does an honest experiment miss a real effect?

The tools (nothing to buy or download): a spreadsheet — Google Sheets (free) or Excel. Its =RANDBETWEEN(1,10) function is a 10-sided die: a roll of 1 is a shopper who converts on the old page (a 10% chance). =RANDBETWEEN(1,5) is a 5-sided die: a 1 converts on the new page (a 20% chance). No memory, no mercy — exactly like real traffic.

Part 3 — Materials & Procedure

You need: a spreadsheet and about 60–90 minutes. Nothing else.

Warm-up (5 min, no spreadsheet needed). Two small computations by hand or calculator — write both down; they're your benchmarks:

  1. Expected conversions in 200 shoppers per page: old page 200 × 0.10 = __ ; new page 200 × 0.20 = ____ .
  2. In one example run (you'll check your machinery against it later), the old page converted 18 of 200 and the new page 42 of 200. The pooled proportion is (18 + 42) ÷ 400 = ______ .

Now build the simulation:

  1. Set up headers. In a blank sheet, type into row 1: A1 = shopper, B1 = old page roll (1-10), C1 = new page roll (1-5).
  2. Number the shoppers. A2 = 1. A3 = =A2+1. Copy A3 and paste into A4:A201. Column A should count 1 to 200 (check: A201 shows 200).
  3. Send traffic to the old page. B2 = =RANDBETWEEN(1,10). Copy B2, paste into B3:B201. Every 1 in column B is a conversion — a 10% chance per shopper.
  4. Send traffic to the new page. C2 = =RANDBETWEEN(1,5). Copy into C3:C201. Every 1 in column C is a conversion — a 20% chance per shopper.
  5. Build the test machinery in a results block (labels in column E, formulas in F):
    - F2 (old conversions) = =COUNTIF(B2:B201,1)
    - F3 (new conversions) = =COUNTIF(C2:C201,1)
    - F4 (p̂ old) = =F2/200 · F5 (p̂ new) = =F3/200
    - F6 (pooled p̂) = =(F2+F3)/400
    - F7 (SE) = =SQRT(F6*(1-F6)*(1/200+1/200))
    - F8 (z) = =(F5-F4)/F7
    - F9 (two-sided p-value) = =2*(1-NORM.S.DIST(ABS(F8),TRUE))
  6. Verify your machinery against the example run. Temporarily type 18 over F2 and 42 over F3 (plain numbers). F6 should show 0.15, F7 ≈ 0.0357, F8 ≈ 3.36, F9 ≈ 0.0008. If yours disagree, fix the formulas now. Then restore F2 and F3 to their =COUNTIF(...) formulas.
  7. Record Run 1. Copy the current values of F2, F3, F4, F5, F8, F9 into the first row of Scaffold Table 2, and mark whether p < 0.05 ("reject? yes/no").
  8. Re-roll for Runs 2–4. Type any letter into an empty cell (say J1) and press Enter — every RANDBETWEEN re-rolls, and your results block updates. Record the new F-values as Run 2. Repeat twice more for Runs 3 and 4. (Four runs = four independent A/B experiments on the same truth.)
  9. Freeze your final run. After recording Run 4: select columns A:F → Copy → Paste special ▸ Values only onto the same cells (Excel: Paste Values). Your last run is now frozen data; fill in Scaffold Table 1 from it.

Plausibility note (read before judging your runs): with these true rates, the old page's conversions almost always land between about 12 and 28, and the new page's between about 29 and 51. Your four z-values will bounce around — mostly somewhere between about 0.8 and 4.8 — and most runs, but not all, will reject at α = 0.05. A non-rejecting run is not a broken spreadsheet. It's the point.

Fallback (no spreadsheet available): analyze the two embedded runs instead. Run A: old 18/200, new 42/200 → pooled 0.15, z ≈ 3.36, p ≈ 0.0008. Run B: old 24/200, new 36/200 → pooled 0.15, z ≈ 1.68, p ≈ 0.093. Fill both into Table 2 (leave Runs 3–4 blank), use Run A for Table 1, and answer every question treating Run B as one of "your" runs. State in your submission that you used the fallback.

Part 4 — The Scaffold (complete both tables)

Table 1 captures your final (frozen) run's complete test, top to bottom.

Quantity Your value
Old-page conversions (of 200) ______
New-page conversions (of 200) ______
p̂ (old) ______
p̂ (new) ______
Pooled p̂ ______
SE (from the pooled p̂) ______
z ______
Two-sided p-value ______
Decision at α = 0.05 ______

Table 2 logs all four of your independent runs — four experiments on the same truth.

Run Old conversions New conversions p̂ old p̂ new z p-value Reject at α = 0.05?
1 ______ ______ ______ ______ ______ ______ ______
2 ______ ______ ______ ______ ______ ______ ______
3 ______ ______ ______ ______ ______ ______ ______
4 ______ ______ ______ ______ ______ ______ ______

Part 5 — Analysis Questions

Answer in 1–3 sentences each:

  1. For your frozen run (Table 1): state the hypotheses in symbols and words, and give the conclusion at α = 0.05 in context — which page, which direction, and the licensed sentence for your p-value.
  2. Your SE formula used the pooled proportion. Explain why the two-proportion test pools — what exactly is H₀ claiming that makes one combined rate the honest estimate?
  3. Look across Table 2. The two pages' true rates differ by 10 points — you built that. Did every run reject? If any run failed to (yours or a classmate's — most four-run logs contain one), name the kind of error that run made and what this teaches about reading "no significant difference" in real studies. If all four of yours rejected, explain why a classmate's non-rejecting run still isn't evidence your spreadsheets disagree. (This is the lab's error-source question.)
  4. Suppose a run rejects H₀. In this experiment — unlike the support-teams comparison in the chapter — you may say the new page causes more conversions. What feature of your simulation design earns that causal license, and which Week 1 idea is it? (Hint: how did each simulated shopper end up on their page?)
  5. (Connect forward.) Real checkout traffic doesn't just convert-or-not: an order can end paid, cart-saved, abandoned, or errored — four categories, two pages. This week's test compares one success rate between two groups. What about next week's question: whether a whole table of category counts differs from what's expected? (Next week makes this official: the chi-square test.)

Part 6 — AI-Critique Moment (required — the BYOAI step)

Now bring in your chatbot and be the analyst who checks its work.

  1. Paste your completed Table 2 (all four runs) into your chatbot and ask: "These are four simulated A/B tests of two checkout pages. Which runs show the new page converts better? For any run with p above 0.05, does that run prove the two pages convert equally? Also re-check the z-statistic for Run 1 from its counts."
  2. Check everything it says against your own work. This week's three named AI failure modes:
    - The accept-the-null slip — calling a p > 0.05 run evidence that the pages "convert at the same rate." You know they don't — you built the rates — so you hold a counterexample the AI doesn't: that run is a miss, not a tie.
    - Recomputation drift — re-deriving your z with the unpooled SE recipe (or fumbling the square root), then "correcting" your correct value to a slightly different one. Re-run your own formula chain before accepting any correction.
    - Probability-language errors — "there's a 95% chance the new page is better" or "the p-value is the chance the pages are equal." A p-value is the probability of data this extreme if H₀ were true — nothing else.
  3. Write 2–3 sentences reporting what the AI got right and at least one thing you had to correct, question, or verify. If it happened to get everything right, say how you verified each claim — that's the skill.

The habit all term: the tool drafts, you judge. A chatbot will confidently promote a shrug into a verdict — and you are the only one in the room who knows the truth. Catching it is the point.

Part 7 — What to Submit

One document or text entry containing: your two warm-up answers, your completed Table 1 and Table 2, your Part 5 answers (1–5), and your Part 6 AI-critique paragraph. Due at the end of Week 15 · 50 points (rubric below the key).

Instructor answer key & model values — REMOVE BEFORE PUBLISHING TO STUDENTS

Every student's 400 simulated shoppers are their own, so counts, z's, and p-values are graded on method and interpretation, not on matching any printed value. The theoretical values and plausibility bands below are pre-computed and verified in the week's math-check script; no claim here depends on an unverified number.

Warm-up (fixed — every student should match): expected conversions 200 × 0.10 = 20 (old) and 200 × 0.20 = 40 (new). Example-run pooled proportion: (18 + 42) ÷ 400 = 60 ÷ 400 = 0.15.

Machinery-check values (fixed — step 6 forces every student through them): with counts 18 and 42: p̂'s 0.09 and 0.21; pooled 0.15; SE = √(0.15 × 0.85 × (1/200 + 1/200)) = 0.0357; z = 0.12 ⁄ 0.0357 = 3.36; two-sided p ≈ 0.0008 → reject. Fallback Run B (24 and 36): pooled 0.15 (same total, by design — an easy spot-check); z = 0.06 ⁄ 0.0357 = 1.68; p ≈ 0.093 → fail to reject at α = 0.05 — the embedded miss, so even fallback students experience a Type II error.

Simulation plausibility bands (theory, verified in script): old-page conversions: mean 20, SD √(200 × 0.1 × 0.9) ≈ 4.24 → ~95% of runs land in 12–28. New page: mean 40, SD √(200 × 0.2 × 0.8) ≈ 5.66 → 29–51. Under the true rates the z-statistic centers near 2.80 (true difference 0.10 over the H₀-world SE ≈ 0.0357), so z's mostly land ~0.8–4.8. Power ≈ 0.80 by the normal approximation — roughly 4 honest runs in 5 reject; about 1 in 5 misses. Across a class's four-run logs, most students will see 3–4 rejections; a log with 2 is unusual but not alarming; flag only formula errors (a COUNTIF over the wrong range, an unpooled SE), never luck.

Table 1 grading: all nine rows computed from the student's own frozen counts with the specified formulas; the decision row must be consistent with their p-value. Re-run one line of their chain per submission (pooled p̂ from their two counts is the quickest re-check — it's also the row students most often break by dividing by 200 instead of 400).

Table 2 grading: four runs recorded with reject-calls consistent with each run's p-value; counts inside (or defensibly near) the plausibility bands. Identical numbers across all four runs means they forgot to re-roll (step 8) — partial credit with a note, since every downstream answer still works.

Part 5 expected answers:
1. H₀: p_old = p_new (the pages share one true conversion rate); Hₐ: p_old ≠ p_new. Conclusion from their p: most runs — reject, convincing evidence the new page's rate is higher; a non-rejecting frozen run earns full credit with "the data don't provide convincing evidence of a difference" (bonus insight if they add: "though we know the difference exists — this run is a miss").
2. H₀ claims the two pages share one common rate; if that's true, the best estimate of that single rate uses everyone — combined conversions over combined shoppers. Pooling is the null hypothesis doing its own arithmetic; the SE then measures chance wobble in that one-rate world.
3. Full credit names the non-rejecting run a Type II error (a miss of a real effect) and lands the lesson: "no significant difference" in a published study can mean no difference or a miss — you can't tell from one study, which is why power matters. The all-four-rejected variant must attribute a classmate's miss to sampling variability plus ~80% power, not to a broken sheet.
4. Each shopper was assigned to a page by chance (their row's RANDBETWEEN), which is random assignment — the Week 1 requirement that converts an association into a causal claim. This simulation is a true experiment; the support-teams comparison was observational.
5. Any answer recognizing that a 4-category × 2-page outcome table outgrows a single-proportion comparison and needs a test for whole tables of counts — the chi-square test, next week. (Full credit for the recognition; the name is a bonus.)

AI-critique full credit: names a specific claim checked or corrected — most commonly the accept-the-null slip on a p > 0.05 run (the student holds the counterexample: the true rates differ by construction), an unpooled-SE "correction" of a correct z (for the example run the unpooled recipe gives z ≈ 3.41 vs. the course's pooled 3.36 — verified in the math script), or a p-value described as the chance the pages are equal — OR documents verification of each claim (re-ran the formula chain, re-checked each reject call). Zero credit for "the AI agreed with everything" with no evidence of checking.

Grading rubric — 50 points

The rubric below maps each part to its points; bands are full / partial / none.

Criterion Full Partial None
Warm-up + build — 20 / 40 / 0.15 correct; columns and results block built with the specified formulas; machinery verified against the example run (8) 8 4–6 0–3
Frozen run (Table 1) — all nine rows from their own run; decision consistent with their own p-value (12) 12 6–10 0–5
Four-run log (Table 2) — four distinct runs recorded; reject-calls consistent with each p-value (10) 10 5–8 0–4
Analysis questions — Q1–Q5 hit the expected ideas, especially pooling-as-H₀ (Q2), the Type II/power reading (Q3), and the causal license (Q4) (15) 15 8–12 0–7
AI-critique — specific catch or verification reported in 2–3 sentences (5) 5 3 0–2

Quality gate (self-checked): warm-up arithmetic re-verified (20; 40; 0.15); every theoretical value and band in this key (SE 0.0357; z's 3.36 / 1.68 / unpooled 3.41; p's 0.0008 / 0.093; count bands 12–28 and 29–51; center z 2.80; power ≈ 0.80) re-computed in the week's math script; simulation uses =RANDBETWEEN per the course's simulation-lab convention with expected ranges, not exact values, and no assertion about any student's random results beyond method and plausibility; the embedded fallback contains one rejecting and one non-rejecting run so the power lesson survives without a spreadsheet; no graded-item leakage (the quiz, practice, assignment, and chapter self-checks use different surfaces and numbers); optional background link verified live; rubric sums 8 + 12 + 10 + 15 + 5 = 50. ✓