Title: Week 15 Data Lab — Testing Proportions & Two-Sample Inference
Course: Introduction to Statistics (18-Week Generic Edition)

Course: Introduction to Statistics (18-week generic edition) Objective: Objective 7 — the two-proportion z-test, run end to end; power and Type II error experienced firsthand · SLO A (reason quantitatively from data) Worth 50 points · Data labs group = 15% of the grade · Data Lab 15 · Due: end of Week 15 Format: a hands-on data lab — this week you play both roles in a tech company's most common experiment: you rig the truth (two checkout pages with genuinely different conversion rates), simulate 400 shoppers, and then see whether the two-proportion z-test catches the difference you know is there; then you catch the AI's mistakes when it interprets your runs.

This is the course's signature weekly component. Every instructional week has a Data Lab: real, free, linked public data — nothing to buy, download, or sign up for — worked in your spreadsheet, finished with an AI-critique step. Plan on 60–90 minutes. (This week the "dataset" is one your own spreadsheet generates — 400 simulated shoppers who exist nowhere else in the world.)

Part 1 — The Big Picture

Every A/B test in industry has a dirty secret: the analyst never knows the truth. They see two conversion rates wobble in, run the test, and hope the verdict matches a reality they can't check. Today you get the one thing they never have — omniscience. You will build the truth: an old checkout page that converts exactly 10% of shoppers in the long run, and a redesigned page that converts exactly 20%. Then you'll simulate 200 shoppers on each, run the week's two-proportion z-test, and find out whether the test catches a difference you know is real.

Here is the spoiler worth the whole lab: it won't always. Run the experiment several times and most runs will reject H₀ loudly — but about one honest run in five will shrug and fail to find anything, even though the difference is baked in. That miss is Week 13's Type II error made personal, and the catch-rate has a name — power. After today, "the study found no significant difference" will never again sound to you like "there is no difference."

Optional 6-minute background read before you start: An Easy Introduction to Statistical Significance (Scribbr) 🔗 https://www.scribbr.com/statistics/statistical-significance/

Part 2 — The Task

The guiding question: The redesigned page really is better — you built it that way. Will YOUR 400 simulated shoppers reveal it, how loudly, and how often does an honest experiment miss a real effect?

The tools (nothing to buy or download): a spreadsheet — Google Sheets (free) or Excel. Its =RANDBETWEEN(1,10) function is a 10-sided die: a roll of 1 is a shopper who converts on the old page (a 10% chance). =RANDBETWEEN(1,5) is a 5-sided die: a 1 converts on the new page (a 20% chance). No memory, no mercy — exactly like real traffic.

Part 3 — Materials & Procedure

You need: a spreadsheet and about 60–90 minutes. Nothing else.

Warm-up (5 min, no spreadsheet needed). Two small computations by hand or calculator — write both down; they're your benchmarks:

1. Expected conversions in 200 shoppers per page: old page 200 × 0.10 = __ ; new page 200 × 0.20 = ____ .
2. In one example run (you'll check your machinery against it later), the old page converted 18 of 200 and the new page 42 of 200. The pooled proportion is (18 + 42) ÷ 400 = ______ .

Now build the simulation:

1. Set up headers. In a blank sheet, type into row 1: A1 = shopper, B1 = old page roll (1-10), C1 = new page roll (1-5).
2. Number the shoppers. A2 = 1. A3 = =A2+1. Copy A3 and paste into A4:A201. Column A should count 1 to 200 (check: A201 shows 200).
3. Send traffic to the old page. B2 = =RANDBETWEEN(1,10). Copy B2, paste into B3:B201. Every 1 in column B is a conversion — a 10% chance per shopper.
4. Send traffic to the new page. C2 = =RANDBETWEEN(1,5). Copy into C3:C201. Every 1 in column C is a conversion — a 20% chance per shopper.
5. Build the test machinery in a results block (labels in column E, formulas in F): - F2 (old conversions) = =COUNTIF(B2:B201,1) - F3 (new conversions) = =COUNTIF(C2:C201,1) - F4 (p̂ old) = =F2/200 · F5 (p̂ new) = =F3/200 - F6 (pooled p̂) = =(F2+F3)/400 - F7 (SE) = =SQRT(F6*(1-F6)*(1/200+1/200)) - F8 (z) = =(F5-F4)/F7 - F9 (two-sided p-value) = =2*(1-NORM.S.DIST(ABS(F8),TRUE))
6. Verify your machinery against the example run. Temporarily type 18 over F2 and 42 over F3 (plain numbers). F6 should show 0.15, F7 ≈ 0.0357, F8 ≈ 3.36, F9 ≈ 0.0008. If yours disagree, fix the formulas now. Then restore F2 and F3 to their =COUNTIF(...) formulas.
7. Record Run 1. Copy the current values of F2, F3, F4, F5, F8, F9 into the first row of Scaffold Table 2, and mark whether p < 0.05 ("reject? yes/no").
8. Re-roll for Runs 2–4. Type any letter into an empty cell (say J1) and press Enter — every RANDBETWEEN re-rolls, and your results block updates. Record the new F-values as Run 2. Repeat twice more for Runs 3 and 4. (Four runs = four independent A/B experiments on the same truth.)
9. Freeze your final run. After recording Run 4: select columns A:F → Copy → Paste special ▸ Values only onto the same cells (Excel: Paste Values). Your last run is now frozen data; fill in Scaffold Table 1 from it.

Plausibility note (read before judging your runs): with these true rates, the old page's conversions almost always land between about 12 and 28, and the new page's between about 29 and 51. Your four z-values will bounce around — mostly somewhere between about 0.8 and 4.8 — and most runs, but not all, will reject at α = 0.05. A non-rejecting run is not a broken spreadsheet. It's the point.

Fallback (no spreadsheet available): analyze the two embedded runs instead. Run A: old 18/200, new 42/200 → pooled 0.15, z ≈ 3.36, p ≈ 0.0008. Run B: old 24/200, new 36/200 → pooled 0.15, z ≈ 1.68, p ≈ 0.093. Fill both into Table 2 (leave Runs 3–4 blank), use Run A for Table 1, and answer every question treating Run B as one of "your" runs. State in your submission that you used the fallback.

Part 4 — The Scaffold (complete both tables)

Table 1 captures your final (frozen) run's complete test, top to bottom.

Quantity | Your value
Old-page conversions (of 200) | ______
New-page conversions (of 200) | ______
p̂ (old) | ______
p̂ (new) | ______
Pooled p̂ | ______
SE (from the pooled p̂) | ______
z | ______
Two-sided p-value | ______
Decision at α = 0.05 | ______

Table 2 logs all four of your independent runs — four experiments on the same truth.

Run | Old conversions | New conversions | p̂ old | p̂ new | z | p-value | Reject at α = 0.05?
1 | ______ | ______ | ______ | ______ | ______ | ______ | ______
2 | ______ | ______ | ______ | ______ | ______ | ______ | ______
3 | ______ | ______ | ______ | ______ | ______ | ______ | ______
4 | ______ | ______ | ______ | ______ | ______ | ______ | ______

Part 5 — Analysis Questions

Answer in 1–3 sentences each:

1. For your frozen run (Table 1): state the hypotheses in symbols and words, and give the conclusion at α = 0.05 in context — which page, which direction, and the licensed sentence for your p-value.
2. Your SE formula used the pooled proportion. Explain why the two-proportion test pools — what exactly is H₀ claiming that makes one combined rate the honest estimate?
3. Look across Table 2. The two pages' true rates differ by 10 points — you built that. Did every run reject? If any run failed to (yours or a classmate's — most four-run logs contain one), name the kind of error that run made and what this teaches about reading "no significant difference" in real studies. If all four of yours rejected, explain why a classmate's non-rejecting run still isn't evidence your spreadsheets disagree. (This is the lab's error-source question.)
4. Suppose a run rejects H₀. In this experiment — unlike the support-teams comparison in the chapter — you may say the new page causes more conversions. What feature of your simulation design earns that causal license, and which Week 1 idea is it? (Hint: how did each simulated shopper end up on their page?)
5. (Connect forward.) Real checkout traffic doesn't just convert-or-not: an order can end paid, cart-saved, abandoned, or errored — four categories, two pages. This week's test compares one success rate between two groups. What about next week's question: whether a whole table of category counts differs from what's expected? (Next week makes this official: the chi-square test.)

Part 6 — AI-Critique Moment (required — the BYOAI step)

Now bring in your chatbot and be the analyst who checks its work.

1. Paste your completed Table 2 (all four runs) into your chatbot and ask: "These are four simulated A/B tests of two checkout pages. Which runs show the new page converts better? For any run with p above 0.05, does that run prove the two pages convert equally? Also re-check the z-statistic for Run 1 from its counts."
2. Check everything it says against your own work. This week's three named AI failure modes: - The accept-the-null slip — calling a p > 0.05 run evidence that the pages "convert at the same rate." You know they don't — you built the rates — so you hold a counterexample the AI doesn't: that run is a miss, not a tie. - Recomputation drift — re-deriving your z with the unpooled SE recipe (or fumbling the square root), then "correcting" your correct value to a slightly different one. Re-run your own formula chain before accepting any correction. - Probability-language errors — "there's a 95% chance the new page is better" or "the p-value is the chance the pages are equal." A p-value is the probability of data this extreme if H₀ were true — nothing else.
3. Write 2–3 sentences reporting what the AI got right and at least one thing you had to correct, question, or verify. If it happened to get everything right, say how you verified each claim — that's the skill.

The habit all term: the tool drafts, you judge. A chatbot will confidently promote a shrug into a verdict — and you are the only one in the room who knows the truth. Catching it is the point.

Part 7 — What to Submit

One document or text entry containing: your two warm-up answers, your completed Table 1 and Table 2, your Part 5 answers (1–5), and your Part 6 AI-critique paragraph. Due at the end of Week 15 · 50 points (rubric below the key).

Grading rubric — 50 points

The rubric below maps each part to its points; bands are full / partial / none.

Criterion | Full | Partial | None
Warm-up + build — 20 / 40 / 0.15 correct; columns and results block built with the specified formulas; machinery verified against the example run (8) | 8 | 4–6 | 0–3
Frozen run (Table 1) — all nine rows from their own run; decision consistent with their own p-value (12) | 12 | 6–10 | 0–5
Four-run log (Table 2) — four distinct runs recorded; reject-calls consistent with each p-value (10) | 10 | 5–8 | 0–4
Analysis questions — Q1–Q5 hit the expected ideas, especially pooling-as-H₀ (Q2), the Type II/power reading (Q3), and the causal license (Q4) (15) | 15 | 8–12 | 0–7
AI-critique — specific catch or verification reported in 2–3 sentences (5) | 5 | 3 | 0–2