Learn / Math / Problem-Solving and Data Analysis

Evaluating Statistical Claims: Observational Studies and Experiments

I worked these out from the practice questions. Each code says where to start reading, where to stop, what you can skip, and why it is safe.

Start:: The sentences that tell how people were chosen and how groups were made.
Stop:: After you write down Sample: random or not, Assignment: random or not.
Skip:: The topic details (the medicine, the fertilizer) and the numbers.
Why:: Every one of the 8 questions is solved by these two yes/no marks. The topic is only decoration.

Used in questions 1, 2, 3, 4, 5, 6, 7, 8

Start:: The word 'assigned' in the passage.
Stop:: When you see if it says randomly.
Skip:: Options that say 'only associated' when assignment was random (questions 4 and 6).
Why:: If groups were made by chance, a cause is allowed. If people chose their group (question 8), a cause is not allowed.

Used in questions 2, 4, 6, 8

Start:: The word 'selected' or 'volunteers'.
Stop:: When you know the exact population.
Skip:: Options that name a bigger group (all adults, the whole country).
Why:: Volunteers (questions 3 and 4) limit the result to people like them. A random sample (questions 1, 5, 6) allows the whole population it came from, and no larger.

Used in questions 1, 3, 4, 5, 6

Start:: The options.
Stop:: After you remove those that say 'all', 'every' or 'cause' when the study does not allow it.
Skip:: Re-reading the passage again and again.
Why:: Wrong options are made by claiming too much (questions 3, 4, 8) or too little (questions 4, 6). The right one is the exact match.

Used in questions 3, 4, 6, 8

Start:: The question line 'which design would best...'.
Stop:: When you find the option with random selection and random assignment.
Skip:: Options with volunteers or self-chosen groups.
Why:: To claim a cause for a whole population you need both checks, as in question 5.

Used in question 5

Start:: Words such as 'only 220 answered' or 'chose to respond'.
Stop:: When you spot a gap between who was picked and who gave data.
Skip:: Sample size arguments if the size is already large.
Why:: A random pick can be spoiled by nonresponse (question 7). Options that deny the random pick are false there.

Used in question 7

Start:: The verbs in the options.
Stop:: At 'is associated with' or 'is linked to' versus 'causes'.
Skip:: Your own opinion about whether the cause makes sense in real life.
Why:: The SAT tests design, not common sense. A sensible cause with no random assignment is still not proven (question 3 and 8).

Used in questions 3, 8

Start:: The sentence that says if the difference could be due to chance.
Stop:: When you know whether the difference is unlikely or likely to be chance.
Skip:: Options that say the treatment 'proves' or 'causes' when the difference is small.
Why:: A perfect design can still end with no convincing evidence. 'No convincing evidence of an effect' is the safe answer, and 'proves no effect' goes too far (question 13).

Used in question 13

Start:: The words 'called', 'clicked', 'chose to respond' or 'volunteered'.
Stop:: When you see that the people picked themselves.
Skip:: The size of the sample, such as 1,800 callers.
Why:: How people were chosen matters more than how many there were. Self-selected groups cannot speak for everybody (question 14, also questions 3 and 7).

Used in question 14

Start:: The column that says how people were chosen, then the column that says how groups were formed.
Stop:: When you find the row that says random in both columns and covers the whole population.
Skip:: The results column, because every row has the same result.
Why:: Each wrong row fails one check: groups chosen by the students, volunteers only, or only one grade (question 16). A confounder, like income, explains a link when nobody was assigned (question 11), and a placebo group is only a comparison (question 12).

Used in questions 11, 12, 16