Bias in Statistical Claims
Analyse claims, inferences and conclusions of statistical reports in the media, including ethical considerations and identification of potential sources of bias.
A number in a headline has been chosen by someone. This is the one topic where the maths is easy and the thinking is hard — you are not calculating, you are deciding whether to believe.
Builds onYr 10 · Boxplots and Comparing Distributions
The five questions to ask
Who was asked?
A sample taken at a gym will overstate how much people exercise. Ask who could not have been in the sample.
What exactly was measured?
Average can mean mean, median or mode. A single outlier can drag a mean a long way from what is typical.
Compared with what?
Sales doubled means nothing without the starting figure. Doubling from 2 to 4 is not news.
Who paid for it?
Funding does not make a study wrong, but it is a legitimate thing to weigh — and it is an ethical consideration you are expected to raise.
Correlation is not cause
Two things rising together may share a third cause, or be coincidence. This is the most misused claim in the media.
Worked example
e.g.
- Find the actual claim, stripped of the wording.
- Ask who was asked, and how many.
- Ask what recommend meant. Recommend over what alternative?
- Look for the comparison. Ninety per cent compared with which other product?
- Ask who funded and published the study.
- State what would need to be true for the claim to be fair.
- Write your conclusion as a judgement with a reason, not a dismissal.
Practice
There is no calculation here. Say what is missing, and why it matters.
ReasoningExplain why. Say it in your own words.
1
A survey conducted online finds that 94% of people use the internet daily. What is wrong?
AnswerThe sample can only contain people who are online, so the result is guaranteed before anyone is asked. This is selection bias — the group that could have answered is not the group being described.
2
A report says the average salary at a company is R95 000. Twenty staff earn R40 000 and the owner earns R1 200 000. Is the average misleading?
AnswerYes. The mean is dragged up by one extreme value. The median of R40 000 describes a typical employee far better. Whenever an average is quoted, ask which average it is.
3
Crime rose 200% in our suburb. What must you know before reacting?
AnswerThe starting figure. A rise from 1 incident to 3 is a 200% rise and is close to meaningless. Percentage change without the base number is the commonest way to make a small change sound large.
4
Ice cream sales and drowning deaths rise together each summer. Does ice cream cause drowning?
AnswerNo. Both are caused by a third factor — hot weather. Correlation shows two things moving together; it never shows one causes the other.
AppliedThe same maths, inside a real question.
1
A soft drink company funds a study concluding that sugar has no effect on concentration in children. How should you treat it?
AnswerNot as automatically false, but as needing more care. Ask who was studied, over how long, how concentration was measured, and whether the study was published where others could check it. Funding is a reason to scrutinise, not a reason to dismiss — and saying so is the ethical judgement being assessed.
2
A school reports that students who do homework score higher, and concludes homework causes better marks. Give one alternative explanation.
AnswerStudents who already find the work easier are more likely to complete homework, so the marks may cause the homework rather than the reverse. A fair test would assign homework at random rather than observing who chooses to do it.
3
A graph's vertical axis starts at 88 rather than 0, making a rise from 89 to 92 look enormous. Is this dishonest?
AnswerIt is a legitimate choice for showing small variation, but misleading when presented as dramatic growth. The judgement depends on whether the axis is clearly labelled and whether the accompanying claim matches the true size of the change.