Introduction to Significance Tests

By the end of this lesson, you will be able to explain the logic of hypothesis testing, correctly interpret a p-value, and distinguish between the two types of decision errors.

The Logic of Hypothesis Testing

Suppose you suspect a coin is biased. You flip it 100 times and get 65 heads.

Could this happen by chance if the coin were fair? Yes — but how likely?

Hypothesis testing formalizes this question: assume the null hypothesis is true, then measure how surprising your data would be under that assumption.

Null and Alternative Hypotheses

Null hypothesis ($H_0$): the default claim to be tested; usually "no effect," "no difference," or "equals a specific value." Written with an equals sign.

Alternative hypothesis ($H_a$): what you're trying to find evidence for. Three forms:

• Two-sided: $H_a: p \neq p_0$ (don't know which direction)
• Right-tailed: $H_a: p > p_0$ (testing for an increase)
• Left-tailed: $H_a: p < p_0$ (testing for a decrease)

The alternative hypothesis should be stated before collecting data, based on the research question.

The Test Statistic

The test statistic measures how far the observed data is from what we'd expect if $H_0$ were true, in standard error units.

For a proportion: $z = \dfrac{\hat{p} - p_0}{\sqrt{p_0(1-p_0)/n}}$

A large |z| means the data is far from what $H_0$ predicts.

The p-Value

$$p\text{-value} = P(\text{observing data at least as extreme as what we got} \mid H_0 \text{ is true})$$

Small p-value: the data is very unlikely under $H_0$ → evidence against $H_0$.

Large p-value: the data is consistent with $H_0$ → no evidence against $H_0$.

Significance level α: the threshold for "too unlikely." Most commonly α = 0.05.

Decision rule:

• p < α → Reject $H_0$ ("statistically significant")
• p ≥ α → Fail to reject $H_0$ (no significant evidence)

What the p-Value Is NOT

The p-value is NOT the probability that $H_0$ is true. It's the probability of getting this data (or more extreme) assuming $H_0$ is true.