Introduction to Hypothesis Testing

By the end of this lesson, you will be able to formulate null and alternative hypotheses, understand the logic of hypothesis testing (assuming the null is true, then assessing evidence against it), interpret $p$-values conceptually, and distinguish Type I and Type II errors.

The Big Idea

Hypothesis testing is the formal framework for making data-driven decisions about populations. The core logic: assume the null hypothesis is true, collect data, and ask "how likely is this data if the null were true?" If the data is very unlikely under the null, we reject it. This process underlies every medical trial, A/B test, and scientific study.

The Hypotheses

Null hypothesis ($H_0$): the default assumption — "no effect," "no difference," "the parameter equals a specific value."

Alternative hypothesis ($H_a$ or $H_1$): what we're trying to establish evidence for — the claim we want to test.

Examples:

Scenario	$H_0$	$H_a$
Is a new drug better?	$\mu_{\text{drug}} = \mu_{\text{placebo}}$	$\mu_{\text{drug}} > \mu_{\text{placebo}}$
Is the coin fair?	$p = 0.5$	$p \neq 0.5$
Are test scores improved?	$\mu_{\text{after}} = \mu_{\text{before}}$	$\mu_{\text{after}} > \mu_{\text{before}}$

The $p$-Value

$p$-value: the probability of observing data as extreme as (or more extreme than) what was actually observed, assuming $H_0$ is true.

• Small $p$-value → the data is very unlikely under $H_0$ → evidence against $H_0$
• Large $p$-value → the data is not surprising under $H_0$ → no evidence against $H_0$

Significance level $\alpha$: the threshold for "too unlikely." Most common: $\alpha = 0.05$.

Decision rule:

• If $p < \alpha$: Reject $H_0$ — "statistically significant"
• If $p \geq \alpha$: Fail to reject $H_0$ — not sufficient evidence

The Logic of Hypothesis Testing