What does a p-value represent?

• A. The probability that the null hypothesis is true.
• B. The probability of obtaining a result exactly as observed under the null hypothesis.
• C. The probability of making a Type I error.
• D. The probability, under the null hypothesis, of obtaining a result at least as extreme as the observed one.

Answer: (D)

A p-value measures how compatible your data are with the assumption that the null hypothesis is true. It is the probability, under the null, of obtaining a result at least as extreme as the one actually observed (in the direction of the alternative you’re testing). In other words, if you could repeat the experiment many times with the null true, the p-value would be the fraction of studies that produce results as or more extreme than what you saw. This isn’t the probability that the null is true, nor the probability of observing exactly the same result, and it isn’t the Type I error rate itself (that rate is set by your alpha level).