p-value vs Confidence Interval vs Effect Size: What They Actually Mean

“Statistically significant” is one of the most quoted and least understood phrases in research reporting. The American Statistical Association issued a formal 2016 statement specifically because these three numbers — p-value, confidence interval, and effect size — get misinterpreted so consistently, even by people running the analysis. Here’s what each one actually says.

What a p-value Actually Is (and Isn’t)

The correct definition: a p-value is the probability of obtaining a result at least as extreme as the one observed, assuming the null hypothesis is true.

The p-value is the probability of landing in this shaded region, if the null hypothesis were true

The ASA’s 2016 statement, built on six formal principles, is explicit about what a p-value is not: it does not measure the probability that the studied hypothesis is true; it does not measure the probability that the data arose by pure chance; it does not measure the size or importance of an effect; and it should never be the sole basis for a conclusion. All four are extremely common misreadings. Calculate p-values directly from a Z, t, F, or chi-square statistic with the P-Value Calculator, or run the underlying test yourself with the t-Test Calculator and Chi-Square Test Calculator.

Confidence Interval: What “95%” Actually Refers To

A 95% confidence interval does not mean “there’s a 95% probability the true value falls inside this specific interval.” Once an interval is calculated from your actual sample, the true population parameter either is or isn’t inside it — there’s no probability left to assign to that one interval. The 95% instead describes the long-run behavior of the method: if you repeated the sampling process many times and built a new interval each time, approximately 95% of those intervals would contain the true parameter.

About 95% of intervals from repeated sampling cover the true value — but any one interval either does or doesn't

Calculate a confidence interval for a mean or proportion, with the exact margin of error and critical value, using the Confidence Interval Calculator. The width of that interval is directly controlled by your sample size — plan the sample size needed for a target margin of error with the Sample Size Calculator.

Effect Size: The Number Significance Testing Leaves Out

Here’s the gap a p-value can never fill: it tells you whether an effect is likely real, but says nothing about whether that effect actually matters. With a large enough sample, even a genuinely trivial difference can produce a tiny, “statistically significant” p-value. Effect size measures the actual magnitude of a difference, independent of sample size.

Cohen’s d, the most common standardized effect size measure, uses benchmarks proposed by Jacob Cohen in 1988: 0.2 (small), 0.5 (medium), and 0.8 (large). In practical terms, a small effect (d = 0.2) means the average person in the higher-scoring group outperforms only about 58% of the other group — a difference easy to miss just by looking. A large effect (d = 0.8) means roughly 79% — a clearly visible separation. Cohen himself explicitly cautioned that these are arbitrary, last-resort benchmarks meant for when no field-specific standard exists, not universal rules. Calculate Cohen’s d, Pearson r, and eta-squared — with magnitude labels and cross-conversions — using the Effect Size Calculator.

Putting It Together: Why You Need All Three

A study can be statistically significant (small p-value) with an effect size too small to matter in practice, especially with a very large sample. It can also fail to reach significance despite a real, meaningful effect simply because the sample was too small to detect it reliably — which is exactly why planning sample size in advance, with the Sample Size Calculator, matters as much as the test itself. None of these three numbers tells the whole story alone.

Frequently Asked Questions

What does p < 0.05 actually mean?
It means that, assuming the null hypothesis is true, a result at least this extreme would occur less than 5% of the time by chance — it does not mean there’s a 95% probability the null hypothesis is false.

Does statistical significance mean an effect is important?
No — statistical significance only reflects whether an effect is likely real given the data and sample size. A tiny, practically meaningless effect can still be “statistically significant” with a large enough sample, which is exactly why effect size is reported separately.

What’s wrong with saying “there’s a 95% chance the true value is in this interval”?
The 95% describes the long-run success rate of the method across repeated sampling, not a probability attached to any one already-calculated interval — once computed, that specific interval either contains the true value or it doesn’t.

What are Cohen’s small, medium, and large effect size benchmarks?
0.2, 0.5, and 0.8 respectively for Cohen’s d, proposed by Jacob Cohen in 1988 as arbitrary last-resort guidelines, not universal cutoffs — field-specific context should take priority when available.

Why do I need effect size if I already have a p-value?
Because a p-value only speaks to whether an effect is likely real, not how large or practically meaningful it is — effect size fills exactly that gap, independent of sample size.

Start with the P-Value Calculator, t-Test Calculator, and Chi-Square Test Calculator for significance testing, use the Confidence Interval Calculator and Sample Size Calculator for estimation and planning, and always pair your result with the Effect Size Calculator to judge practical importance.

External Resources

The ASA Statement on p-Values: Context, Process, and Purpose — the official American Statistical Association 2016 statement
P-value — Wikipedia — background on the formal definition and common misinterpretations