Type I and Type II Errors, Power, and Sample Size

The short answer

Start with the decision you need to support

Before deciding on a sample size, state the population and outcome, primary comparison, smallest effect that would matter, planned analysis, and the consequences of an incorrect conclusion.

For a study comparing an intervention with usual care, the question is not only “How many people can we recruit?” It is also “What difference in the outcome would be meaningful enough to change a scientific, clinical, or operational decision?” This meaningful difference is often called the target effect or smallest effect size of interest.

The two errors in a hypothesis-test decision

In a conventional hypothesis test, a decision rule compares a p-value with a pre-specified significance level, often α = 0.05. Across many comparable studies, that rule can produce two kinds of error.

With one pre-specified and correctly analyzed comparison, a two-sided α of 0.05 is designed to reject a true null model about 5% of the time over repeated comparable studies. Multiple unplanned tests, selective analyses, invalid interim decisions, or a poorly fitting model can increase that risk.

A Type II error is conditional on the effect considered, outcome variability, sample size, design, analysis, and alpha level. A study may have high power for a large effect but low power for a smaller effect that still matters.

Power is not a property of a study in general

Power, often written as 1 − β, is the chance that the planned test will reach its pre-specified decision threshold if a particular alternative scenario is true. “This study has 80% power” is incomplete without stating 80% power to detect what, using which analysis, at what alpha, and under what assumptions?

For a two-group mean comparison, power may depend on the target difference, expected standard deviation, sample size in each group, allocation ratio, alpha level, one- or two-sided analysis, planned covariate adjustment, and expected missing outcomes. For binary or survival outcomes, event rates and follow-up matter; clustered, paired, and repeated-measures designs also need their correlations reflected in the calculation.

How the choices move together

These are not inputs to tune until a calculation fits a budget. They make the scientific and logistical implications of a design explicit.

Why more participants is not the only answer

A larger study can improve precision, but it can also make very small effects statistically detectable. Statistical significance alone does not establish practical importance. Compare the estimate and confidence interval with a pre-specified meaningful-effect range.

Conversely, a small study may estimate a potentially useful effect but have a wide interval that includes no effect and important benefit or harm. A non-significant p-value in that setting is often best described as inconclusive, not as evidence that the intervention has no effect.

A practical sample-size workflow

1. Define one primary question. State the population, outcome, comparison, time point, and effect measure.
2. Set a meaningful target effect. Base it on clinical relevance, biological knowledge, stakeholder priorities, prior evidence, or a decision threshold.
3. Choose the planned analysis. It must match independent groups, paired measurements, repeated observations, clustering, time-to-event data, or another structure.
4. Specify assumptions. Include variability, event rate, correlation, attrition, allocation, follow-up duration, and alpha as applicable.
5. Choose a power target. Explain why it fits the stakes of the study.
6. Calculate analyzable participants or events. Then increase recruitment for expected missing data or dropout.
7. Run sensitivity analyses. Show how requirements change under plausible assumptions.
8. Document the plan before results are known.

An illustrative planning statement

“The primary comparison is the mean difference in eight-week systolic blood-pressure change between intervention and usual-care groups. We will use a two-sided alpha of 0.05 and target 80% power to detect a 5 mmHg difference, selected as the smallest clinically meaningful difference. Assuming a standard deviation of 10 mmHg and equal allocation, a two-sample t-test calculation requires 64 analyzable participants per group. We will recruit 72 per group (144 total) to allow for 10% missing primary-outcome data.”

The numbers are illustrative. A real calculation should use the planned method and defensible study-specific assumptions. If the analysis adjusts for baseline values, uses repeated follow-up measurements, clusters participants, or has unequal allocation, the calculation should reflect that structure.

Common mistakes

“A result above 0.05 shows there is no effect.”

No. The estimate and confidence interval may still be compatible with effects that matter. If the study had low power for those effects, it cannot rule them out.

“Power can be calculated after the study to explain a non-significant result.”

Usually not usefully. Examine the estimate, interval, planned target effect, and what effects the study could reasonably exclude instead.

“Eighty percent power is automatically acceptable.”

Not necessarily. It is a convention; the appropriate target depends on the study goal, feasibility, ethics, costs, and harm of missing an effect that would matter.

“Increasing sample size fixes a biased study.”

No. More observations do not correct confounding, selection bias, measurement error, unrepresentative sampling, or a mismatch between the question and analysis.

What to report

For a planned study, report the primary outcome and analysis, target effect, alpha, desired power, assumed variability or event rate, allocation, anticipated missingness, calculation method, and final recruitment target. Include sensitivity analyses when assumptions are uncertain.

For completed research, report the observed estimate, confidence interval, sample size or event count actually analyzed, and p-value when relevant. Do not use a non-significant result alone to claim equivalence, no meaningful effect, or adequate power; those conclusions need a design and analysis aimed at that question.

When to seek statistical support

Seek statistical support early when the study includes several primary outcomes, interim looks at the data, adaptive decisions, rare events, clustering, matching, repeated measurements, complex missingness, noninferiority or equivalence questions, or high-stakes clinical, regulatory, or policy decisions.

Related resources

For the broader framework of null hypotheses, p-values, and hypothesis-test decisions, see How Hypothesis Testing Works: A Practical Guide for Research. For interpreting completed-study estimates and uncertainty, see How to Interpret P-Values and Confidence Intervals Together.