How Hypothesis Testing Works: A Practical Guide for Research

The short answer

Start with a research question

Hypothesis testing should follow a clear research question, not a search through many analyses until one produces a preferred result.

For example, a randomized study may ask whether a treatment changes mean systolic blood pressure after eight weeks compared with control:

• Null hypothesis (H₀): the population mean blood-pressure change is the same in the treatment and control groups.
• Alternative hypothesis (H₁): the population mean blood-pressure change differs between the treatment and control groups.

The hypotheses concern population quantities, not whether two observed sample means happen to be numerically identical. The statistical question is whether the observed difference is larger than would be reasonably expected from sampling variation under the specified null model.

The pieces of a hypothesis test

1. State the question and model. Define the outcome, comparison, target population, null hypothesis, and alternative hypothesis.
2. Choose a statistic. Calculate a summary that captures the relevant departure from the null model, such as a difference in means, correlation, odds ratio, chi-square statistic, or regression coefficient.
3. Describe expected variation under the null. The method uses a probability model, exact calculation, resampling procedure, or another justified approach.
4. Calculate the p-value. Compare the observed statistic with its null distribution.
5. Report and interpret the result. Present the estimate, confidence interval where appropriate, p-value, assumptions, and limitations—not only a threshold-based label.

The choice of test must match the outcome type and study design. A t-test, chi-square test, rank-based test, regression model, and survival analysis do not ask exactly the same question or make exactly the same assumptions.

What a p-value means

The p-value is the probability, assuming the null hypothesis and the analysis assumptions are correct, of obtaining data at least as incompatible with that null model as the observed data.

Suppose a study reports p = 0.03 for a pre-specified treatment comparison. Under the null model of no population mean difference, the observed result—or a more extreme result in the direction considered by the test—would occur about 3% of the time across repeated comparable studies.

It does not mean there is a 3% probability that the null hypothesis is true, a 97% probability that the treatment works, that the result has a 3% chance of being “due to chance,” or that the finding is clinically important.

Statistical significance is a decision convention

Many studies specify a significance level, often written α (alpha), before analysis. A common value is 0.05. If the p-value is below the chosen threshold, researchers may describe the result as statistically significant or reject the null hypothesis at that level.

The threshold is a convention for a long-run error-control strategy, not a sharp border between “real” and “not real.” A result of p = 0.049 is not meaningfully different in scientific substance from p = 0.051. Both should be interpreted using the estimated effect, confidence interval, study quality, and the full body of evidence.

Pre-specifying the analysis and threshold matters. Repeatedly trying different outcomes, subgroups, transformations, or stopping points can make small p-values more likely even when the null hypothesis is true. When several tests address a family of questions, use an appropriate plan for multiplicity and report the number and purpose of the tests.

Effect size and confidence interval: the other half of the result

The p-value answers a narrow question about compatibility with a null model. It does not show the size, direction, practical relevance, or precision of an effect.

Report an estimate that matches the question: a mean difference for two groups; a risk difference, risk ratio, or odds ratio for a binary outcome; a correlation coefficient for an association; or a regression coefficient for an adjusted comparison.

A confidence interval gives a range of effect sizes reasonably compatible with the data and model at the stated confidence level. For a two-sided test and matching two-sided interval, a 95% confidence interval that excludes the null value corresponds to a p-value below 0.05. The interval usually communicates more because it shows effect scale and direction.

One-sided and two-sided hypotheses

A two-sided alternative asks whether an effect could differ in either direction—for example, whether a treatment may increase or decrease an outcome. This is usually the appropriate default when both directions would be scientifically relevant.

A one-sided alternative asks whether an effect lies in only one specified direction. Choose it before examining data and only when the opposite direction would not lead to the same scientific claim or decision. Choosing a one-sided test after seeing data is not a valid way to make a p-value smaller.

Type I and Type II errors

The selected alpha level controls the long-run Type I error rate only when the analysis plan and assumptions are appropriate. Power is the probability of detecting a specified effect under a specified model and sample size; it is one way to plan a study before collecting data. A non-significant result from a low-power study is often too imprecise to distinguish no effect from a meaningful effect.

Common mistakes

“A significant result proves my hypothesis.”

No. A small p-value is evidence that the data are difficult to reconcile with the stated null model. Bias, confounding, measurement error, multiple testing, model misspecification, or chance can still affect the result. The analysis does not by itself establish causation or a specific scientific mechanism.

“A non-significant result means there is no effect.”

No. The result may be compatible with no effect, but it may also be compatible with effects that the study cannot estimate precisely. Examine the estimate and confidence interval, not only whether the p-value crossed a threshold.

“A smaller p-value means a larger or more important effect.”

No. With a large sample, a very small effect can produce a very small p-value. With a small sample, a potentially important effect can have a larger p-value because the estimate is imprecise.

“I can choose the test after looking for the best p-value.”

No. Select the analysis from the study design and research question before inspecting results. If exploratory analyses are useful, label them clearly and interpret them cautiously.

“The null hypothesis is either accepted or rejected.”

It is clearer to say that the analysis found evidence against the null model, or that it did not find enough evidence against it under the selected threshold. Failing to reject is not proving the null model true.

A practical reporting checklist

1. The study question, outcome, groups or variables, and what one observation represents.
2. The pre-specified null and alternative hypotheses when they clarify the analysis.
3. The test or model, including whether it was one-sided or two-sided and relevant assumptions.
4. Sample sizes and descriptive summaries for the data being compared.
5. The estimated effect and a confidence interval when appropriate.
6. The test statistic, degrees of freedom when relevant, and p-value.
7. How missing data, outliers, multiple tests, clustering, matching, or repeated measurements were handled.
8. A conclusion that reflects precision, limitations, and scientific context rather than only a significance label.

For example: “The treatment group had a mean reduction 4.2 mmHg greater than control (95% CI 1.1 to 7.3 mmHg; Welch two-sample t-test, p = 0.009). The estimate suggests a reduction, although clinical importance should be evaluated against the pre-specified meaningful difference.”

When to seek additional statistical support

Seek statistical support before interpreting results when the study has a small or complex sample, several outcomes or subgroup analyses, clustering, matching, repeated measurements, substantial missing data, adaptive stopping rules, covariate adjustment, or a causal or clinical decision at stake. The analysis plan should follow the question and design—not the p-value that is easiest to obtain.

Related resources

For a practical route from a research question to a common analysis, see Which Statistical Test Should I Use?. To understand a common distributional diagnostic before a mean-based analysis, see What Is a Normal Distribution? A Practical Guide for Research Data. For practical interpretation of estimates, p-values, and confidence intervals together, see How to Interpret P-Values and Confidence Intervals Together. For planning around errors, power, and sample size, see Type I and Type II Errors, Power, and Sample Size.