Statistical significance versus practical importance

START HERE

The essential idea

Interpret a p value in relation to the specified model. Compare statistical evidence with a prespecified practical threshold. Identify selective testing and weak equivalence claims in a study report.

BUILD THE REASONING · 01

A small p value answers a limited question

A p value describes how incompatible the observed data, or a more extreme result according to the chosen statistic, are with a specified null model under its assumptions. It is not the probability that the null hypothesis is true, the probability the intervention works, or the proportion of participants who benefited. To interpret it, you need the null value, design, test, assumptions, and analysis choices. A bare decimal cannot carry the whole evidence argument.

Imagine two fictional training evaluations. A very large randomized study estimates a 0.3-point difference on a 100-point rubric, 95% CI 0.1–0.5. A much smaller pilot estimates a 3-point difference, 95% CI −1 to 7. The first may produce a conventional significant result, while the second may not. That does not make the first effect practically larger or prove the second program has no benefit. Magnitude and precision differ between the studies.

The practical question may be whether an extra training hour produces enough benefit to justify its burden. A precise 0.3-point difference could fall far below a justified threshold, even if the result is difficult to reconcile with an exact-zero model. The pilot’s interval may include both worthwhile and unhelpful effects. Reporting the estimate and interval makes those possibilities visible; using only significant/nonsignificant labels hides them.

BUILD THE REASONING · 02

Define worthwhile before seeing the favored result

A meaningful threshold should connect to the scale, purpose, people affected, costs, harms, and alternatives. It is not automatically Cohen’s conventional small/medium/large label or an arbitrary percentage. For a teaching rubric, a threshold may be tied to a concrete improvement in reasoning quality that instructors and learners can describe. For clinical outcomes, individual meaningful change, group mean differences, and measurement error are distinct considerations; one threshold should not silently stand in for all three.

Suppose a fictional program sets 2 rubric points as the minimum average benefit worth adding an hour of required training. The large study’s interval 0.1–0.5 lies entirely below that benchmark, so a positive finding is not sufficient for this adoption rationale. The pilot’s −1–7 interval crosses zero and the practical threshold; it is inconclusive for that decision. The right next action may involve a more precise evaluation, a lower-burden version, or a different outcome, depending on the original question.

A threshold chosen after seeing the result can rationalize a preferred conclusion. State who selected it, why, and when. If stakeholders disagree, show how the decision changes across defensible thresholds rather than disguising the value judgment as a statistical fact. A tiny average benefit might matter for a very low-cost population intervention, while the same benefit would not justify an intensive service that displaces other care.

BUILD THE REASONING · 03

Account for how the attractive result was selected

Suppose an analyst tests 20 independent outcomes at a .05 false-positive threshold when all corresponding null hypotheses are true. The chance of at least one rejection is 1−.95^20, approximately .642. This is an illustrative probability under stated independence and true-null assumptions, not a formula to apply blindly to every correlated outcome set. It shows why “we found one p<.05” means something different after many opportunities to select a result.

Prespecify primary outcomes and analysis decisions, report the full relevant set, and use a justified multiplicity approach when the research question requires it. Exploratory analyses can generate useful hypotheses, but label them as exploratory and seek further evidence. Splitting a sample repeatedly until one subgroup looks favorable, changing outcome time points, or omitting inconvenient analyses can create misleading confidence even when each calculation is technically correct.

The same principle applies to stopping data collection as soon as significance appears. A valid sequential design can plan interim decisions and their error control; repeated informal peeking followed by a conventional final p value does not automatically preserve the nominal interpretation. The student’s task is to identify the analysis-selection process and ask whether the reported uncertainty accounts for it, rather than treating a polished table as proof of a prespecified plan.

BUILD THE REASONING · 04

Distinguish no detected difference from evidence of equivalence

A nonsignificant difference test may reflect a small effect, noisy measurements, limited sample information, or several possibilities. It does not establish that two programs are equivalent. Equivalence requires a justified margin describing differences considered negligible, an appropriate design and analysis, and sufficiently precise evidence within that margin. Do not set the margin after seeing the interval merely to make the study pass.

For a decision discussion, compare the uncertainty interval with both zero and the meaningful thresholds, while respecting the analysis actually performed. An interval −5 to 6 on a rubric where 2 points matters leaves major possibilities unresolved. An interval −0.4 to 0.6 is more informative about small differences, but a formal equivalence claim still requires the appropriate prespecified framework rather than borrowing the label from visual inspection alone.

Write the recommendation as a chain: result → uncertainty → study quality → practical criterion → action. For example, a well-conducted study may support a small positive learning effect that does not justify adding an hour of required work under the program’s stated threshold. That is a useful negative adoption decision, not a failed study. Science is more informative when the report allows “the effect is probably too small for this purpose” as well as “promising but uncertain.”

FROM CONCEPT TO DECISION

The significant result is too small for the decision

1. 1

   Interpret the primary estimate

   The average difference is positive and relatively precise under the model. It remains well below the prespecified practical threshold, so the primary result does not justify the proposed burden on that rationale.

2. 2

   Examine the subgroup claim

   Ask whether the subgroup question was prespecified, whether an interaction was evaluated appropriately, how many comparisons were examined, and whether the subgroup estimate is precise. A favorable within-subgroup p value alone does not establish a difference between subgroups.

3. 3

   Consider alternatives without changing the result

   The committee may explore a shorter version or a targeted hypothesis for a new study. It should not lower the threshold or relabel the exploratory subgroup as the primary success to rescue the intervention.

4. 4

   State the decision

   Retain current instruction for now under the stated benefit/burden rule, report the small positive finding honestly, and describe the subgroup result as a hypothesis requiring additional evidence.

MAKE THE REASONING YOURS

Work through a new situation

READ WITH A PURPOSE

Best references

These readings deepen the specific models, evidence, and decisions in this lesson.

1. 01
   American Statistical Association: Statement on P-Values

   Use the six principles to distinguish a statistical model check from effect magnitude, practical value, and the probability that a hypothesis is true.

2. 02
   Cochrane Handbook, Chapter 15: Interpreting Results

   Study how effect magnitude, uncertainty, important differences, harms, and applicability contribute to interpretation.

3. 03
   Cochrane Handbook, Chapter 10: Analysis and Subgroups

   Read the subgroup and interaction sections to distinguish an effect difference between groups from separate within-group significance tests.

4. 04
   CONSORT 2025

   Use transparent reporting of prespecified outcomes, analysis changes, and additional analyses to assess whether a favorable result was selected after the fact.