Effect sizes, intervals & practical meaning

START HERE

The essential idea

An effect size expresses the magnitude of a relationship or difference in a chosen metric. A p-value answers a different statistical question under a specified model. Neither alone tells you whether a change matters to a client or program.

An interval estimate makes uncertainty around an estimate more visible. A wide interval may include meaningfully different possibilities. Examine its scale and the study design instead of treating an interval as a binary pass-or-fail marker.

Practical interpretation depends on baseline conditions, meaningful outcomes, costs, burdens, duration, and whose priorities count. A small average difference can have different implications across contexts. A large result from a biased design still needs scrutiny.

In a class discussion, report direction, magnitude, uncertainty, and limitations together. If the article does not provide enough information, say what is missing rather than inventing precision.

BUILD THE REASONING · 01

State the effect in understandable units first

A fictional program yields a mean case-rubric score of 18 compared with 15 in a comparison group, so the raw difference is 3 rubric points. If the appropriate pooled standard deviation is 6, an illustrative Cohen’s d is 3/6 = 0.50. The standardized number permits some comparisons across scales but loses the immediate meaning of a rubric point and depends on the denominator chosen.

For a binary outcome, suppose 30 of 50 program participants and 20 of 50 comparison participants meet a predefined criterion. The proportions are .60 and .40: risk difference = .20, or 20 percentage points, and risk ratio = .60/.40 = 1.50. “50% higher” is a relative comparison, not a 50-percentage-point gain. These are descriptive contrasts; causal interpretation still depends on design and implementation.

BUILD THE REASONING · 02

Compare uncertainty with a meaningful decision

An estimated mean difference of 3 points with a 95% confidence interval from 1 to 5 is more informative than “significant.” Under the model and repeated sampling, the interval procedure covers the population quantity at its stated long-run rate. The interval does not contain 95% of individual outcomes, and it is not a 95% probability statement about this fixed parameter under the ordinary frequentist interpretation.

Suppose stakeholders defined 4 rubric points as a practically worthwhile group difference before seeing results. The interval 1–5 includes effects below and above that threshold. There is evidence of a positive difference under the model, but whether it is large enough for the intended decision remains uncertain. An interval −1 to 7 would leave even the direction uncertain. Failure to reject a zero difference is not proof of equivalence; equivalence needs its own margin and analysis.

BUILD THE REASONING · 03

Bring burden and study quality into the decision

A precise average difference can still be biased by confounding, selective reporting, or an unsuitable measure. A small effect may matter when a program is inexpensive and reaches many people; a larger effect may not justify a burdensome program if important harms or access costs are omitted. Identify whose outcomes and priorities define “worthwhile.”

Report direction, raw magnitude, uncertainty, the practical benchmark, and limits. Then state an action proportionate to the evidence: continue a limited evaluation, collect a more precise outcome, examine implementation, or adopt with monitoring when justified. An average treatment effect does not guarantee that each individual benefits by that amount; examine variation and adverse outcomes when available.

FROM CONCEPT TO DECISION

A positive result that does not settle adoption

1. 1

   Interpret direction

   The interval is above zero under the reported model, supporting a positive average difference if the design and analysis are credible.

2. 2

   Interpret practical magnitude

   The point estimate is below the 3-point benchmark, and the interval includes both insufficient and worthwhile effects. A significance label cannot settle the time tradeoff.

3. 3

   Check the wider evidence

   Review missingness, adherence, harms, scoring quality, and whether the staff-hour estimate reflects actual delivery. Good allocation and independent scoring are strengths, not the full appraisal.

4. 4

   Choose a bounded next decision

   A limited continuation with a prespecified evaluation may be justified, but universal adoption as clearly worthwhile exceeds the current result.

MAKE THE REASONING YOURS

Work through a new situation

READ WITH A PURPOSE

Best references

These readings deepen the specific models, evidence, and decisions in this lesson.

1. 01
   ASA: Statement on Statistical Significance and P-Values

   Use the statement’s principles to separate model incompatibility from effect size, practical importance, and the probability that a hypothesis is true.

2. 02
   NIST: What Are Confidence Intervals?

   Review the repeated-sampling meaning of confidence and the role of interval width; a confidence interval is not the range containing 95% of individual observations.

3. 03
   Cochrane Handbook, Chapter 10: Analysing Data

   Compare mean differences, standardized effects, binary effect measures, and uncertainty; note that the handbook’s synthesis context requires additional assumptions.