Read evidence with a critical eye

START HERE

The essential idea

Ask how large or meaningful a reported difference is, how uncertain the estimate may be, and whether it matters in context. Statistical significance does not automatically mean a large or useful effect.

Consider recruitment, missing data, measurement, comparison conditions, and alternative explanations. A study’s limitations do not necessarily make it useless; they shape how cautiously the result should be used.

For comprehensive-exam study, practice explaining the difference between reliability, validity, sampling, assignment, association, and causal inference. Retrieval and explanation are more useful than recognizing familiar words alone.

BUILD THE REASONING · 01

Read the result as an argument

Start with the question and intended decision. Identify the population, intervention/exposure, comparator, outcome, timing, and design. Then extract recruitment, assignment, sample flow, measurement, analysis, effect estimate, uncertainty, and harms. An abstract’s favorable conclusion is the authors’ interpretation; check whether the methods and complete results support it.

Create an evidence row rather than copying the discussion. Record the actual comparison and time point, because a post-treatment symptom score differs from long-term functioning. Note who was excluded and what the comparator received. A program outperforming a waitlist does not by itself show superiority to another active treatment with similar attention and expectancy.

BUILD THE REASONING · 02

Bias, imprecision, and applicability are different

Risk of bias concerns systematic error in a specific result: the randomization process, deviations, missing data, outcome measurement, or selective reporting may favor an estimate. Imprecision concerns how uncertain the estimate is under the analysis. Applicability concerns whether the people, setting, intervention, comparator, and outcomes match the decision. A narrow interval can surround a biased estimate, and a rigorous result can still address a different population.

For a review, examine the search, eligibility criteria, included-study quality, heterogeneity, missing studies/results, and whether pooling is clinically and methodologically sensible. A pooled effect is not automatically more trustworthy than its underlying studies. Compare the review’s question with the current client or program question, and examine whether favorable average results conceal harms or important subgroup uncertainties.

BUILD THE REASONING · 03

Write a calibrated evidence statement

Use four sentences: what was studied; the effect and uncertainty; the most consequential bias/design limit; and what the evidence supports for the present decision. Distinguish “not reported” from “did not occur.” Absence of a harms section does not establish absence of harm. Distinguish a prespecified primary analysis from exploratory subgroup findings selected after inspecting the data.

A practical recommendation integrates the evidence with preferences, access, resources, clinical judgment, and ongoing outcome monitoring. It should state what remains uncertain and what information would change the decision. Avoid ranking studies solely by p value, prestige, sample size, or a design label. Each can be relevant, but none substitutes for the complete argument.

FROM CONCEPT TO DECISION

A promising abstract, a narrower defensible claim

1. 1

   Extract the actual result

   The result concerns self-reported stress at four weeks versus waitlist, not long-term functioning or superiority to active care.

2. 2

   Assess magnitude and bias

   The interval lies below the proposed 4-point worthwhile threshold. Differential missingness may bias the estimate, so inspect reasons, analysis assumptions, and sensitivity results.

3. 3

   Assess applicability

   Excluded shift workers differ in access and context from the target clinic. Generalization requires an explicit argument and likely further local evaluation.

4. 4

   Write the conclusion

   The trial suggests a short-term score difference under its conditions, but practical magnitude, missingness, and population mismatch limit claims of clinically worthwhile benefit for the target clinic.

MAKE THE REASONING YOURS

Work through a new situation

READ WITH A PURPOSE

Best references

These readings deepen the specific models, evidence, and decisions in this lesson.

1. 01
   Cochrane Handbook, Chapter 8: Risk of Bias in a Randomized Trial

   Follow the domains for randomization, deviations, missing outcomes, measurement, and selective reporting when judging the fictional study.

2. 02
   ASA: Statement on Statistical Significance and P-Values

   Use the statement’s principles to separate model incompatibility from effect size, practical importance, and the probability that a hypothesis is true.

3. 03
   Cochrane Handbook, Chapter 13: Missing Evidence

   Examine how missing studies and selectively missing results can affect a synthesis even when reported individual results look precise.

4. 04
   CONSORT 2025: Reporting Randomised Trials

   Inspect the original reporting guideline for allocation, participant flow, prespecified outcomes, analysis, and harms. Transparent reporting permits appraisal; a completed checklist does not guarantee low bias.