Sampling, assignment & valid conclusions

START HERE

The essential idea

Random sampling selects people from a population using a chance process. Random assignment places enrolled participants into study conditions using a chance process. They solve different problems: representation and group comparability are related but distinct.

A study can randomly assign a convenience sample. That may strengthen a causal comparison within the study while leaving questions about generalizability. Conversely, a representative sample does not turn an observational association into evidence that one variable caused another.

Internal validity concerns the credibility of a causal interpretation within the study. External validity concerns how findings may apply to other people, settings, or times. Attrition, measurement quality, implementation, and context can affect what is defensible.

When reading a paper, identify the recruitment method, allocation method, comparison, measurement timing, and missing data. Then state what the design supports and what it leaves uncertain. Avoid treating “random” as a general stamp of quality.

BUILD THE REASONING · 01

Who entered and how were conditions assigned?

Define the target population, the sampling frame that can actually be contacted, the invited people, the enrolled sample, and the analyzed sample. Random sampling uses a chance process to select from a defined frame; it cannot include people absent from that frame. Random assignment allocates enrolled participants to conditions. Assignment can strengthen a causal comparison while leaving the sample unrepresentative of the target population.

A campus study can recruit 60 volunteers from one evening class and randomly assign two learning methods. The assignment addresses systematic allocation differences within those volunteers. It does not make them representative of all counseling students, remove differential dropout, or ensure a valid measure. “Randomized” is a statement about a particular procedure, not a complete quality judgment.

BUILD THE REASONING · 02

Follow the threats through a study

Internal validity concerns whether the observed difference supports the proposed causal explanation. Selection, concurrent changes, differential attrition, measurement differences, and departures from assignment can create alternatives. External validity concerns application to other populations, settings, times, and implementation conditions. Measurement validity asks whether the outcome represents the intended construct. These questions interact but cannot substitute for one another.

BUILD THE REASONING · 03

Write a conclusion matched to the sample

Report numbers at each stage, recruitment and assignment methods, exclusions, outcome availability, and reasons for missingness when known. Compare relevant baseline and retention patterns without assuming that a nonsignificant test proves balance. A large sample shrinks some random uncertainty but cannot repair a consistently excluded population or a biased outcome process.

A defensible conclusion can be specific: “Among volunteers from this evening course, assigned retrieval practice produced a higher mean case score than assigned rereading under these conditions; generalization to other programs and students with different access needs remains uncertain.” That causal wording still requires credible execution, appropriate analysis, and reasonable handling of missing data; assignment alone is not sufficient.

FROM CONCEPT TO DECISION

The randomized study with unequal losses

1. 1

   Separate the strengths

   Assignment can reduce baseline allocation confounding; recruitment remains a convenience sample from one course.

2. 2

   Calculate retention

   Outcome availability is 36/40 = 90% versus 25/40 = 62.5%. The 27.5 percentage-point difference makes the missingness process consequential.

3. 3

   Identify the alternative explanation

   If outcome availability relates to overwhelm and performance differently across arms, completers may no longer represent the assigned groups comparably.

4. 4

   Request the evidence needed

   Inspect participant flow, reasons and timing of missingness, prespecified analysis, and sensitivity analyses under plausible missing-outcome assumptions. Do not invent the absent outcomes or assume randomization cured the problem.

MAKE THE REASONING YOURS

Work through a new situation

READ WITH A PURPOSE

Best references

These readings deepen the specific models, evidence, and decisions in this lesson.

1. 01
   CONSORT 2025: Reporting Randomised Trials

   Inspect the original reporting guideline for allocation, participant flow, prespecified outcomes, analysis, and harms. Transparent reporting permits appraisal; a completed checklist does not guarantee low bias.

2. 02
   Cochrane Handbook, Chapter 8: Risk of Bias in a Randomized Trial

   Follow the domains for randomization, deviations, missing outcomes, measurement, and selective reporting when judging the fictional study.

3. 03
   Cochrane Handbook, Chapter 25: Non-Randomized Studies

   Compare confounding and selection in nonrandomized studies with the separate problems of recruitment and generalization.