Reliability is not validity

START HERE

The essential idea

Reliability concerns consistency or precision. Validity concerns the evidence supporting a particular interpretation and use of scores. A measure can be consistent while being inappropriate for the decision you want to make.

The question is not simply whether a test is valid in the abstract. Ask: valid evidence for which interpretation, for whom, under what conditions, and for what use? Consider the population, context, and consequences.

In an exam item, identify whether the scenario describes repeatability, agreement, internal consistency, or evidence for meaning and use. Avoid inferring diagnostic conclusions from a score without appropriate assessment.

BUILD THE REASONING · 01

Ask what is supposed to remain consistent

Internal consistency concerns relationships among items in a score; test–retest evidence concerns stability across occasions when the construct is expected to be stable; interrater evidence concerns agreement across raters. These are different questions. A high internal-consistency coefficient cannot establish that two supervisors will score the same performance similarly or that a measure detects meaningful change.

Reliability estimates depend on the sample, administration, score, and method. Correlation between two raters can be high even if one consistently gives higher scores; agreement methods must match the decision. Before using a coefficient, identify the design, population, time interval, estimate and interval, and what variation counts as error. Avoid treating a single universal cutoff as sufficient for every use.

BUILD THE REASONING · 02

A worked instrument-review dossier

Suppose a clinic must choose a fictional support-needs measure for intake discussion and monthly progress review. The table is invented teaching data, not an evaluation of a real product. The preferred choice depends on the intended use; a numerically impressive coefficient cannot compensate for missing content or an unsuitable reference population.

BUILD THE REASONING · 03

Validity is an argument about an interpretation

Content evidence asks whether relevant material is represented and understandable. Response-process evidence asks how people understand/respond or how raters score. Internal structure examines whether the score organization fits the intended construct. Relations with other variables test predicted convergent, discriminant, or criterion relationships. Consequences and fairness matter when judging a use. No single correlation or coefficient settles the whole argument.

For intake discussion in the fictional adult clinic, B has a better content/population match. For declaring an individual clinically improved, B still lacks an established meaningful-change threshold and needs instrument-specific change procedures plus other evidence. A new translation, remote format, or high-stakes decision can change the validity question. State the permitted interpretation and its limits in the recommendation.

FROM CONCEPT TO DECISION

The highest alpha is not the best decision

1. 1

   Identify the intended decision

   Ending services is an individual, consequential decision about need and change, not simply a description of student academic stress.

2. 2

   Match the evidence to that decision

   A’s internal consistency does not address omitted caregiving/work content, population fit, longitudinal precision, or a supported threshold for ending care.

3. 3

   Make a limited recommendation

   B may better support collaborative discussion, but its preliminary change evidence still cannot independently justify discharge.

4. 4

   Specify the missing evidence

   Seek appropriate change/error evidence, meaningful functional criteria, fair administration, and a multifactor clinical decision process rather than substituting another single score.

MAKE THE REASONING YOURS

Work through a new situation

READ WITH A PURPOSE

Best references

These readings deepen the specific models, evidence, and decisions in this lesson.

1. 01
   AERA/APA/NCME: Standards for Educational and Psychological Testing (2014)

   Read Chapters 1–3 on intended interpretations, precision, and fairness. Use the standards to evaluate evidence for a particular use, not to label an instrument universally valid.

2. 02
   COSMIN Manual, Version 2: Outcome Measurement Instruments

   Use the distinctions among content validity, structural validity, reliability, measurement error, and responsiveness to evaluate the fictional instrument dossier.

3. 03
   COSMIN: Reporting Guideline for Measurement-Property Studies

   Use the reporting checklist to locate population, design, analysis, and estimates needed to appraise a reliability or validity claim; it is a separate reporting tool from the COSMIN review manual.