Process versus outcome evaluation: choosing useful measures

START HERE

The essential idea

Select measures that answer distinct delivery and outcome questions. Create operational definitions with denominators, timing, and missing-data rules. Interpret conflicting process and outcome findings without changing the goal after the fact.

BUILD THE REASONING · 01

Measure the question you actually need answered

Process evaluation asks how a program was implemented: whom it reached, what was delivered, whether the intended components occurred, and what conditions shaped participation. Outcome evaluation asks what changed. Impact evaluation asks what change can credibly be attributed to the program compared with what would otherwise have happened. A local evaluation can usefully answer the first two without possessing a design that settles the third. The report should keep those claims separate.

Consider an original fictional peer-consultation program for early-placement counseling trainees. Its intended benefit is better use of supervision, demonstrated by focused questions and follow-through on feedback. Staff can count meetings, measure satisfaction, inspect performance on fictional cases, and ask about later use. Each supplies a different part of the picture. High satisfaction is valuable experience information but cannot replace evidence of the skill the program promised to teach.

A measure should have a direct link to the decision. If the program is considering a shorter session, collect the time actually used for modeling, practice, feedback, and closure, not only total attendance. If the concern is transfer, a repeated identical task may show familiarity more than flexible use. Choose a new but comparable scenario and clear scoring anchors. If the concern is access, include information from nonparticipants rather than inferring their reasons from attendees.

BUILD THE REASONING · 02

Create a measure specification someone else can apply

For every measure, define the unit, numerator and denominator when applicable, source, collection time, collector, missing-data category, and interpretation. “Eighty percent participation” is uninterpretable without knowing whether the denominator is invited trainees, enrolled trainees, meetings offered, or available seats. A participant who attended three of four sessions contributes differently to person-level reach and session-level attendance; both can be useful if named accurately.

Use original rubrics as transparent learning measures, not validated clinical scales. For the fictional supervision-question task, score whether the trainee identifies relevant evidence, states uncertainty, and makes a specific feedback request. Give examples for each anchor, train raters, and examine disagreement before using scores in a consequential decision. If raters know who attended the program, consider whether expectancy could influence scoring and whether masking is feasible.

Keep burden proportional. A ten-minute survey after every short meeting may reduce participation and generate incomplete data. Use brief measures where they answer the question, collect richer feedback at a planned point, and avoid repeating information already available through authorized records. Make clear how voluntary evaluation participation differs from training requirements and how feedback is protected from grading or employment consequences under the actual arrangement.

BUILD THE REASONING · 03

Read the flow before the percentage

In a fictional cohort, 50 trainees are eligible and invited, 32 attend at least once, 24 complete both skill tasks, and 16 provide transfer follow-up. Reach is 32/50 = 64%. Paired skill data cover 24/32 = 75% of attendees, while follow-up covers 16/32 = 50%. These proportions answer three different coverage questions; their denominators are invitees or attendees as specified. If 12 follow-up respondents describe using a focused question, the observed proportion is 12/16 = 75% of respondents, not 75% of all invited trainees.

You can state that at least 12 of the 32 attendees reported the behavior, while the behavior of nonrespondents remains unknown. Do not treat every nonresponse as either failure or success without a justified analysis framework. Ask whether missingness relates to workload, dissatisfaction, difficulty, or access, using proportionate and noncoercive follow-up. Report what is known about the reasons and what remains uncertain.

Disaggregate thoughtfully when an access question warrants it, while protecting small groups from identification. A schedule category may be more directly relevant than a broad demographic label, and a tiny subgroup percentage can be unstable or expose someone. Specify why the comparison is useful, whether people can decline providing the information, and how small-cell reporting is handled under the actual policy. More detailed data are not automatically better evaluation.

BUILD THE REASONING · 04

Use conflicting indicators to locate the next question

High reach with low skill gains could reflect weak instruction, insufficient practice, a poor measure, or an outcome assessed too soon. Low reach with strong gains among attendees could mean the program works for a selected group but has an access problem; it does not justify claiming that the full eligible population benefited. Strong skill gains with little reported transfer could reflect workplace opportunity, supervisor response, or a mismatch between the practice case and actual tasks.

A process measure helps distinguish these possibilities. If logs show that feedback was omitted, a null skill result does not evaluate the intended full program. If feedback occurred and ratings are reliable but the practice tasks resemble the post-test exactly, apparent gains may reflect rehearsal of the assessment. If transfer respondents are only the most engaged trainees, the follow-up cannot establish widespread use. The right interpretation uses the indicators together without averaging them into one reassuring score.

Write a decision record: finding; competing explanations; next evidence needed; proposed change; owner and review date. Keep the original outcome visible even if another measure looks more favorable. A decision to improve access, calibrate a rubric, or protect feedback time can be justified by process evidence. A claim that the program caused better client outcomes would require a different outcome and stronger design. Evaluation becomes useful when it directs a specific revision and then checks whether that revision solved the documented problem.

FROM CONCEPT TO DECISION

Everyone liked the meetings, but practice was missing

1. 1

   Preserve the positive finding accurately

   Respondents valued the experience. That is worth knowing, but the denominator and response pattern must be stated. Satisfaction is not the same construct as producing a focused supervision question.

2. 2

   Identify the delivery mismatch

   The intended active cycle was seldom completed. The result therefore informs the version actually delivered, not a fully implemented model–practice–feedback program. Examine why discussion displaced practice before adding another outcome survey.

3. 3

   Evaluate measurement and missingness

   Check task comparability, scoring anchors, rater agreement, and whether those missing follow-up differ in relevant ways. Do not attribute every weak result to delivery if the measure or data collection is also inadequate.

4. 4

   Choose a concrete revision and check

   Protect a defined practice/feedback segment, train facilitators to preserve it, and sample delivery observations. Repeat a comparable new-case assessment and follow-up with clear denominators. Keep satisfaction as a separate experience indicator.

MAKE THE REASONING YOURS

Work through a new situation

READ WITH A PURPOSE

Best references

These readings deepen the specific models, evidence, and decisions in this lesson.

1. 01
   CDC Program Evaluation Framework, 2024

   Use the framework to connect context, program description, evaluation design, evidence, conclusions, and use. Its public-health examples require adaptation to a counseling setting.

2. 02
   AHRQ: Elements of an Evaluation Plan

   Use the planning guidance to connect a question with an appropriate data source, collection method, criteria, and practical decision.

3. 03
   AERA/APA/NCME: Testing Standards (2014)

   Review reliability/precision and validity for the intended interpretation before treating an original learning rubric as a dependable outcome measure.