Ask a useful program-evaluation question

COMPANION VIDEO

Watch this concept unfold

The captioned companion video is available to accounts with video access. Read and practice the lesson below for free.

START HERE

The essential idea

A process question asks how a program was implemented or experienced. An outcome question asks whether an intended result changed. An impact question asks whether the program produced the change rather than other factors.

Begin with a clear purpose and the people who will use the findings. Define an indicator that matches the question and consider feasibility, ethics, access, and unintended consequences.

Satisfaction, participation, and improvement are different outcomes. A well-attended workshop does not prove that participants learned, and a pre–post difference alone does not establish causation.

BUILD THE REASONING · 01

A logic model makes the program’s reasoning visible

Begin with the need, intended participants, context, and decision the evaluation will inform. A logic model links inputs to activities, outputs, proximal outcomes, and longer-term outcomes. The arrows contain assumptions: offering a workshop leads to attendance only if people can access it; attendance leads to skill only if the activities teach and practice that skill; skill changes real-world behavior only if opportunities and supports exist.

This original fictional program helps commuting counseling students use academic support before a crisis. The purpose of evaluation is to decide whether to continue the format and which access barriers to fix. Stakeholders include students who attend, students who decline, faculty, access staff, and program administrators; relying only on enthusiastic attendees would narrow the question.

Swipe or scroll horizontally to compare all columns. With a keyboard, focus this table and use the arrow keys.

Logic-model element	Fictional program specification	Assumption or contextual condition
Need	Students report difficulty locating and requesting timely academic support.	Student-defined barriers differ by schedule, disability, language, and care responsibilities.
Inputs	Two trained facilitators, accessible room/remote option, four workshop slots, support-directory time.	Staff and spaces are actually available within the existing budget.
Activities	Map services; model a request; rehearse with feedback; help students plan one contact.	Examples fit actual university procedures and permit limited personal disclosure.
Outputs	Invitations delivered, accessible sessions held, students reached, rehearsal attempts completed.	Counts retain denominators and do not equate attendance with benefit.
Proximal outcomes	More accurate knowledge of access routes and improved request-rubric performance.	The assessment samples the taught skill with clear scoring.
Intermediate outcomes	Students make useful contacts and report fewer unresolved access barriers over four weeks.	Receiving services have capacity; students can follow through.
Longer-term outcomes	Improved continuity of academic support and participation in training.	Many other influences affect retention; the pilot alone cannot establish causal impact.

BUILD THE REASONING · 02

Build an indicator and decision table

For each question specify a measure, denominator, time, data source, responsible person, and decision rule. A benchmark can guide a local program decision without being a validated clinical cutoff. Set it before results when possible and state its rationale. Use minimal necessary information, protect identities, and obtain the institution’s required oversight determination; evaluation data can expose barriers and power relationships.

Swipe or scroll horizontally to compare all columns. With a keyboard, focus this table and use the arrow keys.

Question	Indicator and timing	Interpretation/action
Did the program reach intended students?	Attendees / invited eligible students, with voluntary access-barrier feedback after invitations.	Low reach prompts recruitment and access review, not a claim that students lack motivation.
Was the active component delivered?	Completed feedback/retry cycles / planned cycles each session.	Missing rehearsal changes what intervention was actually delivered.
Did the targeted skill change?	Same rubric on different pre/post request cases, with rater calibration.	Report distributions, paired availability, and scoring limits; avoid practice-task identity effects.
Did support become usable?	Useful contacts / attempted contacts and unresolved barriers at four weeks.	Low usefulness despite skill gains directs attention to service capacity and fit.
Was the program burdensome or harmful?	Optional report of pressure, privacy concerns, time burden, or adverse experiences.	Review and respond to negative effects alongside favorable ratings.

BUILD THE REASONING · 03

Delivery, outcomes, and impact are separate claims

If 60 students are invited, 30 attend, 24 complete paired tasks, and 12 answer follow-up, report all four numbers. A satisfaction rate among the 12 respondents does not describe all 60 invitees. Examine why data are missing and whether respondent experiences differ; do not fill absent responses with presumed success or failure.

A useful evaluation report explains what happened, for whom, how it was delivered, what changed, and what decision follows. An outcome increase is not necessarily program impact: causal impact requires a defensible comparison with what would otherwise have happened. A small local pilot can still identify a clear operational improvement, such as moving sessions to an accessible time, without claiming that it caused retention to rise.

FROM CONCEPT TO DECISION

Good skill scores, poor access outcomes

FICTIONAL CASE

A fictional four-session support program invites 60 students; 30 attend, 24 complete pre/post cases, and 12 respond at four weeks. The 24 improve on request clarity. Of 18 reported support contacts among the 12 respondents, 6 lead to useful support; several services have long waits.

Keep the denominators visible
Reach is 30/60 = 50%; paired data cover 24/30 attendees = 80%; follow-up covers 12/30 = 40%. The 6/18 contact usefulness rate concerns reported contacts, not all students.
Locate the weak arrow
Skill improvement among measured attendees does not ensure available services. Long waits challenge the skill-to-useful-support link in the logic model.
Choose a proportional revision
Update the service map, engage relevant access staff through an authorized project process, and revise timing/referral options. Do not respond only by adding more request practice.
State limits and next evaluation
Follow-up is incomplete and there is no causal comparison. Reassess reach, usable contacts, and burden after the access changes while reporting missingness.

What the case shows

The evaluation produces an actionable improvement by locating where the program’s intended chain breaks. It does not need to exaggerate a causal retention effect to be useful.

MAKE THE REASONING YOURS

Work through a new situation

Preparing your practice space…

TRY IT IN CONTEXT

Put the lesson to work.

4 case-based questions. Plausible choices. Feedback that explains the distinction.

Start case practice

READ WITH A PURPOSE

Best references

These readings deepen the specific models, evidence, and decisions in this lesson.

CDC: Program Evaluation Framework, 2024
Use the six-step framework and cross-cutting actions to connect context, program description, evaluation design, evidence, conclusions, and use.
CDC: Developing and Using a Logic Model
Compare inputs, activities, outputs, and outcomes, then test the assumptions in the original counseling-program example.
AHRQ: Elements of an Evaluation Plan
Use the practical planning questions to specify measures, collection methods, resources, and the intended use of findings.

KEEP THE TAKEAWAY

Define success before choosing the measurement.

Save this lesson to your study log. Your case-practice result is shown separately in the quiz.

CONTINUE IN PROGRAM EVALUATIONRead evidence with a critical eye

Independent educational material for study alongside course readings and supervision. Examples are fictional. Completion and quiz scores do not establish clinical competence, university credit, or exam readiness. AI-assisted content review is separate from qualified human educational approval.