Choosing a statistical test & reading its result

START HERE

The essential idea

A statistical test addresses a specific question under assumptions. Comparing two independent group means differs from comparing repeated measurements from the same people. A t test, analysis of variance, chi-square test, correlation, and regression are not interchangeable labels for significance. Match the method to the design, outcome, relationship of observations, and distributional assumptions.

A p value summarizes how incompatible the observed data are with a specified model under its assumptions; it is not the probability that the hypothesis is true. Statistical significance alone does not show that an effect is large, useful, causal, or replicable. Interpret an effect estimate and uncertainty alongside the test.

Type I error concerns rejecting a true null hypothesis; Type II error concerns failing to reject a false null hypothesis. Power relates to the probability of detecting a specified effect under the design's conditions. Multiple comparisons, missing data, measurement quality, and selective reporting can complicate conclusions.

For a fictional pre/post training study, ask whether the measurements are paired, whether there is a comparison group, how missing cases were handled, and whether the outcome captures the intended skill. A change in average score does not automatically establish the program caused the change.

BUILD THE REASONING · 01

A decision path for common analyses

Write the research question, unit of analysis, outcome scale, number of conditions, and relationship between observations before naming a test. Independent groups contain different people; paired data link observations from the same person or a meaningful matched pair. Equal group sizes do not create pairing. Students nested within classrooms are not independent merely because each has one row.

Inspect the data, missingness, and model assumptions. For a paired t procedure, the relevant distribution is the difference scores, and pairs should be independent of other pairs. A small sample is sensitive to strong skew/outliers. A nonparametric alternative changes assumptions and sometimes the estimand; it is not an automatic cure for a flawed design. Obtain statistical guidance when clustering, severe missingness, or the study structure exceeds the introductory model.

BUILD THE REASONING · 02

A small dataset you can calculate

Six fictional trainees complete a 0–20 case-rubric task before and after a workshop. Higher scores are better. Each row is one trainee. Differences are after minus before. The sample is deliberately small for arithmetic practice; it is not evidence about an actual workshop.

BUILD THE REASONING · 03

From differences to a report

The differences sum to 18, so mean change = 18/6 = 3 points. Deviations from 3 are −2, −1, −1, 0, 1, 3; their squared sum is 16. Sample variance of differences = 16/(6−1) = 3.2; SD = √3.2 ≈ 1.789. Standard error = 1.789/√6 ≈ 0.730. To test a zero mean difference, t = 3/0.730 ≈ 4.108 with 5 degrees of freedom.

For a prespecified two-sided test, p ≈ .0093. The 95% t critical value for df = 5 is approximately 2.571, so the interval is 3 ± 2.571×0.730 = [1.123, 4.877]. A standardized paired effect using the SD of differences is d_z = 3/1.789 ≈ 1.677; it uses a different denominator from an independent-groups Cohen’s d. These arithmetic results assume an appropriate t model; six differences cannot establish the population shape convincingly.

Model results paragraph: “Among six fictional trainees with complete paired observations, mean case-rubric scores increased from 7.0 to 10.0. Mean within-person change was 3.0 points (95% CI 1.12 to 4.88), t(5) = 4.11, two-sided p = .009. The small uncontrolled sample and possible repeated-task practice effects limit interpretation; the result describes observed change and does not isolate a workshop effect.”

BUILD THE REASONING · 04

What the analysis leaves open

A p value evaluates incompatibility with the specified model and null value under assumptions; it is not the probability that the workshop works. A pre/post improvement can reflect practice, maturation, outside learning, regression toward the mean, or the workshop. A comparison design addresses a different causal question than the paired test alone.

If a trainee lacks the post score, do not replace it with zero or silently analyze unmatched rows. Report the missingness and use an appropriate prespecified strategy. If many outcomes were tried and only the smallest p value was reported, the nominal result omits the selection process. A report should show effect, uncertainty, analysis choices, missingness, and limitations together.

FROM CONCEPT TO DECISION

A second study requires a different test

1. 1

   Identify dependence

   These are independent groups. Unequal sample sizes are not the decisive issue; absence of meaningful pairs is.

2. 2

   Choose a candidate mean comparison

   A Welch two-sample t test is a candidate because it does not require equal population variances. Inspect distribution/outliers and verify independent observations.

3. 3

   Specify the estimand and report

   Estimate A minus B in rubric points with its confidence interval and test output. State the sample sizes and analysis assumptions.

4. 4

   Limit causal language

   If trainees selected their workshop, selection differences remain possible. The choice of a valid statistical test does not randomize the study.

MAKE THE REASONING YOURS

Work through a new situation

READ WITH A PURPOSE

Best references

These readings deepen the specific models, evidence, and decisions in this lesson.

1. 01
   NIST: Analysis of Paired Observations

   Reproduce the difference-score mean, standard deviation, t statistic, and degrees of freedom used in the original fictional dataset.

2. 02
   NIST: Two-Sample t Test for Equal Means

   Compare paired versus independent observations and the Welch–Satterthwaite approach for unequal variances.

3. 03
   ASA: Statement on Statistical Significance and P-Values

   Use the statement’s principles to separate model incompatibility from effect size, practical importance, and the probability that a hypothesis is true.

4. 04
   NIST: What Are Confidence Intervals?

   Review the repeated-sampling meaning of confidence and the role of interval width; a confidence interval is not the range containing 95% of individual observations.