Working Paper: Analysis of Within-Study Comparisons

Full Title: Assessing Correspondence between Experimental and Non-Experimental Results in Within-Study-Comparisons

Full Abstract: In within-study comparison (WSC) designs, treatment effects from a non-experimental design, such as an observational study or a regression-discontinuity design, are compared to results obtained from a well-designed randomized control trial (RCT) with the same target population. The goal of the WSC is to assess whether the non-experimental and experimental designs yield the same results in field settings, and the contexts and conditions under which non-experimental methods replicate benchmark RCT estimates. A common analytic challenge with WSCs, however, is identifying appropriate criteria for determining whether non-experimental and benchmark results replicate. This paper examines methods for assessing correspondence in benchmark RCT and non-experimental results in WSC designs. We examine measures for assessing correspondence in results, and the relative advantages and limitations of these approaches. We identify two classes of measures: conclusion- and distance-based correspondence approaches. Conclusion-based measures indicate correspondence in results if a researcher or policy-maker draws the same conclusion from the experiment and non-experiment. Distance-based measures investigate whether the difference in estimates is small enough to claim correspondence between methods. We use a simulation study to examine the statistical properties of correspondence measures, and recommend a new approach that combines traditional significance testing and equivalence testing in the same framework. The paper concludes with practical advice on assessing and interpreting results in WSC contexts.

EdPolicyWorks Working Paper Series No. 46. April 2016.