VEST Workshop with Dr. Robert Olsen

Over the last 20 years, the use of randomized controlled trials (RCTs) and strong quasi-experimental designs (QEDs) has elevated the internal validity of causal research in education. More recently, researchers and research funders have turned their attention to the external validity or generalizability of these studies-that is, whether the study findings are relevant to the populations of students and schools that motivate the studies. This workshop will help researchers conduct RCTs or QEDs that generalize better to the populations about which the researchers aim to learn. The workshop will include presentations by Dr. Robert Olsen and instructor-led small group exercises by workshop participants.

The workshop will cover:

• Selection of a target population-how to choose the collection of students and schools for which the study will aim to estimate the average impact.
• External validity bias from unrepresentative samples-how convenience sampling of districts and schools yields unrepresentative samples and biased estimates of the average impact in the target population.
• Sampling designs to reduce external validity bias-how to obtain more representative samples of students and schools to participate in CTs and QEDs.
• Analysis techniques to reduce external validity bias-how to statistically adjust for differences between the sample and the target population using weighting or regression
• Assessment of likely external validity bias-how to assess whether the impact estimates from an RCT or QED generalize to the target population.

This lecture Is open to people in the greater Charlottesville community and beyond as well as all UvA faculty, post-doctoral researchers, students, and staff. It is sponsored by the Virginia Education Sciences Training Program (VEST) through a grant from the U.S. Department of Education.

SPEAKER BIO

Dr. Robert Olsen is a research professor at the George Washington Institute of Public Policy at George Washington University. He has 25 years of experience conducting rigorous evaluations of social programs, including educational interventions for disadvantaged students. He has played lead roles on randomized trials of the Upward Bound Program, college advising for high school students, and smartphone apps to help community college students persist in school. Over the past 12 years, he has served as Principal Investigator or co-Principal Investigator for numerous competitive grants on the generalizability of impact studies, including grants from the Institute of Education Sciences (IES), the National Science Foundation, and the William T. Grant Foundation. This research has addressed the limitations of randomized trials conducted in convenience samples, the benefits of formal sampling methods when selecting districts and schools to recruit for randomized trials, and statistical methods for extrapolating from randomized trials to predict the impacts of educational interventions in individual districts or schools. His research on generalizability has been published in the Journal of Research on Educational Effectiveness, the Journal of Policy Analysis and Management, Educational Evaluation and Policy Analysis, Educational Researcher, and New Directions in Evaluation. Finally, with Dr. Elizabeth Tipton and with funding from IES, he developed a guide for applied education researchers on how to conduct impact studies with greater generalizability. Olsen received his Ph.D. in Labor Economics from Cornell University in 1999.