Estimators for Topic-Sampling Designs

When researchers design an experiment, they must usually fix important details, or what we call the “topic” of the experiment. For example, researchers studying the impact of party cues on attitudes must inform respondents of the parties’ positions on a particular policy. In doing so, the researchers implement just one of many possible designs. Clifford, Leeper, and Rainey (2023) argue that researchers should implement many of the possible designs in parallel—what they call “topic sampling”—to generalize to a larger population of topics. We describe two estimators for topic-sampling designs. First, we describe a nonparametric estimator of the typical effect that is unbiased under the assumptions of the design. Second, we describe a hierarchical model that researchers can use to describe the heterogeneity. We suggest describing the variation in three ways: (1) the standard deviation in treatment effects across topics, (2) the treatment effects for particular topics, and (3) perhaps how the treatment effects for particular topics vary with topic-level predictors. We evaluate the performance of the hierarchical model using the Strengthening Democracy Challenge megastudy and show that the hierarchical model works well.