Scientific Inference from Field and Laboratory Economic Experiments: Empirical Evidence

RField experiments can help improve scientific inference by providing access to diverse samples that are representative in terms of demographic backgrounds, and by availing the use of assets that relate directly to the economic problem of interest. We present a study comparing claims based on laboratory and field experiments in 520 publications in 2018 and 2019 at leading general and field journals in economics. Each paper is surveyed for their key claims and matches along the dimensions of profession, age, and gender of experimental subjects; country of experiment; and experimental asset in relation to which a claim is made. We find that, particularly in the realm of policy testing, field experiments are more likely to match the key claims than laboratory experiments. However, depending on the dimension, less than 20% or only up to around 65% of field experiments including natural field experiments achieve a match. Around four out of five field experiments fail to match in at least three out of the five dimensions. We conclude that the methodological challenge of generalizing results beyond what is within the domain of the experiments themselves also applies to many papers based on field experiments, given the claims being made. In addition, we find that publications by top 20 institutions authors or with experiments conducted in Caucasian-majority countries have a substantially higher likelihood of wide generalizations.