IDEAS home Printed from https://ideas.repec.org/a/aea/apandp/v110y2020p75-79.html
   My bibliography  Save this article

Forecasting the Results of Experiments: Piloting an Elicitation Strategy

Author

Listed:
  • Stefano DellaVigna
  • Nicholas Otis
  • Eva Vivalt

Abstract

Forecasts of experimental results can clarify the interpretation of research results, mitigate publication bias, and improve experimental designs. We collect forecasts of the results of three Registered Reports preliminarily accepted to the Journal of Development Economics, randomly varying four features: (1) small versus large reference values, (2) whether predictions are in raw units or standard deviations, (3) text-entry versus slider responses, and (4) small versus large slider bounds. Forecasts are generally robust to elicitation features, though wider slider bounds are associated with higher forecasts throughout the forecast distribution. We make preliminary recommendations on how many forecasts should be gathered.

Suggested Citation

  • Stefano DellaVigna & Nicholas Otis & Eva Vivalt, 2020. "Forecasting the Results of Experiments: Piloting an Elicitation Strategy," AEA Papers and Proceedings, American Economic Association, vol. 110, pages 75-79, May.
    • Handle: RePEc:aea:apandp:v:110:y:2020:p:75-79
    DOI: 10.1257/pandp.20201080
    as

    Download full text from publisher

    File URL: https://www.aeaweb.org/doi/10.1257/pandp.20201080
    Download Restriction: no
    File URL: http://doi.org/10.3886/E117601V1
    Download Restriction: no
    File URL: https://www.aeaweb.org/doi/10.1257/pandp.20201080.appx
    Download Restriction: no
    File URL: https://www.aeaweb.org/doi/10.1257/pandp.20201080.ds
    Download Restriction: Access to full text is restricted to AEA members and institutional subscribers.
    File URL: https://libkey.io/10.1257/pandp.20201080?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Stefano DellaVigna & Nicholas Otis & Eva Vivalt, 2020. "Forecasting the Results of Experiments: Piloting an Elicitation Strategy," AEA Papers and Proceedings, American Economic Association, vol. 110, pages 75-79, May.
    2. Stefano DellaVigna & Devin Pope, 2018. "Predicting Experimental Results: Who Knows What?," Journal of Political Economy, University of Chicago Press, vol. 126(6), pages 2410-2456.
    3. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Dirk Bergemann & Marco Ottaviani, 2021. "Information Markets and Nonmarkets," Cowles Foundation Discussion Papers 2296, Cowles Foundation for Research in Economics, Yale University.
    2. Stefano DellaVigna & Nicholas Otis & Eva Vivalt, 2020. "Forecasting the Results of Experiments: Piloting an Elicitation Strategy," AEA Papers and Proceedings, American Economic Association, vol. 110, pages 75-79, May.
    3. Chadimová, Kateřina & Cahlíková, Jana & Cingl, Lubomír, 2022. "Foretelling what makes people pay: Predicting the results of field experiments on TV fee enforcement," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    4. Frederico Finan & Demian Pouzo, 2021. "Reinforcing RCTs with Multiple Priors while Learning about External Validity," Papers 2112.09170, arXiv.org, revised Mar 2023.
    5. Bah, Tijan L. & Batista, Catia & Gubert, Flore & McKenzie, David, 2023. "Can information and alternatives to irregular migration reduce “backway” migration from The Gambia?," Journal of Development Economics, Elsevier, vol. 165(C).
    6. Henning Schaak & Jens Rommel & Julian Sagebiel & Jesus Barreiro-Hurlé & Douadia Bougherara & Luigi Cemablo & Marija Cerjak & Tajana Čop & Mikołaj Czajkowski & María Espinosa-Goded & Julia Höhler & Car, 2022. "How Well Can Experts Predict Farmers' Risk Preferences ?," Post-Print hal-03738351, HAL.
    7. Olckers, Matthew, 2021. "On track for retirement?," Journal of Economic Behavior & Organization, Elsevier, vol. 190(C), pages 76-88.
    8. Yang, Dean & Allen, James & Mahumane, Arlete & Riddell, James & Yu, Hang, 2023. "Knowledge, stigma, and HIV testing: An analysis of a widespread HIV/AIDS program," Journal of Development Economics, Elsevier, vol. 160(C).
    9. Del Carmen,Giselle & Espinal Hernandez,Edgardo Enrique & De Gouvea Scot De Arruda,Thiago, 2022. "Targeting in Tax Compliance Interventions : Experimental Evidence from Honduras," Policy Research Working Paper Series 9967, The World Bank.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chadimová, Kateřina & Cahlíková, Jana & Cingl, Lubomír, 2022. "Foretelling what makes people pay: Predicting the results of field experiments on TV fee enforcement," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    2. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    3. Bergemann, Dirk & Ottaviani, Marco, 2021. "Information Markets and Nonmarkets," CEPR Discussion Papers 16459, C.E.P.R. Discussion Papers.
    4. Stefano DellaVigna & Devin Pope, 2022. "Stability of Experimental Results: Forecasts and Evidence," American Economic Journal: Microeconomics, American Economic Association, vol. 14(3), pages 889-925, August.
    5. Tierney, Warren & Hardy, Jay H. & Ebersole, Charles R. & Leavitt, Keith & Viganola, Domenico & Clemente, Elena Giulia & Gordon, Michael & Dreber, Anna & Johannesson, Magnus & Pfeiffer, Thomas & Uhlman, 2020. "Creative destruction in science," Organizational Behavior and Human Decision Processes, Elsevier, vol. 161(C), pages 291-309.
    6. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    7. Frederico Finan & Demian Pouzo, 2021. "Reinforcing RCTs with Multiple Priors while Learning about External Validity," Papers 2112.09170, arXiv.org, revised Mar 2023.
    8. Schaerer, Michael & du Plessis, Christilene & Nguyen, My Hoang Bao & van Aert, Robbie C.M. & Tiokhin, Leo & Lakens, Daniël & Giulia Clemente, Elena & Pfeiffer, Thomas & Dreber, Anna & Johannesson, Mag, 2023. "On the trajectory of discrimination: A meta-analysis and forecasting survey capturing 44 years of field experiments on gender and hiring decisions," Organizational Behavior and Human Decision Processes, Elsevier, vol. 179(C).
    9. Merl, Robert & Stöckl, Thomas & Palan, Stefan, 2023. "Insider trading regulation and shorting constraints. Evaluating the joint effects of two market interventions," Journal of Banking & Finance, Elsevier, vol. 154(C).
    10. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    11. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 48(1), pages 62-83.
    12. Chávez, Carlos A. & Murphy, James J. & Stranlund, John K., 2018. "Managing and defending the commons: Experimental evidence from TURFs in Chile," Journal of Environmental Economics and Management, Elsevier, vol. 91(C), pages 229-246.
    13. Stefano Della & Jörg Heining & Johannes F Schmieder & Simon Trenkle, 2023. "Evidence on Job Search Models from a Survey of Unemployed Workers in Germany," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 137(2), pages 1181-1232.
    14. Erik Snowberg & Leeat Yariv, 2018. "Testing the Waters: Behavior across Participant Pools," CESifo Working Paper Series 7136, CESifo.
    15. Christina Korting & Carl Lieberman & Jordan Matsudaira & Zhuan Pei & Yi Shen, 2023. "Visual Inference and Graphical Representation in Regression Discontinuity Designs," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(3), pages 1977-2019.
    16. Marcella Alsan & Sarah Eichmeyer, 2024. "Experimental Evidence on the Effectiveness of Nonexperts for Improving Vaccine Demand," American Economic Journal: Economic Policy, American Economic Association, vol. 16(1), pages 394-414, February.
    17. Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Do economists replicate?," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 219-232.
    18. Mohammad H. Sepahvand & Roujman Shahbazian, 2021. "Intergenerational transmission of risk attitudes in Burkina Faso," Empirical Economics, Springer, vol. 61(1), pages 503-527, July.
    19. Alexander Frankel & Maximilian Kasy, 2022. "Which Findings Should Be Published?," American Economic Journal: Microeconomics, American Economic Association, vol. 14(1), pages 1-38, February.
    20. D’Exelle, Ben & Gutekunst, Christine & Riedl, Arno, 2023. "The effect of gender and gender pairing on bargaining: Evidence from an artefactual field experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 205(C), pages 237-269.

    More about this item

    JEL classification:

    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • O10 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:aea:apandp:v:110:y:2020:p:75-79. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Michael P. Albert (email available below). General contact details of provider: https://edirc.repec.org/data/aeaaaea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.