IDEAS home Printed from https://ideas.repec.org/a/zbw/ifweej/201839.html
   My bibliography  Save this article

A replication recipe: List your ingredients before you start cooking

Author

Listed:
  • Chang, Andrew C.

Abstract

The author argues that researchers should do replications using preanalysis plans. These plans should specify at least three characteristics that would act as stopping rules for the replicator: (1) how much flowtime the replicator will spend, (2) how much money and effort (working hours) the replicator will spend, and (3) the intended results and the precision of the replication necessary for 'success'. A researcher's replication will be 'successful' according to context-specific criteria in the preanalysis plan. The author also argues that the two biggest drawbacks of preanalysis plans-(1) that they discount unexpected but extraordinary findings and (2) that they make it difficult for researchers to prespecify all possible actions in their decision trees-are less relevant for replications compared with new research. He concludes with describing a preanalysis plan for replicating a paper on housing demand and household formation.

Suggested Citation

  • Chang, Andrew C., 2018. "A replication recipe: List your ingredients before you start cooking," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 12, pages 1-8.
    • Handle: RePEc:zbw:ifweej:201839
    DOI: 10.5018/economics-ejournal.ja.2018-39
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.5018/economics-ejournal.ja.2018-39
    Download Restriction: no
    File URL: https://www.econstor.eu/bitstream/10419/179930/1/1025134567.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.5018/economics-ejournal.ja.2018-39?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Donald R. Haurin & Stuart S. Rosenthal, 2007. "The Influence of Household Formation on Homeownership Rates Across Time and Race," Real Estate Economics, American Real Estate and Urban Economics Association, vol. 35(4), pages 411-450, December.
    2. Lucas C. Coffman & Muriel Niederle, 2015. "Pre-analysis Plans Have Limited Upside, Especially Where Replications Are Feasible," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 81-98, Summer.
    3. David Neumark, 1999. "The Employment Effects of Recent Minimum Wage Increases: Evidence from a Pre-specified Research Design," NBER Working Papers 7171, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Benjamin D K Wood & Rui Müller & Annette N Brown, 2018. "Push button replication: Is impact evaluation evidence for international development verifiable?," PLOS ONE, Public Library of Science, vol. 13(12), pages 1-15, December.
    2. Andrew C. Chang & Trace J. Levinson, 2023. "Raiders of the lost high‐frequency forecasts: New data and evidence on the efficiency of the Fed's forecasting," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(1), pages 88-104, January.
    3. Reed, W. Robert, 2019. "Takeaways from the special issue on The Practice of Replication," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 13, pages 1-11.
    4. Colliard, Jean-Edouard & Hurlin, Christophe & Pérignon, Christophe, 2019. "Reproducibility Certification in Economics Research," HEC Research Papers Series 1345, HEC Paris.
    5. Sarah A. Janzen & Jeffrey D. Michler, 2021. "Ulysses' pact or Ulysses' raft: Using pre‐analysis plans in experimental and nonexperimental research," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 43(4), pages 1286-1304, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chang, Andrew C., 2017. "A replication recipe: List your ingredients before you start cooking," Economics Discussion Papers 2017-74, Kiel Institute for the World Economy (IfW Kiel).
    2. Burlig, Fiona, 2018. "Improving transparency in observational social science research: A pre-analysis plan approach," Economics Letters, Elsevier, vol. 168(C), pages 56-60.
    3. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    4. Charness, Gary & Le Bihan, Yves & Villeval, Marie Claire, 2024. "Mindfulness training, cognitive performance and stress reduction," Journal of Economic Behavior & Organization, Elsevier, vol. 217(C), pages 207-226.
    5. Pedro Carneiro & Sokbae Lee & Daniel Wilhelm, 2020. "Optimal data collection for randomized control trials," The Econometrics Journal, Royal Economic Society, vol. 23(1), pages 1-31.
    6. Bauhoff, Sebastian & Busch, Jonah, 2020. "Does deforestation increase malaria prevalence? Evidence from satellite data and health surveys," World Development, Elsevier, vol. 127(C).
    7. Zhou Yu & Dowell Myers, 2010. "Misleading Comparisons of Homeownership Rates when the Variable Effect of Household Formation Is Ignored: Explaining Rising Homeownership and the Homeownership Gap between Blacks and Asians in the US," Urban Studies, Urban Studies Journal Limited, vol. 47(12), pages 2615-2640, November.
    8. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    9. Maximilian Kasy & Jann Spiess, 2022. "Optimal Pre-Analysis Plans: Statistical Decisions Subject to Implementability," Papers 2208.09638, arXiv.org, revised Jul 2024.
    10. Joel Ferguson & Rebecca Littman & Garret Christensen & Elizabeth Levy Paluck & Nicholas Swanson & Zenan Wang & Edward Miguel & David Birke & John-Henry Pezzuto, 2023. "Survey of open science practices and attitudes in the social sciences," Nature Communications, Nature, vol. 14(1), pages 1-13, December.
    11. Reed, W. Robert, 2019. "Takeaways from the special issue on The Practice of Replication," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 13, pages 1-11.
    12. Maximilian Kasy & Jann Spiess, 2022. "Rationalizing Pre-Analysis Plans:Statistical Decisions Subject to Implementability," Economics Series Working Papers 975, University of Oxford, Department of Economics.
    13. Brown, Martin & Hentschel, Nicole & Mettler, Hannes & Stix, Helmut, 2022. "The convenience of electronic payments and consumer cash demand," Journal of Monetary Economics, Elsevier, vol. 130(C), pages 86-102.
    14. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    15. Luigi Butera & Philip Grossman & Daniel Houser & John List & Marie-Claire Villeval, 2020. "A New Mechanism to Alleviate the Crises of Confidence in Science - With an Application to the Public Goods Game," Artefactual Field Experiments 00684, The Field Experiments Website.
    16. Jan Jozwik, 2024. "Eliciting risk preferences in an artefactual field experiment via replication and an alternative approach," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 10(2), pages 442-456, December.
    17. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-01651080, HAL.
    18. Rebecca M. Blank & David T. Ellwood, 2001. "The Clinton Legacy for America's Poor," NBER Working Papers 8437, National Bureau of Economic Research, Inc.
    19. Dongshin Kim & Youngme Seo & Julia Freybote, 2021. "Urbanity, Financial Crisis and the Timing of Homebuying Decisions by Young Households," The Journal of Real Estate Finance and Economics, Springer, vol. 62(4), pages 481-507, May.
    20. Lars Ivar Oppedal Berge & Kjetil Bjorvatn & Simon Galle & Edward Miguel & Daniel N. Posner & Bertil Tungodden & Kelly Zhang, 2015. "How Strong are Ethnic Preferences?," NBER Working Papers 21715, National Bureau of Economic Research, Inc.

    More about this item

    Keywords

    data and code files; household formation; housing demand; preanalysis; prespecification; publication bias; replication;
    All these keywords.

    JEL classification:

    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • R21 - Urban, Rural, Regional, Real Estate, and Transportation Economics - - Household Analysis - - - Housing Demand

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:ifweej:201839. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/iwkiede.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.