IDEAS home Printed from https://ideas.repec.org/a/zbw/ifweej/201839.html
   My bibliography  Save this article

A replication recipe: List your ingredients before you start cooking

Author

Listed:
  • Chang, Andrew C.

Abstract

The author argues that researchers should do replications using preanalysis plans. These plans should specify at least three characteristics that would act as stopping rules for the replicator: (1) how much flowtime the replicator will spend, (2) how much money and effort (working hours) the replicator will spend, and (3) the intended results and the precision of the replication necessary for 'success'. A researcher's replication will be 'successful' according to context-specific criteria in the preanalysis plan. The author also argues that the two biggest drawbacks of preanalysis plans-(1) that they discount unexpected but extraordinary findings and (2) that they make it difficult for researchers to prespecify all possible actions in their decision trees-are less relevant for replications compared with new research. He concludes with describing a preanalysis plan for replicating a paper on housing demand and household formation.

Suggested Citation

  • Chang, Andrew C., 2018. "A replication recipe: List your ingredients before you start cooking," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 12, pages 1-8.
    • Handle: RePEc:zbw:ifweej:201839
    DOI: 10.5018/economics-ejournal.ja.2018-39
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.5018/economics-ejournal.ja.2018-39
    Download Restriction: no
    File URL: https://www.econstor.eu/bitstream/10419/179930/1/1025134567.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.5018/economics-ejournal.ja.2018-39?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Donald R. Haurin & Stuart S. Rosenthal, 2007. "The Influence of Household Formation on Homeownership Rates Across Time and Race," Real Estate Economics, American Real Estate and Urban Economics Association, vol. 35(4), pages 411-450, December.
    2. Lucas C. Coffman & Muriel Niederle, 2015. "Pre-analysis Plans Have Limited Upside, Especially Where Replications Are Feasible," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 81-98, Summer.
    3. David Neumark, 1999. "The Employment Effects of Recent Minimum Wage Increases: Evidence from a Pre-specified Research Design," NBER Working Papers 7171, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Benjamin D K Wood & Rui Müller & Annette N Brown, 2018. "Push button replication: Is impact evaluation evidence for international development verifiable?," PLOS ONE, Public Library of Science, vol. 13(12), pages 1-15, December.
    2. Reed, W. Robert, 2019. "Takeaways from the special issue on The Practice of Replication," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 13, pages 1-11.
    3. Sarah A. Janzen & Jeffrey D. Michler, 2021. "Ulysses' pact or Ulysses' raft: Using pre‐analysis plans in experimental and nonexperimental research," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 43(4), pages 1286-1304, December.
    4. Andrew C. Chang & Trace J. Levinson, 2023. "Raiders of the lost high‐frequency forecasts: New data and evidence on the efficiency of the Fed's forecasting," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(1), pages 88-104, January.
    5. Christophe Hurlin & Christophe Pérignon, 2020. "Reproducibility Certification in Economics Research," Working Papers hal-02896404, HAL.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chang, Andrew C., 2017. "A replication recipe: List your ingredients before you start cooking," Economics Discussion Papers 2017-74, Kiel Institute for the World Economy (IfW Kiel).
    2. Burlig, Fiona, 2018. "Improving transparency in observational social science research: A pre-analysis plan approach," Economics Letters, Elsevier, vol. 168(C), pages 56-60.
    3. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    4. Pedro Carneiro & Sokbae Lee & Daniel Wilhelm, 2020. "Optimal data collection for randomized control trials," The Econometrics Journal, Royal Economic Society, vol. 23(1), pages 1-31.
    5. Lucas C. Coffman & Muriel Niederle & Alistair J. Wilson, 2017. "A Proposal to Organize and Promote Replications," American Economic Review, American Economic Association, vol. 107(5), pages 41-45, May.
    6. Roy Chen & Yan Chen & Yohanes E. Riyanto, 2021. "Best practices in replication: a case study of common information in coordination games," Experimental Economics, Springer;Economic Science Association, vol. 24(1), pages 2-30, March.
    7. Charness, Gary & Le Bihan, Yves & Villeval, Marie Claire, 2024. "Mindfulness training, cognitive performance and stress reduction," Journal of Economic Behavior & Organization, Elsevier, vol. 217(C), pages 207-226.
    8. Bauhoff, Sebastian & Busch, Jonah, 2020. "Does deforestation increase malaria prevalence? Evidence from satellite data and health surveys," World Development, Elsevier, vol. 127(C).
    9. Kasy, Maximilian & Spiess, Jann, 2024. "Optimal Pre-analysis Plans: Statistical Decisions Subject to Implementability," IZA Discussion Papers 17187, Institute of Labor Economics (IZA).
    10. Zhou Yu & Dowell Myers, 2010. "Misleading Comparisons of Homeownership Rates when the Variable Effect of Household Formation Is Ignored: Explaining Rising Homeownership and the Homeownership Gap between Blacks and Asians in the US," Urban Studies, Urban Studies Journal Limited, vol. 47(12), pages 2615-2640, November.
    11. Weili Ding, 2020. "Laboratory experiments can pre-design to address power and selection issues," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(2), pages 125-138, December.
    12. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    13. David Byrne & David Duffy & John FitzGerald, 2018. "Household Formation and Tenure Choice: Did the Great Irish Housing Bust Alter Consumer Behaviour?," The Economic and Social Review, Economic and Social Studies, vol. 49(3), pages 287-317.
    14. Maximilian Kasy & Jann Spiess, 2024. "Optimal Pre-Analysis Plans: Statistical Decisions Subject to Implementability," CESifo Working Paper Series 11258, CESifo.
    15. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    16. Coulson, N. Edward & Dalton, Maurice, 2010. "Temporal and ethnic decompositions of homeownership rates: Synthetic cohorts across five censuses," Journal of Housing Economics, Elsevier, vol. 19(3), pages 155-166, September.
    17. Sevim, Dilek & Baranov, Victoria & Bhalotra, Sonia R. & Maselko, Joanna & Biroli, Pietro, 2023. "Socioemotional Skills in Early Childhood: Evidence from a Maternal Psychosocial Intervention," IZA Discussion Papers 15925, Institute of Labor Economics (IZA).
    18. Joel Ferguson & Rebecca Littman & Garret Christensen & Elizabeth Levy Paluck & Nicholas Swanson & Zenan Wang & Edward Miguel & David Birke & John-Henry Pezzuto, 2023. "Survey of open science practices and attitudes in the social sciences," Nature Communications, Nature, vol. 14(1), pages 1-13, December.
    19. Reed, W. Robert, 2019. "Takeaways from the special issue on The Practice of Replication," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 13, pages 1-11.
    20. Maximilian Kasy & Jann Spiess, 2022. "Rationalizing Pre-Analysis Plans:Statistical Decisions Subject to Implementability," Economics Series Working Papers 975, University of Oxford, Department of Economics.

    More about this item

    Keywords

    data and code files; household formation; housing demand; preanalysis; prespecification; publication bias; replication;
    All these keywords.

    JEL classification:

    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • R21 - Urban, Rural, Regional, Real Estate, and Transportation Economics - - Household Analysis - - - Housing Demand

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:ifweej:201839. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/iwkiede.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.