IDEAS home Printed from https://ideas.repec.org/p/ags/aaea18/274249.html
   My bibliography  Save this paper

Robustness and External Validity: What do we Learn from Repeated Study Designs over Time?

Author

Listed:
  • Akogun, Oladele
  • Adesina, Adedoyin
  • Dillon, Andrew
  • Friedman, Jed
  • Njobdi, Sani
  • Serneels, Pieter

Abstract

The replication of studies is foundational to the scientific method. But replication can have different meanings varying from results verification, to reproduction, to reanalysis with an alternative specification. Few studies are repeated in the same setting. Yet this may be key when environmental or other mediating factors change over time, as it may help establish validity over time or shed light on why impact changes over time. One example is the varied impact a health intervention can have, particularly when targeting infectious diseases, as disease incidence may vary across years. This paper presents the findings of a study that repeats the same health intervention in the same site in three different years, estimating the effect of malaria testing and treatment on agricultural worker earnings, labor supply and productivity. We find a significant impact on worker earnings across the years, but the impact size varies over time. The treatment on the treated estimates are lower in a year when the malaria prevalence rate is low. The ‘treatment on the medically untreated’, which captures an information and behavioral effect identified in an earlier study, is smaller in years when prevalence is lower and the possibility of substituting into and out of lower effort, lower return tasks is absent. These results underline the importance of changes in the prevalence rate as well as the worker’s labor constraints. The results demonstrate that repetition, apart from providing a useful tool for validation, can also help shed light on reasons why effects may vary over time.
(This abstract was borrowed from another version of this item.)

Suggested Citation

  • Akogun, Oladele & Adesina, Adedoyin & Dillon, Andrew & Friedman, Jed & Njobdi, Sani & Serneels, Pieter, 2018. "Robustness and External Validity: What do we Learn from Repeated Study Designs over Time?," 2018 Annual Meeting, August 5-7, Washington, D.C. 274249, Agricultural and Applied Economics Association.
  • Handle: RePEc:ags:aaea18:274249
    DOI: 10.22004/ag.econ.274249
    as

    Download full text from publisher

    File URL: https://ageconsearch.umn.edu/record/274249/files/Abstracts_18_05_23_15_27_15_33__68_48_166_174_0.pdf
    Download Restriction: no

    File URL: https://libkey.io/10.22004/ag.econ.274249?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Carlos A. Flores & Oscar A. Mitnik, 2013. "Comparing Treatments across Labor Markets: An Assessment of Nonexperimental Multiple-Treatment Strategies," The Review of Economics and Statistics, MIT Press, vol. 95(5), pages 1691-1707, December.
    2. Ashis Das & Jed Friedman & Eeshani Kandpal, 2018. "Does involvement of local NGOs enhance public service delivery? Cautionary evidence from a malaria‐prevention program in India," Health Economics, John Wiley & Sons, Ltd., vol. 27(1), pages 172-188, January.
    3. Dillon, Andrew & Friedman, Jed & Serneels, Pieter, 2014. "Health Information, Treatment, and Worker Productivity: Experimental Evidence from Malaria Testing and Treatment among Nigerian Sugarcane Cutters," IZA Discussion Papers 8074, Institute of Labor Economics (IZA).
    4. repec:pri:cheawb:deaton%20instruments%20of%20development%20keynes%20lecture%202009 is not listed on IDEAS
    5. Angus Deaton, 2009. "Instruments of development: Randomization in the tropics, and the search for the elusive keys to economic development," Working Papers 1128, Princeton University, Woodrow Wilson School of Public and International Affairs, Center for Health and Wellbeing..
    6. Andrew Dillon & Jed Friedman & Pieter Serneels, 0. "Health Information, Treatment, and Worker Productivity," Journal of the European Economic Association, European Economic Association, vol. 19(2), pages 1077-1115.
    7. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    8. Tessa Bold & Mwangi Kimenyi & Germano Mwabu & Alice Ng'ang'a & Justin Sandefur, 2013. "Scaling-up What Works: Experimental Evidence on External Validity in Kenyan Education," CSAE Working Paper Series 2013-04, Centre for the Study of African Economies, University of Oxford.
    9. Hunt Allcott, 2015. "Site Selection Bias in Program Evaluation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 130(3), pages 1117-1165.
    10. Ahmed, Akhter U. & Bouis, Howarth E., 2002. "Weighing what's practical: proxy means tests for targeting food subsidies in Egypt," Food Policy, Elsevier, vol. 27(5-6), pages 519-540.
    11. Grosh, M.E. & Baker, J.L., 1995. "Proxy Means Tests for Targetting Social Programs. Simulations and Speculation," Papers 118, World Bank - Living Standards Measurement.
    12. Andrew C. Chang & Phillip Li, 2015. "Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say \"Usually Not\"," Finance and Economics Discussion Series 2015-83, Board of Governors of the Federal Reserve System (U.S.).
    13. repec:pri:cheawb:deaton%20instruments%20of%20development%20keynes%20lecture%202009.pdf is not listed on IDEAS
    14. Chima, Reginald Ikechukwu & Goodman, Catherine A. & Mills, Anne, 2003. "The economic impact of malaria in Africa: a critical review of the evidence," Health Policy, Elsevier, vol. 63(1), pages 17-36, January.
    15. Elizabeth A. Stuart & Stephen R. Cole & Catherine P. Bradshaw & Philip J. Leaf, 2011. "The use of propensity scores to assess the generalizability of results from randomized trials," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(2), pages 369-386, April.
    16. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    17. repec:pri:rpdevs:instruments_of_development.pdf is not listed on IDEAS
    18. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 241-270.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank, vol. 33(1), pages 34-64.
    2. Akogun, Oladele & Dillon, Andrew & Friedman, Jed & Prasann, Ashesh & Serneels, Pieter, 2017. "Productivity and Health: Alternative Productivity Measures using Physical Activity," 2017 Annual Meeting, July 30-August 1, Chicago, Illinois 258380, Agricultural and Applied Economics Association.
    3. Omar Al-Ubaydli & John List & Claire Mackevicius & Min Sok Lee & Dana Suskind, 2019. "How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling," Artefactual Field Experiments 00679, The Field Experiments Website.
    4. Akogun, Oladele & Dillon, Andrew & Friedman, Jed & Prasann, Ashesh & Serneels, Pieter, 2017. "Productivity and Health: Alternative Productivity Estimates Using Physical Activity," IZA Discussion Papers 11115, Institute of Labor Economics (IZA).
    5. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," SciencePo Working papers Main hal-03455978, HAL.
    6. Hunt Allcott, 2012. "Site Selection Bias in Program Evaluation," NBER Working Papers 18373, National Bureau of Economic Research, Inc.
    7. Gauri, Varun & Jamison, Julian C. & Mazar, Nina & Ozier, Owen, 2019. "Motivating Bureaucrats through Social Recognition: External Validity — A Tale of Two States," IZA Discussion Papers 12251, Institute of Labor Economics (IZA).
    8. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    9. Sebastian Galiani & Patrick J. McEwan & Brian Quistorff, 2017. "External and Internal Validity of a Geographic Quasi-Experiment Embedded in a Cluster-Randomized Experiment," Advances in Econometrics, in: Regression Discontinuity Designs, volume 38, pages 195-236, Emerald Group Publishing Limited.
    10. Gauri, Varun & Jamison, Julian C. & Mazar, Nina & Ozier, Owen, 2021. "Motivating bureaucrats through social recognition: External validity—A tale of two states," Organizational Behavior and Human Decision Processes, Elsevier, vol. 163(C), pages 117-131.
    11. Ashis Das & Jed Friedman & Eeshani Kandpal, 2018. "Does involvement of local NGOs enhance public service delivery? Cautionary evidence from a malaria‐prevention program in India," Health Economics, John Wiley & Sons, Ltd., vol. 27(1), pages 172-188, January.
    12. Cohen, Jessica & Saran, Indrani, 2018. "The impact of packaging and messaging on adherence to malaria treatment: Evidence from a randomized controlled trial in Uganda," Journal of Development Economics, Elsevier, vol. 134(C), pages 68-95.
    13. Benjamin A. Olken, 2020. "Banerjee, Duflo, Kremer, and the Rise of Modern Development Economics," Scandinavian Journal of Economics, Wiley Blackwell, vol. 122(3), pages 853-878, July.
    14. Benjamin Lu & Eli Ben-Michael & Avi Feller & Luke Miratrix, 2023. "Is It Who You Are or Where You Are? Accounting for Compositional Differences in Cross-Site Treatment Effect Variation," Journal of Educational and Behavioral Statistics, , vol. 48(4), pages 420-453, August.
    15. Bernard, Tanguy & Hidrobo, Melissa & Le Port, Agnès & Rawat, Rahul, 2017. "Nutrition incentives in dairy contract farming in northern Senegal," IFPRI discussion papers 1629, International Food Policy Research Institute (IFPRI).
    16. Hensel, Przemysław G., 2021. "Reproducibility and replicability crisis: How management compares to psychology and economics – A systematic review of literature," European Management Journal, Elsevier, vol. 39(5), pages 577-594.
    17. Florent Bedecarrats & Isabelle Guérin & François Roubaud, 2017. "L'étalon-or des évaluations randomisées : du discours de la méthode à l'économie politique," Working Papers ird-01445209, HAL.
    18. Das, Ashis & Friedman, Jed & Kandpal, Eeshani, 2014. "Does involvement of local NGOs enhance public service delivery ? cautionary evidence from a Malaria-prevention evaluation in India," Policy Research Working Paper Series 6931, The World Bank.
    19. Barrera-Osorio, Felipe & Raju, Dhushyanth, 2017. "Teacher performance pay: Experimental evidence from Pakistan," Journal of Public Economics, Elsevier, vol. 148(C), pages 75-91.
    20. Peters, Jörg & Langbein, Jörg & Roberts, Gareth, 2016. "Policy evaluation, randomized controlled trials, and external validity—A systematic review," Economics Letters, Elsevier, vol. 147(C), pages 51-54.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ags:aaea18:274249. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: AgEcon Search (email available below). General contact details of provider: https://edirc.repec.org/data/aaeaaea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.