IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v33y2009i3p281-306.html
   My bibliography  Save this article

Old Wine in New Skins: The Sensitivity of Established Findings to New Methods

Author

Listed:
  • E. Michael Foster

    (University of North Carolina at Chapel Hill, emfoster@unc.edu)

  • Elizabeth Wiley-Exley

    (University of North Carolina at Chapel Hill)

  • Leonard Bickman

    (Vanderbilt University)

Abstract

Findings from an evaluation of a model system for delivering mental health services to youth were reassessed to determine the robustness of key findings to the use of methodologies unavailable to the original analysts. These analyses address a key concern about earlier findings—that the quasi-experimental design involved the comparison of two noncomparable groups. The authors employed propensity score methodology to reconsider between-group baseline differences in observed characteristics of participating families. The authors also considered the possible effect of unobserved between-group differences. The data support previous studies that show few differences in outcomes, but the findings are sensitive to unobserved heterogeneity.

Suggested Citation

  • E. Michael Foster & Elizabeth Wiley-Exley & Leonard Bickman, 2009. "Old Wine in New Skins: The Sensitivity of Established Findings to New Methods," Evaluation Review, , vol. 33(3), pages 281-306, June.
    • Handle: RePEc:sae:evarev:v:33:y:2009:i:3:p:281-306
    DOI: 10.1177/0193841X09334028
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X09334028
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X09334028?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 605-654.
    2. Verbeek, Marno & Nijman, Theo, 1992. "Testing for Selectivity Bias in Panel Data Models," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 33(3), pages 681-703, August.
    3. Lee, Myoung-jae, 2005. "Micro-Econometrics for Policy, Program and Treatment Effects," OUP Catalogue, Oxford University Press, number 9780199267699.
    4. James J. Heckman & Vytlacil, Edward J., 2007. "Econometric Evaluation of Social Programs, Part I: Causal Models, Structural Models and Econometric Policy Evaluation," Handbook of Econometrics, in: J.J. Heckman & E.E. Leamer (ed.), Handbook of Econometrics, edition 1, volume 6, chapter 70, Elsevier.
    5. Donald B. Rubin, 2005. "Causal Inference Using Potential Outcomes: Design, Modeling, Decisions," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 322-331, March.
    6. Nijman, T.E. & Verbeek, M.J.C.M., 1992. "Testing for selectivity in panel data models," Other publications TiSEM 7ec34a6c-1d84-4052-971c-d, Tilburg University, School of Economics and Management.
    7. Hahn, Jinyong & Todd, Petra & Van der Klaauw, Wilbert, 2001. "Identification and Estimation of Treatment Effects with a Regression-Discontinuity Design," Econometrica, Econometric Society, vol. 69(1), pages 201-209, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    2. Haiyang Lu & Peng Nie & Alfonso Sousa-Poza, 2021. "The Effect of Parental Educational Expectations on Adolescent Subjective Well-Being and the Moderating Role of Perceived Academic Pressure: Longitudinal Evidence for China," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 14(1), pages 117-137, February.
    3. Foster, E. Michael & McCombs-Thornton, Kimberly, 2013. "Child welfare and the challenge of causal inference," Children and Youth Services Review, Elsevier, vol. 35(7), pages 1130-1142.
    4. Jiang, Miao & Foster, E. Michael & Gibson-Davis, Christina M., 2010. "The effect of WIC on breastfeeding: A new look at an established relationship," Children and Youth Services Review, Elsevier, vol. 32(2), pages 264-273, February.
    5. E. Santarelli & H. T. Tran, 2013. "Diversification Strategies and Firm Performance: A Sample Selection Approach," Working Papers wp896, Dipartimento Scienze Economiche, Universita' di Bologna.
    6. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    7. Nicoletti, Cheti, 2006. "Nonresponse in dynamic panel data models," Journal of Econometrics, Elsevier, vol. 132(2), pages 461-489, June.
    8. James J. Heckman, 1991. "Randomization and Social Policy Evaluation Revisited," NBER Technical Working Papers 0107, National Bureau of Economic Research, Inc.
    9. Jan Fałkowski & Maciej Jakubowski & Paweł Strawiński, 2014. "Returns from income strategies in rural Poland," The Economics of Transition, The European Bank for Reconstruction and Development, vol. 22(1), pages 139-178, January.
    10. Magnus, Jan R. & Melenberg, Bertrand & Muris, Chris, 2011. "Global Warming and Local Dimming: The Statistical Evidence," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 452-464.
    11. Lionel WILNER, 2019. "The Dynamics of Individual Happiness," Working Papers 2019-18, Center for Research in Economics and Statistics.
    12. Verbeek, M.J.C.M. & Nijman, T.E., 1992. "Incomplete panels and selection bias : A survey," Discussion Paper 1992-7, Tilburg University, Center for Economic Research.
    13. Patricia Cubí‐Mollá & Mireia Jofre‐Bonet & Victoria Serra‐Sastre, 2017. "Adaptation to health states: Sick yet better off?," Health Economics, John Wiley & Sons, Ltd., vol. 26(12), pages 1826-1843, December.
    14. Keisuke Hirano & Guido W. Imbens & Geert Ridder & Donald B. Rubin, 2001. "Combining Panel Data Sets with Attrition and Refreshment Samples," Econometrica, Econometric Society, vol. 69(6), pages 1645-1659, November.
    15. Shabbar Jaffry & Yaseen Ghulam & Vyoma Shah, 2007. "Returns to Education in Pakistan," The Pakistan Development Review, Pakistan Institute of Development Economics, vol. 46(4), pages 833-852.
    16. Paweł Strawiński, 2013. "Controlling for overlap in matching," Working Papers 2013-10, Faculty of Economic Sciences, University of Warsaw.
    17. Aïssata COULIBALY & Urbain Thierry YOGO, 2016. "Access to Financial Services and Working Poverty in Developing Countries," Working Papers 201620, CERDI.
    18. E. Michael Foster & Grace Y. Fang, 2004. "Alternative Methods for Handling Attrition," Evaluation Review, , vol. 28(5), pages 434-464, October.
    19. E. Michael Foster & Leonard Bickman, 1996. "An Evaluator's Guide To Detecting Attrition Problems," Evaluation Review, , vol. 20(6), pages 695-723, December.
    20. Giambona, Erasmo & Golec, Joseph, 2010. "Strategic trading in the wrong direction by a large institutional insider," Journal of Empirical Finance, Elsevier, vol. 17(1), pages 1-22, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:33:y:2009:i:3:p:281-306. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.