IDEAS home Printed from https://ideas.repec.org/p/pra/mprapa/42444.html
   My bibliography  Save this paper

A note on the impact evaluation of public policies: the counterfactual analysis

Author

Listed:
  • Loi, Massimo
  • Rodrigues, Margarida

Abstract

This report describes concisely, and in an intuitive way, the policy evaluation framework and the different counterfactual analysis evaluation strategies: propensity score matching, regression discontinuity design, differences-in-differences and instrumental variables. For each method we present the main assumptions it relies on and the data requirements. These methodologies apply to any type of policy and, in general, to any type of intervention. A selection of papers applying this approach in the context of labour market interventions is also included.

Suggested Citation

  • Loi, Massimo & Rodrigues, Margarida, 2012. "A note on the impact evaluation of public policies: the counterfactual analysis," MPRA Paper 42444, University Library of Munich, Germany.
    • Handle: RePEc:pra:mprapa:42444
    as

    Download full text from publisher

    File URL: https://mpra.ub.uni-muenchen.de/42444/1/MPRA_paper_42444.pdf
    File Function: original version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Alberto Abadie & Joshua Angrist & Guido Imbens, 2002. "Instrumental Variables Estimates of the Effect of Subsidized Training on the Quantiles of Trainee Earnings," Econometrica, Econometric Society, vol. 70(1), pages 91-117, January.
    2. Richard Blundell & Monica Costa Dias, 2009. "Alternative Approaches to Evaluation in Empirical Microeconomics," Journal of Human Resources, University of Wisconsin Press, vol. 44(3).
    3. Bruno Crépon & Esther Duflo & Marc Gurgand & Roland Rathelot & Philippe Zamora, 2013. "Do Labor Market Policies have Displacement Effects? Evidence from a Clustered Randomized Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 128(2), pages 531-580.
    4. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    5. Battistin, Erich & Rettore, Enrico, 2008. "Ineligibles and eligible non-participants as a double comparison group in regression-discontinuity designs," Journal of Econometrics, Elsevier, vol. 142(2), pages 715-730, February.
    6. Barbara Sianesi, 2004. "An Evaluation of the Swedish System of Active Labor Market Programs in the 1990s," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 133-155, February.
    7. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    8. Hujer, Reinhard & Caliendo, Marco, 2000. "Evaluation of Active Labour Market Policy: Methodological Concepts and Empirical Estimates," IZA Discussion Papers 236, Institute of Labor Economics (IZA).
    9. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    10. Michael Lechner, 1999. "Nonparametric bounds on employment and income effects of continuous vocational training in East Germany," Econometrics Journal, Royal Economic Society, vol. 2(1), pages 1-28.
    11. Lechner, Michael, 1999. "Earnings and Employment Effects of Continuous Off-the-Job Training in East Germany after Unification," Journal of Business & Economic Statistics, American Statistical Association, vol. 17(1), pages 74-90, January.
    12. Marco Caliendo & Reinhard Hujer & Stephan L. Thomsen, 2008. "The employment effects of job-creation schemes in Germany: A microeconometric evaluation," Advances in Econometrics, in: Modelling and Evaluating Treatment Effects in Econometrics, pages 381-428, Emerald Group Publishing Limited.
    13. Peter Z. Schochet & John Burghardt & Sheena McConnell, 2008. "Does Job Corps Work? Impact Findings from the National Job Corps Study," American Economic Review, American Economic Association, vol. 98(5), pages 1864-1886, December.
    14. Brian Bell & Richard Blundell & John Reenen, 1999. "Getting the Unemployed Back to Work: The Role of Targeted Wage Subsidies," International Tax and Public Finance, Springer;International Institute of Public Finance, vol. 6(3), pages 339-360, August.
    15. David E. Card & Pablo Ibarraran & Juan Miguel Villa, 2011. "Building in an Evaluation Component for Active Labor Market Programs: A Practitioner's Guide," SPD Working Papers 1101, Inter-American Development Bank, Office of Strategic Planning and Development Effectiveness (SPD).
    16. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 605-654.
    17. repec:mpr:mprres:6097 is not listed on IDEAS
    18. Michael Lechner, 2002. "Some practical issues in the evaluation of heterogeneous labour market programmes by matching methods," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 165(1), pages 59-82, February.
    19. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
    20. Shahidur R. Khandker & Gayatri B. Koolwal & Hussain A. Samad, . "Handbook on Impact Evaluation : Quantitative Methods and Practices," World Bank Publications, The World Bank, number 2693, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Liesbeth Colen & Sergio Gomez y Paloma & Uwe Latacz-Lohmann & Marianne Lefebvre & Raphaële Préget & Sophie Thoyer, 2016. "Economic Experiments as a Tool for Agricultural Policy Evaluation: Insights from the European CAP," Canadian Journal of Agricultural Economics/Revue canadienne d'agroeconomie, Canadian Agricultural Economics Society/Societe canadienne d'agroeconomie, vol. 64(4), pages 667-694, December.
    2. Liesbeth Colen & Sergio Gomez Y Paloma & Uwe Latacz-Lohmann & Marianne Lefebvre & Sophie Thoyer & Raphaële Préget, 2015. "(How) can economic experiments inform EU agricultural policy?," JRC Research Reports JRC97340, Joint Research Centre.
    3. Javier Castaño & Maria Blanco & Pilar Martinez, 2019. "Reviewing Counterfactual Analyses to Assess Impacts of EU Rural Development Programmes: What Lessons Can Be Learned from the 2007–2013 Ex-Post Evaluations?," Sustainability, MDPI, vol. 11(4), pages 1-22, February.
    4. Sara Pavone & Elena Ragazzi & Lisa Sella, 2015. "Sostenere le imprese agro-industriali in Piemonte: un?analisi controfattuale," SCIENZE REGIONALI, FrancoAngeli Editore, vol. 2015(3 Suppl.), pages 129-143.
    5. Brunetti, Irene & Corsini, Lorenzo, 2014. "Workplace Training Programs: Instruments for Human Capital Improvements or Screening Devices?," MPRA Paper 55943, University Library of Munich, Germany.
    6. Adejumo, O. & Okoruwa, V. & Abass, A., 2018. "Improved Post-Harvest Technology: What Impact on Nigeria s Smallholder Cassava Starch Processors Welfare?," 2018 Conference, July 28-August 2, 2018, Vancouver, British Columbia 277054, International Association of Agricultural Economists.
    7. Adenuga, Adewale H. & Jack, Claire & Ashfield, Austen & Wallace, Michael, 2021. "Estimating the impact of membership of participatory extension programme across different enterprise groups: a conditional difference-in-differences approach," 95th Annual Conference, March 29-30, 2021, Warwick, UK (Hybrid) 311082, Agricultural Economics Society - AES.
    8. Andrea Pronti & Julio Berbel, 2020. "Analysis of the impact of a volumetric tariff for irrigation in Northern Italy through the “Inverse DiD†approach," SEEDS Working Papers 1320, SEEDS, Sustainability Environmental Economics and Dynamics Studies, revised Jul 2020.
    9. Darius Erlangga & Marc Suhrcke & Shehzad Ali & Karen Bloor, 2019. "The impact of public health insurance on health care utilisation, financial protection and health status in low- and middle-income countries: A systematic review," PLOS ONE, Public Library of Science, vol. 14(8), pages 1-20, August.
    10. Jelica Rastoka & Saša Petković & Dragana Radicic, 2022. "Impact of Entrepreneurship on the Quality of Public Health Sector Institutions and Policies," IJERPH, MDPI, vol. 19(3), pages 1-25, January.
    11. Whelan, Adele & McGuinness, Seamus & Barrett, Alan, 2021. "Review of International Approaches to Evaluating Rural and Community Development Investment and Supports," Research Series, Economic and Social Research Institute (ESRI), number RS124.
    12. Nemlioglu, Ilayda & Mallick, Sushanta, 2021. "Effective innovation via better management of firms: The role of leverage in times of crisis," Research Policy, Elsevier, vol. 50(7).
    13. Dvouletý Ondřej & Čadil Jan & Mirošník Karel, 2019. "Do Firms Supported by Credit Guarantee Schemes Report Better Financial Results 2 Years After the End of Intervention?," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 19(1), pages 1-20, January.
    14. Abdulrahman Alrabiah & Steve Drew, 2020. "Proactive Management of Regulatory Policy Ripple Effects via a Computational Hierarchical Change Management Structure," Risks, MDPI, vol. 8(2), pages 1-29, May.
    15. Adewale H. Adenuga & Claire Jack & Austen Ashfield & Michael Wallace, 2021. "Assessing the Impact of Participatory Extension Programme Membership on Farm Business Performance in Northern Ireland," Agriculture, MDPI, vol. 11(10), pages 1-12, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    2. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    3. Dettmann, E. & Becker, C. & Schmeißer, C., 2011. "Distance functions for matching in small samples," Computational Statistics & Data Analysis, Elsevier, vol. 55(5), pages 1942-1960, May.
    4. Dettmann, Eva & Becker, Claudia & Schmeißer, Christian, 2010. "Is there a Superior Distance Function for Matching in Small Samples?," IWH Discussion Papers 3/2010, Halle Institute for Economic Research (IWH).
    5. Pennacchio, Luca, 2013. "The causal effect of venture capital backing on the underpricing of Italian IPOs," MPRA Paper 48695, University Library of Munich, Germany.
    6. Wendimu, Mengistu Assefa & Henningsen, Arne & Gibbon, Peter, 2016. "Sugarcane Outgrowers in Ethiopia: “Forced” to Remain Poor?," World Development, Elsevier, vol. 83(C), pages 84-97.
    7. Daniel Zaga, 2014. "The Impact of Three Mexican Nutritional Programs: The Case of Dif-Puebla," CFD Working Papers 09-2015, Centre for Finance and Development, The Graduate Institute.
    8. Michael Lechner & Ruth Miquel & Conny Wunsch, 2011. "Long‐Run Effects Of Public Sector Sponsored Training In West Germany," Journal of the European Economic Association, European Economic Association, vol. 9(4), pages 742-784, August.
    9. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2006. "Moving the Goalposts: Addressing Limited Overlap in the Estimation of Average Treatment Effects by Changing the Estimand," NBER Technical Working Papers 0330, National Bureau of Economic Research, Inc.
    10. James Alm & Yongzheng Liu, 2014. "China's Tax-for-Fee Reform and Village Inequality," Oxford Development Studies, Taylor & Francis Journals, vol. 42(1), pages 38-64, March.
    11. Winterhager, Henrik & Heinze, Anja & Spermann, Alexander, 2006. "Deregulating job placement in Europe: A microeconometric evaluation of an innovative voucher scheme in Germany," Labour Economics, Elsevier, vol. 13(4), pages 505-517, August.
    12. Markus Gangl & Thomas A. DiPrete, 2004. "Kausalanalyse durch Matchingverfahren," Discussion Papers of DIW Berlin 401, DIW Berlin, German Institute for Economic Research.
    13. Alm, Bastian & Bade, Franz-Josef, 2009. "The impact of firm subsidies: Evaluating German regional policy," EconStor Preprints 103402, ZBW - Leibniz Information Centre for Economics.
    14. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2010. "How to Control for Many Covariates? Reliable Estimators Based on the Propensity Score," IZA Discussion Papers 5268, Institute of Labor Economics (IZA).
    15. James Alm & Yongzheng Liu, 2013. "Did China's Tax-for-Fee Reform Improve Farmers' Welfare in Rural Areas?," Journal of Development Studies, Taylor & Francis Journals, vol. 49(4), pages 516-532, April.
    16. Marco Caliendo & Stefan Tübbicke, 2020. "New evidence on long-term effects of start-up subsidies: matching estimates and their robustness," Empirical Economics, Springer, vol. 59(4), pages 1605-1631, October.
    17. Kodjo Adandohoin & Vigninou Gammadigbe, 2022. "The revenue efficiency consequences of the announcement of a tax transition reform: The case of WAEMU countries," African Development Review, African Development Bank, vol. 34(S1), pages 195-218, July.
    18. Frölich, Markus & Michaelowa, Katharina, 2004. "Peer effects and textbooks in primary education: Evidence from francophone sub-Saharan Africa," HWWA Discussion Papers 311, Hamburg Institute of International Economics (HWWA).
    19. Marco Caliendo & Reinhard Hujer, 2006. "The microeconometric estimation of treatment effects—An overview," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 90(1), pages 199-215, March.
    20. Volpe Martincus, Christian & Carballo, Jerónimo, 2010. "Beyond the average effects: The distributional impacts of export promotion programs in developing countries," Journal of Development Economics, Elsevier, vol. 92(2), pages 201-214, July.

    More about this item

    Keywords

    counterfactual analysis; propensity score matching; regression discontinuity; difference in difference; instrumental variables;
    All these keywords.

    JEL classification:

    • C1 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:42444. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Joachim Winter (email available below). General contact details of provider: https://edirc.repec.org/data/vfmunde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.