IDEAS home Printed from https://ideas.repec.org/a/gam/jsusta/v11y2019i4p1105-d207509.html
   My bibliography  Save this article

Reviewing Counterfactual Analyses to Assess Impacts of EU Rural Development Programmes: What Lessons Can Be Learned from the 2007–2013 Ex-Post Evaluations?

Author

Listed:
  • Javier Castaño

    (Department of Agricultural Economics, Universidad Politécnica de Madrid, ETSIAAB, Avda. Puerta Hierro 2, 28040 Madrid, Spain
    CEIGRAM, Universidad Politécnica de Madrid, Calle Senda del Rey 13, 28040 Madrid, Spain)

  • Maria Blanco

    (Department of Agricultural Economics, Universidad Politécnica de Madrid, ETSIAAB, Avda. Puerta Hierro 2, 28040 Madrid, Spain
    CEIGRAM, Universidad Politécnica de Madrid, Calle Senda del Rey 13, 28040 Madrid, Spain)

  • Pilar Martinez

    (Department of Agricultural Economics, Universidad Politécnica de Madrid, ETSIAAB, Avda. Puerta Hierro 2, 28040 Madrid, Spain
    CEIGRAM, Universidad Politécnica de Madrid, Calle Senda del Rey 13, 28040 Madrid, Spain)

Abstract

Counterfactual analysis has been recommended as a means of assessing the impacts of European Rural Development Programmes (RDP) over recent years, although its application has been scarce to date. This paper examines the use of counterfactual analysis to assess socioeconomic impacts in a set of 2007–2013 ex-post evaluations. The analysis undertaken shows that a wide variety of counterfactual approaches have been applied, although certain barriers still remain to address the estimation of RDP impacts following the EU evaluation standards. Furthermore, we noted that impacts provided by individual RDP evaluations may hardly be aggregated, making it difficult to draw clear conclusions about the effectiveness of rural development policy at the EU level.

Suggested Citation

  • Javier Castaño & Maria Blanco & Pilar Martinez, 2019. "Reviewing Counterfactual Analyses to Assess Impacts of EU Rural Development Programmes: What Lessons Can Be Learned from the 2007–2013 Ex-Post Evaluations?," Sustainability, MDPI, vol. 11(4), pages 1-22, February.
    • Handle: RePEc:gam:jsusta:v:11:y:2019:i:4:p:1105-:d:207509
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2071-1050/11/4/1105/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2071-1050/11/4/1105/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Linda Arata & Paolo Sckokai, 2016. "The Impact of Agri-environmental Schemes on Farm Performance in Five E.U. Member States: A DID-Matching Approach," Land Economics, University of Wisconsin Press, vol. 92(1), pages 167-186.
    2. Kuhfuss, Laure & Subervie, Julie, 2018. "Do European Agri-environment Measures Help Reduce Herbicide Use? Evidence From Viticulture in France," Ecological Economics, Elsevier, vol. 149(C), pages 202-211.
    3. Andrea Pufahl & Christoph R. Weiss, 2009. "Evaluating the effects of farm programmes: results from propensity score matching," European Review of Agricultural Economics, Oxford University Press and the European Agricultural and Applied Economics Publications Foundation, vol. 36(1), pages 79-101, March.
    4. Imbens, Guido W. & Lemieux, Thomas, 2008. "Regression discontinuity designs: A guide to practice," Journal of Econometrics, Elsevier, vol. 142(2), pages 615-635, February.
    5. Eleni PAPADOPOULOU & Christos PAPALEXIOU & Nikolaos HASANAGAS, 2012. "Participatory Evaluation Of Rural Development Programmes: A Qualitative Approach In The Case Of Modernisation Of Agricultural Holdings In Greece," Regional Science Inquiry, Hellenic Association of Regional Scientists, vol. 0(1), pages 81-94, June.
    6. Carolyn Heinrich & Alessandro Maffioli & Gonzalo Vázquez, 2010. "A Primer for Applying Propensity-Score Matching," SPD Working Papers 1005, Inter-American Development Bank, Office of Strategic Planning and Development Effectiveness (SPD).
    7. Jerzy Michalek, 2012. "Counterfactual impact evaluation of EU rural development programmes - Propensity Score Matching methodology applied to selected EU Member States. Volume 2: A regional approach," JRC Research Reports JRC72060, Joint Research Centre.
    8. Lilli Aline Schroeder & Alexander Gocht & Wolfgang Britz, 2015. "The Impact of Pillar II Funding: Validation from a Modelling and Evaluation Perspective," Journal of Agricultural Economics, Wiley Blackwell, vol. 66(2), pages 415-441, June.
    9. Liesbeth Colen & Sergio Gomez y Paloma & Uwe Latacz-Lohmann & Marianne Lefebvre & Raphaële Préget & Sophie Thoyer, 2016. "Economic Experiments as a Tool for Agricultural Policy Evaluation: Insights from the European CAP," Canadian Journal of Agricultural Economics/Revue canadienne d'agroeconomie, Canadian Agricultural Economics Society/Societe canadienne d'agroeconomie, vol. 64(4), pages 667-694, December.
    10. Mara Giua, 2014. "Spatial Discontinuity for the Impact Assessment of the EU Regional Policy. The Case of Italian Objective 1 Regions," Departmental Working Papers of Economics - University 'Roma Tre' 0197, Department of Economics - University Roma Tre.
    11. Medonos, Tomas & Ratinger, Tomas & Hruska, Martin & Spicka, Jindrich, 2012. "The Assessment of the Effects of Investment Support Measures of the Rural Development Programmes: the Case of the Czech Republic," AGRIS on-line Papers in Economics and Informatics, Czech University of Life Sciences Prague, Faculty of Economics and Management, vol. 4(4), pages 1-14, December.
    12. Jerzy Michalek & Pavel Ciaian & D'Artis Kancs, 2016. "Investment Crowding Out: Firm-Level Evidence from Northern Germany," Regional Studies, Taylor & Francis Journals, vol. 50(9), pages 1579-1594, September.
    13. López Santiago, Luís Antonio & Monsalve, Fabio & Zafrilla, Jorge Enrique, 2011. "Análisis Input-Output de la eficacia de la Política Europea de Desarrollo Rural 2007-2013. Propuesta metodológica y resultados para Castilla-La Mancha/The Effectiveness of European Rural Development P," Estudios de Economia Aplicada, Estudios de Economia Aplicada, vol. 29, pages 223-246, Abril.
    14. Riccardo D’Alberto & Matteo Zavalloni & Meri Raggi & Davide Viaggi, 2018. "AES Impact Evaluation With Integrated Farm Data: Combining Statistical Matching and Propensity Score Matching," Sustainability, MDPI, vol. 10(11), pages 1-24, November.
    15. Mara Giua, 2017. "Spatial Discontinuity For The Impact Assessment Of The Eu Regional Policy: The Case Of Italian Objective 1 Regions," Journal of Regional Science, Wiley Blackwell, vol. 57(1), pages 109-131, January.
    16. Loi, Massimo & Rodrigues, Margarida, 2012. "A note on the impact evaluation of public policies: the counterfactual analysis," MPRA Paper 42444, University Library of Munich, Germany.
    17. Guido Pellegrini & Flavia Terribile & Ornella Tarola & Teo Muccigrosso & Federica Busillo, 2013. "Measuring the effects of European Regional Policy on economic growth: A regression discontinuity approach," Papers in Regional Science, Wiley Blackwell, vol. 92(1), pages 217-233, March.
    18. Chihiro Udagawa & Ian Hodge & Mark Reader, 2014. "Farm Level Costs of Agri-environment Measures: The Impact of Entry Level Stewardship on Cereal Farm Incomes," Journal of Agricultural Economics, Wiley Blackwell, vol. 65(1), pages 212-233, January.
    19. Ratinger, Tomas & Medonos, Tomas & Spicka, Jindrich & Hruska, Martin & Vilhelm, Vaclav, 2012. "The assessment of the effects of investment support measures of the Rural Development Programmes: the case of the Czech Republic," 131st Seminar, September 18-19, 2012, Prague, Czech Republic 135775, European Association of Agricultural Economists.
    20. Liesbeth Colen & Sergio Gomez y Paloma & Uwe Latacz-Lohmann & Marianne Lefebvre & Raphaële Préget & Sophie Thoyer, 2016. "Economic Experiments as a Tool for Agricultural Policy Evaluation: Insights from the European CAP," Canadian Journal of Agricultural Economics/Revue canadienne d'agroeconomie, Canadian Agricultural Economics Society/Societe canadienne d'agroeconomie, vol. 64(4), pages 667-694, December.
    21. Krijn Poppe & Hans Vrolijk, 2018. "Microdata: a critical source for policy evaluation," EuroChoices, The Agricultural Economics Society, vol. 17(1), pages 28-35, April.
    22. Monsalve, Fabio & Zafrilla, Jorge Enrique & Cadarso, María-Ángeles, 2016. "Where have all the funds gone? Multiregional input-output analysis of the European Agricultural Fund for Rural Development," Ecological Economics, Elsevier, vol. 129(C), pages 62-71.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Zoltán Bakucs & Imre Fertő & Zsófia Benedek, 2019. "Success or Waste of Taxpayer Money? Impact Assessment of Rural Development Programs in Hungary," Sustainability, MDPI, vol. 11(7), pages 1-23, April.
    2. Uehleke, Reinhard & Petrick, Martin & Hüttel, Silke, 2022. "Evaluations of agri-environmental schemes based on observational farm data: The importance of covariate selection," Land Use Policy, Elsevier, vol. 114(C).
    3. Whelan, Adele & McGuinness, Seamus & Barrett, Alan, 2021. "Review of International Approaches to Evaluating Rural and Community Development Investment and Supports," Research Series, Economic and Social Research Institute (ESRI), number RS124.
    4. Michalek, Jerzy, 2022. "Environmental and farm impacts of the EU RDP agri-environmental measures: Evidence from Slovak regions," Land Use Policy, Elsevier, vol. 113(C).
    5. Roberto Esposti, 2022. "The Coevolution of Policy Support and Farmers' Behaviour. An investigation on Italian agriculture over the 2008-2019 period," Working Papers 464, Universita' Politecnica delle Marche (I), Dipartimento di Scienze Economiche e Sociali.
    6. Alessandro Bonadonna & Andrea Rostagno & Riccardo Beltramo, 2020. "Improving the Landscape and Tourism in Marginal Areas: The Case of Land Consolidation Associations in the North-West of Italy," Land, MDPI, vol. 9(6), pages 1-15, May.
    7. Lillemets, Jüri & Fertő, Imre & Viira, Ants-Hannes, 2022. "The socioeconomic impacts of the CAP: Systematic literature review," Land Use Policy, Elsevier, vol. 114(C).
    8. Metta, Matteo & Ciliberti, Stefano & Obi, Chinedu & Bartolini, Fabio & Klerkx, Laurens & Brunori, Gianluca, 2022. "An integrated socio-cyber-physical system framework to assess responsible digitalisation in agriculture: A first application with Living Labs in Europe," Agricultural Systems, Elsevier, vol. 203(C).
    9. Ramona Pîrvu & Laurențiu Dragomir & Bogdan Budică & Răducu-Ștefan Bratu & Sorin Dinulescu & Lili Țenea, 2022. "The Impact of RDP Measures on the Rural Development: The Case of Romania," Sustainability, MDPI, vol. 14(8), pages 1-27, April.
    10. Kelly, Elish & McGuinness, Seamus & Devlin, Anne, 2024. "Developing a framework to monitor rural development policy in Ireland: Opportunities and challenges," Research Series, Economic and Social Research Institute (ESRI), number RS172.
    11. Ewa Kiryluk-Dryjska & Barbara Więckowska & Kinga Smolińska, 2022. "Winners and Losers of the CAP’s Rural Development Policy in Poland," Agriculture, MDPI, vol. 12(5), pages 1-15, May.
    12. Beata Bieszk-Stolorz & Krzysztof Dmytrów, 2021. "Impact of Changes to Procedures on the Evaluation of the Effectiveness of Forms of Professional Activation in Poland," Economies, MDPI, vol. 9(2), pages 1-18, June.
    13. Eunji Choi & Jonghoon Park & Seongwoo Lee, 2020. "The Effect of the Comprehensive Rural Village Development Program on Farm Income in South Korea," Sustainability, MDPI, vol. 12(17), pages 1-23, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Whelan, Adele & McGuinness, Seamus & Barrett, Alan, 2021. "Review of International Approaches to Evaluating Rural and Community Development Investment and Supports," Research Series, Economic and Social Research Institute (ESRI), number RS124.
    2. Zoltán Bakucs & Imre Fertő & Zsófia Benedek, 2019. "Success or Waste of Taxpayer Money? Impact Assessment of Rural Development Programs in Hungary," Sustainability, MDPI, vol. 11(7), pages 1-23, April.
    3. Blazy, J.-M. & Subervie, J. & Paul, J. & Causeret, F. & Guindé, L. & Moulla, S. & Thomas, A. & Sierra, J., 2021. "Ex-ante assessment of the cost-effectiveness of public policies to sequester carbon in soils," Ecological Economics, Elsevier, vol. 190(C).
    4. Michalek, Jerzy, 2022. "Environmental and farm impacts of the EU RDP agri-environmental measures: Evidence from Slovak regions," Land Use Policy, Elsevier, vol. 113(C).
    5. Jean-Marc Blazy & Julie Subervie & Jacky Paul & François Causeret & Loic Guinde & Sarah Moulla & Alban Thomas & Jorge Sierra, 2020. "Ex ante assessment of the cost-effectiveness of Agri-Environmental Schemes promoting compost use to sequester carbon in soils in Guadeloupe," CEE-M Working Papers hal-02748634, CEE-M, Universtiy of Montpellier, CNRS, INRA, Montpellier SupAgro.
    6. Ratinger, T. & Medonos, Tomas & Hruška, M., 2013. "An Assessment of the Differentiated Effects of the Investment Support to Agricultural Modernisation: the Case of the Czech Republic," AGRIS on-line Papers in Economics and Informatics, Czech University of Life Sciences Prague, Faculty of Economics and Management, vol. 5(4), pages 1-12, December.
    7. Wawrzyniec Czubak & Krzysztof Piotr Pawłowski, 2020. "Sustainable Economic Development of Farms in Central and Eastern European Countries Driven by Pro-investment Mechanisms of the Common Agricultural Policy," Agriculture, MDPI, vol. 10(4), pages 1-19, March.
    8. Cisilino, Federica & Bodini, Antonella & Zanoli, Agostina, 2019. "Rural development programs’ impact on environment: An ex-post evaluation of organic faming," Land Use Policy, Elsevier, vol. 85(C), pages 454-462.
    9. Riccardo D’Alberto & Matteo Zavalloni & Meri Raggi & Davide Viaggi, 2018. "AES Impact Evaluation With Integrated Farm Data: Combining Statistical Matching and Propensity Score Matching," Sustainability, MDPI, vol. 10(11), pages 1-24, November.
    10. Uehleke, Reinhard & Petrick, Martin & Hüttel, Silke, 2022. "Evaluations of agri-environmental schemes based on observational farm data: The importance of covariate selection," Land Use Policy, Elsevier, vol. 114(C).
    11. Riccardo Crescenzi & Mara Giua, 2018. "One or Many Cohesion Policies of the European Union? On the Diverging Impacts of Cohesion Policy across Member States," SERC Discussion Papers 0230, Centre for Economic Performance, LSE.
    12. Roggendorf, Wolfgang & Schwarze, Stefan, 2020. "Die Wirkung von Agrarumweltmaßnahmen auf betriebliche Stickstoffbilanzen – Empirische Ergebnisse aus Nordrhein-Westfalen," 60th Annual Conference, Halle/ Saale, Germany, September 23-25, 2020 305604, German Association of Agricultural Economists (GEWISOLA).
    13. Huber, Robert & Bakker, Martha & Balmann, Alfons & Berger, Thomas & Bithell, Mike & Brown, Calum & Grêt-Regamey, Adrienne & Xiong, Hang & Le, Quang Bao & Mack, Gabriele & Meyfroidt, Patrick & Millingt, 2018. "Representation of decision-making in European agricultural agent-based models," Agricultural Systems, Elsevier, vol. 167(C), pages 143-160.
    14. Maximilian v. Ehrlich & Henry G. Overman, 2020. "Place-Based Policies and Spatial Disparities across European Cities," Journal of Economic Perspectives, American Economic Association, vol. 34(3), pages 128-149, Summer.
    15. Adewale H. Adenuga & Claire Jack & Austen Ashfield & Michael Wallace, 2021. "Assessing the Impact of Participatory Extension Programme Membership on Farm Business Performance in Northern Ireland," Agriculture, MDPI, vol. 11(10), pages 1-12, September.
    16. Davide Fiaschi & Andrea Mario Lavezzi & Angela Parenti, 2018. "Does EU cohesion policy work? Theory and evidence," Journal of Regional Science, Wiley Blackwell, vol. 58(2), pages 386-423, March.
    17. Lajos Baráth & Imre Fertő & Štefan Bojnec, 2020. "The Effect of Investment, LFA and Agri‐environmental Subsidies on the Components of Total Factor Productivity: The Case of Slovenian Farms," Journal of Agricultural Economics, Wiley Blackwell, vol. 71(3), pages 853-876, September.
    18. repec:lic:licosd:37015 is not listed on IDEAS
    19. Jerzy Michalek & Pavel Ciaian & D'Artis Kancs, 2016. "Investment Crowding Out: Firm-Level Evidence from Northern Germany," Regional Studies, Taylor & Francis Journals, vol. 50(9), pages 1579-1594, September.
    20. Paulina Permatasari & Assyifa Szami Ilman & Carol Ann Tilt & Dian Lestari & Saiful Islam & Rita Helbra Tenrini & Arif Budi Rahman & Agunan Paulus Samosir & Irwanda Wisnu Wardhana, 2021. "The Village Fund Program in Indonesia: Measuring the Effectiveness and Alignment to Sustainable Development Goals," Sustainability, MDPI, vol. 13(21), pages 1-30, November.
    21. Luc Behaghel & Karen Macours & Julie Subervie, 2019. "How can randomised controlled trials help improve the design of the common agricultural policy?," European Review of Agricultural Economics, Oxford University Press and the European Agricultural and Applied Economics Publications Foundation, vol. 46(3), pages 473-493.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jsusta:v:11:y:2019:i:4:p:1105-:d:207509. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.