IDEAS home Printed from https://ideas.repec.org/a/kap/itaxpf/v32y2025i2d10.1007_s10797-024-09875-4.html
   My bibliography  Save this article

Evidence-based policy or beauty contest? An LLM-based meta-analysis of EU cohesion policy evaluations

Author

Listed:
  • Zareh Asatryan

    (ZEW Mannheim)

  • Carlo Birkholz

    (University of Mannheim)

  • Friedrich Heinemann

    (University of Heidelberg)

Abstract

Independent and high-quality evaluations of government policies are an important input for designing evidence-based policy. Institutional frictions and lack of incentives to write such evaluations, on the other hand, carry the risk of turning the system into a costly beauty contest. We study one of the most advanced markets of policy evaluations in the world, the evaluations of EU Cohesion Policy interventions by the EU Member States. We use a large language model to quantify the findings of about 2,300 evaluations, and complement this data with our own survey of the evaluation authors. We show that the findings of evaluations are inconsistent with those of the academic literature on the output impacts of Cohesion Policy. Using further variation across Member States, our analysis suggests that the market of evaluations is rather oligopolistic within Member States, that it is very fragmented across the EU, and that there is often a strong involvement of managing authorities in the work of formally independent evaluators. These factors contribute to making the findings of the evaluations overly optimistic (beautiful) risking their relevance for (evidence-based) policy. We conclude by discussing reform options to make the evaluations of EU Cohesion Policy more unbiased and effective.

Suggested Citation

  • Zareh Asatryan & Carlo Birkholz & Friedrich Heinemann, 2025. "Evidence-based policy or beauty contest? An LLM-based meta-analysis of EU cohesion policy evaluations," International Tax and Public Finance, Springer;International Institute of Public Finance, vol. 32(2), pages 625-655, April.
  • Handle: RePEc:kap:itaxpf:v:32:y:2025:i:2:d:10.1007_s10797-024-09875-4
    DOI: 10.1007/s10797-024-09875-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10797-024-09875-4
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.

    File URL: https://libkey.io/10.1007/s10797-024-09875-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    More about this item

    Keywords

    Policy evaluation; EU cohesion policy; Large language model;
    All these keywords.

    JEL classification:

    • A11 - General Economics and Teaching - - General Economics - - - Role of Economics; Role of Economists
    • C45 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Neural Networks and Related Topics
    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness
    • H43 - Public Economics - - Publicly Provided Goods - - - Project Evaluation; Social Discount Rate
    • H54 - Public Economics - - National Government Expenditures and Related Policies - - - Infrastructures

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:kap:itaxpf:v:32:y:2025:i:2:d:10.1007_s10797-024-09875-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.