IDEAS home Printed from https://ideas.repec.org/a/kap/policy/v52y2019i4d10.1007_s11077-019-09358-y.html
   My bibliography  Save this article

Evaluations as a decent knowledge base? Describing and explaining the quality of the European Commission’s ex-post legislative evaluations

Author

Listed:
  • Stijn Voorst

    (Radboud University)

  • Ellen Mastenbroek

    (Radboud University)

Abstract

Evaluations are a potentially important tool for democratic governments: they provide a basis for accountability and policy learning. To contribute to these key functions, evaluations must be of sufficient methodological quality. However, this quality is threatened by both political influences and technical complexities. This article describes and explains the variance in the quality of ex-post legislative (EPL) evaluations conducted by the European Commission, which is a frontrunner in this realm. A number of potential political and technical explanations of evaluation quality are tested with a unique, self-constructed dataset of 153 EPL evaluations. The results show that the Commission’s EPL evaluations usually apply a robust methodology, while the clarity of their scope, the accuracy of their data and the foundations of their conclusions are problematic. The variance in this quality is mainly explained by the type of evaluator: EPL evaluations conducted by external actors are of higher quality than evaluations conducted internally by the Commission.

Suggested Citation

  • Stijn Voorst & Ellen Mastenbroek, 2019. "Evaluations as a decent knowledge base? Describing and explaining the quality of the European Commission’s ex-post legislative evaluations," Policy Sciences, Springer;Society of Policy Sciences, vol. 52(4), pages 625-644, December.
    • Handle: RePEc:kap:policy:v:52:y:2019:i:4:d:10.1007_s11077-019-09358-y
    DOI: 10.1007/s11077-019-09358-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11077-019-09358-y
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s11077-019-09358-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kaeding, Michael, 2006. "Determinants of Transposition Delay in the European Union," Journal of Public Policy, Cambridge University Press, vol. 26(3), pages 229-253, December.
    2. Christian Adam & Yves Steinebach & Christoph Knill, 2018. "Neglected challenges to evidence-based policy-making: the problem of policy accumulation," Policy Sciences, Springer;Society of Policy Sciences, vol. 51(3), pages 269-290, September.
    3. Bernard Steunenberg, 2006. "Turning Swift Policy-making into Deadlock and Delay," European Union Politics, , vol. 7(3), pages 293-319, September.
    4. Jacopo Torriti, 2010. "Impact Assessment and the Liberalization of the EU Energy Markets: Evidence‐Based Policy‐Making or Policy‐Based Evidence‐Making?," Journal of Common Market Studies, Wiley Blackwell, vol. 48(4), pages 1065-1081, September.
    5. Stijn van Voorst & Ellen Mastenbroek, 2017. "Enforcement tool or strategic instrument? The initiation of ex-post legislative evaluations by the European Commission," European Union Politics, , vol. 18(4), pages 640-657, December.
    6. Falk Daviter, 2015. "The political use of knowledge in the policy process," Policy Sciences, Springer;Society of Policy Sciences, vol. 48(4), pages 491-505, December.
    7. repec:bla:jcmkts:v:48:y:2010:i::p:1065-1081 is not listed on IDEAS
    8. Adcock, Robert & Collier, David, 2001. "Measurement Validity: A Shared Standard for Qualitative and Quantitative Research," American Political Science Review, Cambridge University Press, vol. 95(3), pages 529-546, September.
    9. Lawrence Mead, 2015. "Only connect: Why government often ignores research," Policy Sciences, Springer;Society of Policy Sciences, vol. 48(2), pages 257-272, June.
    10. Schwartz, Robert & Mayne, John, 2005. "Assuring the quality of evaluative information: theory and practice," Evaluation and Program Planning, Elsevier, vol. 28(1), pages 1-14.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Steffen Eckhard & Vytautas Jankauskas, 2020. "Explaining the political use of evaluation in international organizations," Policy Sciences, Springer;Society of Policy Sciences, vol. 53(4), pages 667-695, December.
    2. Kupiec, Tomasz, 2022. "Does evaluation quality matter? Quantitative analysis of the use of evaluation findings in the field of cohesion policy in Poland," Evaluation and Program Planning, Elsevier, vol. 93(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Stijn van Voorst & Ellen Mastenbroek, 2017. "Enforcement tool or strategic instrument? The initiation of ex-post legislative evaluations by the European Commission," European Union Politics, , vol. 18(4), pages 640-657, December.
    2. Carlos Mendez & John Bachtler, 2017. "Financial Compliance in the European Union: A Cross-National Assessment of Financial Correction Patterns and Causes in Cohesion Policy," Journal of Common Market Studies, Wiley Blackwell, vol. 55(3), pages 569-592, May.
    3. Garrett Ward Richards, 2019. "The Science–Policy Relationship Hierarchy (SPRHi) model of co-production: how climate science organizations have influenced the policy process in Canadian case studies," Policy Sciences, Springer;Society of Policy Sciences, vol. 52(1), pages 67-95, March.
    4. Miriam Hartlapp & Gerda Falkner, 2009. "Problems of Operationalization and Data in EU Compliance Research," European Union Politics, , vol. 10(2), pages 281-304, June.
    5. Egert Juuse & Rainer Kattel, 2014. "Financial Regulation in Estonia," Working papers wpaper57, Financialisation, Economy, Society & Sustainable Development (FESSUD) Project.
    6. Jesper Dahl Kelstrup & Jonas Videbæk Jørgensen, 2024. "Explaining differences in research utilization in evidence-based government ministries," Policy Sciences, Springer;Society of Policy Sciences, vol. 57(2), pages 257-280, June.
    7. Asya Zhelyazkova & René Torenvlied, 2009. "The Time-Dependent Effect of Conflict in the Council on Delays in the Transposition of EU Directives," European Union Politics, , vol. 10(1), pages 35-62, March.
    8. Tanja A. Börzel & Tobias Hofmann & Diana Panke, 2011. "Policy Matters But How? Explaining Non-Compliance Dynamics in the EU," KFG Working Papers p0024, Free University Berlin.
    9. Hyensup Shim & Kiyoon Shin, 2021. "Empirical Analysis of Evidence-Based Policymaking in R&D Programmes," Sustainability, MDPI, vol. 14(1), pages 1-15, December.
    10. Dimiter Toshkov, 2008. "Embracing European Law," European Union Politics, , vol. 9(3), pages 379-402, September.
    11. Koetter, Michael & Krause, Thomas & Tonzer, Lena, 2019. "Delay determinants of European Banking Union implementation," European Journal of Political Economy, Elsevier, vol. 58(C), pages 1-20.
    12. Schakel, Arjan Hille, 2009. "A Postfunctionalist Theory of Regional Government," MPRA Paper 21596, University Library of Munich, Germany.
    13. Zim Nwokora & Riccardo Pelizzo, 2017. "Measuring Party System Change: A Systems Perspective," Research Africa Network Working Papers 17/048, Research Africa Network (RAN).
    14. Vargas, Andrés & Sarmiento Erazo, Juan Pablo & Diaz, David, 2020. "Has Cost Benefit Analysis Improved Decisions in Colombia? Evidence from the Environmental Licensing Process," Ecological Economics, Elsevier, vol. 178(C).
    15. J. C. Sharman, 2007. "Rationalist and Constructivist Perspectives on Reputation," Political Studies, Political Studies Association, vol. 55(1), pages 20-37, March.
    16. Muhammad Nabeel Siddiqui, 2013. "Impact Of Work Life Conflict On Employee Performance," Far East Journal of Psychology and Business, Far East Research Centre, vol. 12(3), pages 26-40, September.
    17. Jose Antonio Alonso & Ana Luiza Cortez & Stephan Klasen, 2014. "LDC and other country groupings: How useful are current approaches to classify countries in a more hetergeneous developing world?," CDP Background Papers 021, United Nations, Department of Economics and Social Affairs.
    18. Willem E. Saris & André Pirralha & Diana Zavala-Rojas, 2018. "Testing the Comparability of Different Types of Social Indicators Across Groups," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 135(3), pages 927-939, February.
    19. Anete Veidemane & Frans Kaiser & Daniela Craciun, 2021. "Inclusive Higher Education Access for Underrepresented Groups: It Matters, But How Can Universities Measure It?," Social Inclusion, Cogitatio Press, vol. 9(3), pages 44-57.
    20. Asya Zhelyazkova & Nikoleta Yordanova, 2015. "Signalling ‘compliance’: The link between notified EU directive implementation and infringement cases," European Union Politics, , vol. 16(3), pages 408-428, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:kap:policy:v:52:y:2019:i:4:d:10.1007_s11077-019-09358-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.