IDEAS home Printed from https://ideas.repec.org/a/oup/rseval/v31y2022i4p486-497..html
   My bibliography  Save this article

Making the cut: How panel reviewers use evaluation devices to select applications at the European Research Council

Author

Listed:
  • Lucas Brunet
  • Ruth Müller

Abstract

The European Research Council (ERC) receives many high-quality applications, but funds only a few. We analyze how members of ERC review panels assess applications in the first, highly competitive step of evaluations for ERC Starting and Consolidator Grants. Drawing on interviews with ERC panel members in different fields, we show that they adopt a set of evaluation devices that offer pragmatic and standardized ways of evaluating in a time-constrained and highly competitive setting. Through the use of evaluation devices, panel reviewers enact and generate a distinct reviewing expertise that encompasses subject-specific knowledge and knowledge about how to accomplish evaluation within a situated setting. We find that ERC panel reviewers employ four evaluation devices during the first step of ERC reviews: first, reviewers base judgments on applicants’ prior achievements (delegation devices); second, they adjust their evaluations of individual applications to the quality of a given set of applications (calibration devices); third, they combine multiple elements to assess the feasibility of proposals (articulation devices); and finally, they consider the impact of the proposed research on science and society (contribution devices). We show that the current use of these devices generates what we have termed evaluative pragmatism: a mode of reviewing that is shaped by and accommodated to the need to review many high-quality proposals in a short time period with possibly limited expert knowledge. In conclusion, we discuss how the prevalence of evaluative pragmatism in the first step of ERC panel reviews shapes candidate selection, particularly regarding human and epistemic diversity in European research.

Suggested Citation

  • Lucas Brunet & Ruth Müller, 2022. "Making the cut: How panel reviewers use evaluation devices to select applications at the European Research Council," Research Evaluation, Oxford University Press, vol. 31(4), pages 486-497.
    • Handle: RePEc:oup:rseval:v:31:y:2022:i:4:p:486-497.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/reseval/rvac040
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Pleun van Arensbergen & Inge van der Weijden & Peter van den Besselaar, 2014. "Different views on scholarly talent: What are the talents we are looking for in science?," Research Evaluation, Oxford University Press, vol. 23(4), pages 273-284.
    2. Ruth Müller & Sarah de Rijcke, 2017. "Thinking with Indicators. Exploring the Epistemic Impacts of Academic Performance Indicators in the Life Sciences," Research Evaluation, Oxford University Press, vol. 26(4), pages 361-361.
    3. Laudel, Grit & Gläser, Jochen, 2014. "Beyond breakthrough research: Epistemic properties of research and their consequences for research funding," Research Policy, Elsevier, vol. 43(7), pages 1204-1216.
    4. Ruth Müller & Sarah de Rijcke, 2017. "Exploring the epistemic impacts of academic performance indicators in the life sciences," Research Evaluation, Oxford University Press, vol. 26(3), pages 157-168.
    5. Terttu Luukkonen, 2014. "The European Research Council and the European research funding landscape," Science and Public Policy, Oxford University Press, vol. 41(1), pages 29-43.
    6. Fabian Muniesa & Yuval Millo & Michel Callon, 2007. "An introduction to market devices," Post-Print halshs-00177928, HAL.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tony Ross-Hellauer & Thomas Klebel & Petr Knoth & Nancy Pontika, 2024. "Value dissonance in research(er) assessment: individual and perceived institutional priorities in review, promotion, and tenure," Science and Public Policy, Oxford University Press, vol. 51(3), pages 337-351.
    2. Tóth, Tamás & Demeter, Márton & Csuhai, Sándor & Major, Zsolt Balázs, 2024. "When career-boosting is on the line: Equity and inequality in grant evaluation, productivity, and the educational backgrounds of Marie Skłodowska-Curie Actions individual fellows in social sciences an," Journal of Informetrics, Elsevier, vol. 18(2).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Michaela Strinzel & Josh Brown & Wolfgang Kaltenbrunner & Sarah Rijcke & Michael Hill, 2021. "Ten ways to improve academic CVs for fairer research assessment," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-4, December.
    2. Eugenio Petrovich, 2022. "Bibliometrics in Press. Representations and uses of bibliometric indicators in the Italian daily newspapers," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2195-2233, May.
    3. Frank J. Rijnsoever & Laurens K. Hessels, 2021. "How academic researchers select collaborative research projects: a choice experiment," The Journal of Technology Transfer, Springer, vol. 46(6), pages 1917-1948, December.
    4. Gabriel-Alexandru Vîiu & Mihai Păunescu, 2021. "The citation impact of articles from which authors gained monetary rewards based on journal metrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 4941-4974, June.
    5. Fabio Zagonari, 2019. "Scientific Production and Productivity for Characterizing an Author’s Publication History: Simple and Nested Gini’s and Hirsch’s Indexes Combined," Publications, MDPI, vol. 7(2), pages 1-30, May.
    6. Gabriel-Alexandru Vȋiu & Mihai Păunescu, 2021. "The lack of meaningful boundary differences between journal impact factor quartiles undermines their independent use in research evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1495-1525, February.
    7. Alberto Baccini & Giuseppe De Nicolao & Eugenio Petrovich, 2019. "Citation gaming induced by bibliometric evaluation: A country-level comparative analysis," PLOS ONE, Public Library of Science, vol. 14(9), pages 1-16, September.
    8. Claartje J Vinkenburg & Sara Connolly & Stefan Fuchs & Channah Herschberg & Brigitte Schels, 2020. "Mapping career patterns in research: A sequence analysis of career histories of ERC applicants," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-19, July.
    9. Sandra Rousseau & Ronald Rousseau, 2021. "Bibliometric Techniques And Their Use In Business And Economics Research," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1428-1451, December.
    10. Alexander Schniedermann, 2021. "A comparison of systematic reviews and guideline-based systematic reviews in medical studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9829-9846, December.
    11. Lai Ma, 2023. "Information, platformized," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(2), pages 273-282, February.
    12. Piotr Śpiewanowski & Oleksandr Talavera, 2021. "Journal rankings and publication strategy," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3227-3242, April.
    13. Grouiez, Pascal & Debref, Romain & Vivien, Franck-Dominique & Befort, Nicolas, 2023. "The complex relationships between non-food agriculture and the sustainable bioeconomy: The French case," Ecological Economics, Elsevier, vol. 214(C).
    14. Juliana de Oliveira Becheri & Acsa Hosken Gusmão & Elisa Guimarães Cozadi & Paulo Henrique Montagnana Vicente Leme, 2023. "Instagram in the Modest Fashion Market: Analysis by the Perspective of Sociotechnical Structures," RAC - Revista de Administração Contemporânea (Journal of Contemporary Administration), ANPAD - Associação Nacional de Pós-Graduação e Pesquisa em Administração, vol. 27(Vol. 27 N), pages 220168-2201.
    15. Bear, Laura, 2020. "Speculations on infrastructure: from colonial public works to a postcolonial global asset class on the Indian Railways 1840-2017," LSE Research Online Documents on Economics 103445, London School of Economics and Political Science, LSE Library.
    16. Blanc, Antoine & Huault, Isabelle, 2014. "Against the digital revolution? Institutional maintenance and artefacts within the French recorded music industry," Technological Forecasting and Social Change, Elsevier, vol. 83(C), pages 10-23.
    17. Thomas Wainwright, 2011. "Elite Knowledges: Framing Risk and the Geographies of Credit," Environment and Planning A, , vol. 43(3), pages 650-665, March.
    18. Walter, Christian, 2016. "The financial Logos: The framing of financial decision-making by mathematical modelling," Research in International Business and Finance, Elsevier, vol. 37(C), pages 597-604.
    19. Martha Poon, 2009. "From New Deal institutions to capital markets: commercial consumer risk scores and the making of subprime mortgage finance," Post-Print halshs-00359712, HAL.
    20. Rezende, Daniel Carvalho de, 2014. "Politics in Food Markets: alternative modes of qualification and engaging," Brazilian Journal of Rural Economy and Sociology (Revista de Economia e Sociologia Rural-RESR), Sociedade Brasileira de Economia e Sociologia Rural, vol. 52(2), pages 1-14, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:rseval:v:31:y:2022:i:4:p:486-497.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/rev .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.