IDEAS home Printed from https://ideas.repec.org/p/sru/ssewps/2015-24.html
   My bibliography  Save this paper

Do academics doubt their own research?

Author

Listed:
  • Lee Stapleton

    (SPRU (Science Policy Research Unit), School of Business, Management & Economics, University of Sussex, Brighton, BN1 9SL, U.K.)

Abstract

When do experts doubt or question their own previously published research and why? An online survey was designed and distributed across academic staff and postgraduate research students at different universities in Great Britain. Respondents (n = 202 - 244) identified the likelihoods of six different (quasi) hypothetical occurrences causing them to doubt or question work they have published in peer reviewed journals. They are: two objective and two semi-objective citation based metrics, plus two semi- objective metrics based on verbalised reactions. Only limited support is found from this study to suggest that the authors of primary research would agree with any judgements made by others about their research based on these metrics. The occurrence most likely to cause respondents to doubt or question their previously published research was where the majority of citing studies suggested mistakes in their work. In a multivariate context, only age and nationality are significant determinants of doubt beyond average likelihoods. Understanding and acknowledging what makes authors of primary research doubt their own research could increase the validity of those who pass judgement.

Suggested Citation

  • Lee Stapleton, 2015. "Do academics doubt their own research?," SPRU Working Paper Series 2015-24, SPRU - Science Policy Research Unit, University of Sussex Business School.
    • Handle: RePEc:sru:ssewps:2015-24
    as

    Download full text from publisher

    File URL: https://www.sussex.ac.uk/webteam/gateway/file.php?name=2015-24-swps-stapleton.pdf&site=25
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Michela Nardo & Michaela Saisana & Andrea Saltelli & Stefano Tarantola & Anders Hoffman & Enrico Giovannini, 2005. "Handbook on Constructing Composite Indicators: Methodology and User Guide," OECD Statistics Working Papers 2005/3, OECD Publishing.
    2. Willy Aspinall, 2010. "A route to more tractable expert advice," Nature, Nature, vol. 463(7279), pages 294-295, January.
    3. Daniele Fanelli, 2013. "Positive results receive more citations, but only in some disciplines," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(2), pages 701-709, February.
    4. J. L. Bamber & W. P. Aspinall, 2013. "An expert judgement assessment of future sea level rise from the ice sheets," Nature Climate Change, Nature, vol. 3(4), pages 424-427, April.
    5. James Fleck, 1998. "Expertise: Knowledge, Power and Tradeability," Palgrave Macmillan Books, in: Robin Williams & Wendy Faulkner & James Fleck (ed.), Exploring Expertise, chapter 7, pages 143-171, Palgrave Macmillan.
    6. Koenker, Roger W & Bassett, Gilbert, Jr, 1978. "Regression Quantiles," Econometrica, Econometric Society, vol. 46(1), pages 33-50, January.
    7. C. O. S. Sorzano & J. Vargas & G. Caffarena-Fernández & A. Iriarte, 2014. "Comparing scientific performance among equals," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(3), pages 1731-1745, December.
    8. Ayres, Ian & Vars, Fredrick E, 2000. "Determinants of Citations to Articles in Elite Law Reviews," The Journal of Legal Studies, University of Chicago Press, vol. 29(1), pages 427-450, January.
    9. Andy Stirling, 2010. "Keep it complex," Nature, Nature, vol. 468(7327), pages 1029-1031, December.
    10. Woodward, Richard T. & Wui, Yong-Suhk, 2001. "The economic value of wetland services: a meta-analysis," Ecological Economics, Elsevier, vol. 37(2), pages 257-270, May.
    11. David Moher & Alessandro Liberati & Jennifer Tetzlaff & Douglas G Altman & The PRISMA Group, 2009. "Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement," PLOS Medicine, Public Library of Science, vol. 6(7), pages 1-6, July.
    12. T.D. Stanley & Hristos Doucouliagos & Margaret Giles & Jost H. Heckemeyer & Robert J. Johnston & Patrice Laroche & Jon P. Nelson & Martin Paldam & Jacques Poot & Geoff Pugh & Randall S. Rosenberger & , 2013. "Meta-Analysis Of Economics Research Reporting Guidelines," Journal of Economic Surveys, Wiley Blackwell, vol. 27(2), pages 390-394, April.
    13. T. D. Stanley, 2001. "Wheat from Chaff: Meta-analysis as Quantitative Literature Review," Journal of Economic Perspectives, American Economic Association, vol. 15(3), pages 131-150, Summer.
    14. Donald MacKenzie, 1998. "The Certainty Trough," Palgrave Macmillan Books, in: Robin Williams & Wendy Faulkner & James Fleck (ed.), Exploring Expertise, chapter 15, pages 325-329, Palgrave Macmillan.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Geuna, Aldo & Piolatto, Matteo, 2016. "Research assessment in the UK and Italy: Costly and difficult, but probably worth it (at least for a while)," Research Policy, Elsevier, vol. 45(1), pages 260-271.
    2. Mariana Mazzucato, 2015. "The Green Entrepreneurial State," SPRU Working Paper Series 2015-28, SPRU - Science Policy Research Unit, University of Sussex Business School.
    3. Lorenz Gollwitzer & David Ockwell & Adrian Ely, 2015. "Institutional Innovation in the Management of Pro-Poor Energy Access in East Africa," SPRU Working Paper Series 2015-29, SPRU - Science Policy Research Unit, University of Sussex Business School.
    4. Friedemann Polzin & Steve Sorrell & Colin Nolden, 2015. "Innovative Procurement Frameworks for Energy Performance Contracting in the UK Public Sector," SPRU Working Paper Series 2015-31, SPRU - Science Policy Research Unit, University of Sussex Business School.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Larkin, Matthew P. & Askarov, Zohid & Doucouliagos, Hristos & Dubelaar, Chris & Klona, Maria & Newton, Joshua & Stanley, T.D. & Vocino, Andrea, 2019. "Do house prices ride the wave of immigration?," Journal of Housing Economics, Elsevier, vol. 46(C).
    2. Tomáš Havránek & T. D. Stanley & Hristos Doucouliagos & Pedro Bom & Jerome Geyer‐Klingeberg & Ichiro Iwasaki & W. Robert Reed & Katja Rost & R. C. M. van Aert, 2020. "Reporting Guidelines For Meta‐Analysis In Economics," Journal of Economic Surveys, Wiley Blackwell, vol. 34(3), pages 469-475, July.
    3. Stéphane Goutte & David Guerreiro & Bilel Sanhaji & Sophie Saglio & Julien Chevallier, 2019. "International Financial Markets," Post-Print halshs-02183053, HAL.
    4. Diana Barros & Aurora A. C. Teixeira, 2021. "Unlocking the black box: A comprehensive meta-analysis of the main determinants of within-region income inequality," Review of Regional Research: Jahrbuch für Regionalwissenschaft, Springer;Gesellschaft für Regionalforschung (GfR), vol. 41(1), pages 55-93, February.
    5. Iryna Printezis & Carola Grebitus & Stefan Hirsch, 2019. "The price is right!? A meta-regression analysis on willingness to pay for local food," PLOS ONE, Public Library of Science, vol. 14(5), pages 1-23, May.
    6. Klomp, Jeroen, 2023. "Political budget cycles in military expenditures: A meta-analysis," Economic Analysis and Policy, Elsevier, vol. 77(C), pages 1083-1102.
    7. Hermine Vedogbeton & Robert J. Johnston, 2020. "Commodity Consistent Meta-Analysis of Wetland Values: An Illustration for Coastal Marsh Habitat," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 75(4), pages 835-865, April.
    8. Mattia Filomena & Matteo Picchio, 2023. "Retirement and health outcomes in a meta‐analytical framework," Journal of Economic Surveys, Wiley Blackwell, vol. 37(4), pages 1120-1155, September.
    9. Johanna Choumert & Pascale Combes Motel & Charlain Guegang Djimeli, 2017. "The biofuel-development nexus: A meta-analysis," CERDI Working papers halshs-01512678, HAL.
    10. Nicolas Fleury & Fabrice Gilles, 2015. "A meta-regression analysis on intergenerational transmission of education: publication bias and genuine empirical effect," TEPP Working Paper 2015-02, TEPP.
    11. Choumert Nkolo, Johanna & Combes Motel, Pascale & Guegang Djimeli, Charlain, 2018. "Income-generating Effects of Biofuel Policies: A Meta-analysis of the CGE Literature," Ecological Economics, Elsevier, vol. 147(C), pages 230-242.
    12. Rosenberger, Randall S. & Stanley, Tom D., 2006. "Measurement, generalization, and publication: Sources of error in benefit transfers and their management," Ecological Economics, Elsevier, vol. 60(2), pages 372-378, December.
    13. Fabio Gaetano Santeramo & Emilia Lamonaca, 2019. "The Effects of Non‐tariff Measures on Agri‐food Trade: A Review and Meta‐analysis of Empirical Evidence," Journal of Agricultural Economics, Wiley Blackwell, vol. 70(3), pages 595-617, September.
    14. Bergstrom, John C. & Taylor, Laura O., 2006. "Using meta-analysis for benefits transfer: Theory and practice," Ecological Economics, Elsevier, vol. 60(2), pages 351-360, December.
    15. Zandersen, Marianne & Tol, Richard S.J., 2009. "A meta-analysis of forest recreation values in Europe," Journal of Forest Economics, Elsevier, vol. 15(1-2), pages 109-130, January.
    16. Stefan Hirsch, 2018. "Successful In The Long Run: A Meta†Regression Analysis Of Persistent Firm Profits," Journal of Economic Surveys, Wiley Blackwell, vol. 32(1), pages 23-49, February.
    17. Havranek, Tomas & Irsova, Zuzana & Janda, Karel & Zilberman, David, 2015. "Selective reporting and the social cost of carbon," Energy Economics, Elsevier, vol. 51(C), pages 394-406.
    18. William Nordhaus, 2018. "Evolution of modeling of the economics of global warming: changes in the DICE model, 1992–2017," Climatic Change, Springer, vol. 148(4), pages 623-640, June.
    19. Beltrán, Allan & Alatorre, José Eduardo & Ferrer, Jimy & Galindo, Luis Miguel, 2017. "Efectos potenciales de un impuesto al carbono sobre el producto interno bruto en los países de América Latina: estimaciones preliminares e hipotéticas a partir de un metaanálisis y una función de tran," Documentos de Proyectos 41867, Naciones Unidas Comisión Económica para América Latina y el Caribe (CEPAL).
    20. Pettinotti, Laetitia & de Ayala, Amaia & Ojea, Elena, 2018. "Benefits From Water Related Ecosystem Services in Africa and Climate Change," Ecological Economics, Elsevier, vol. 149(C), pages 294-305.

    More about this item

    Keywords

    citations; criticism; experts; meta-analysis; peers; systematic reviews;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sru:ssewps:2015-24. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: University of Sussex Business School Communications Team (email available below). General contact details of provider: https://edirc.repec.org/data/spessuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.