IDEAS home Printed from https://ideas.repec.org/a/sae/sagope/v10y2020i3p2158244020939530.html
   My bibliography  Save this article

Assessing Quality in Systematic Literature Reviews: A Study of Novice Rater Training

Author

Listed:
  • Sandra Acosta
  • Tiberio Garza
  • Hsien-Yuan Hsu
  • Patricia Goodson

Abstract

This study investigated performance variability when graduate students critically appraised original studies from a systematic review. Fourteen doctoral students from different academic programs, with no systematic review experience, received training on the Methodological Quality Questionnaire (MQQ) rating scale. Participants were mostly male (71%) and non-native English speakers (79%). Each rater was randomly assigned one original study to independently assess using the MQQ. Their scores were compared to an expert rater. Statistical analysis comprised the following: percentage of agreement (POA), Kappa coefficient, and Kendall’s tau-b correlation. On the completed MQQ rating scale, 43% of the novice raters had a POA of 78% or higher with the expert rater. From this case study, a guide for improving training on methodological quality assessment was developed. Benefits include the following: (a) developing and supporting critical reasoning as well as problem-solving skills and (b) increasing research skills and competencies in the systematic review process.

Suggested Citation

  • Sandra Acosta & Tiberio Garza & Hsien-Yuan Hsu & Patricia Goodson, 2020. "Assessing Quality in Systematic Literature Reviews: A Study of Novice Rater Training," SAGE Open, , vol. 10(3), pages 21582440209, July.
    • Handle: RePEc:sae:sagope:v:10:y:2020:i:3:p:2158244020939530
    DOI: 10.1177/2158244020939530
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/2158244020939530
    Download Restriction: no
    File URL: https://libkey.io/10.1177/2158244020939530?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Anonymous, 2013. "Introduction to the Issue," Journal of Wine Economics, Cambridge University Press, vol. 8(2), pages 129-130, November.
    2. Littell, Julia H., 2008. "Evidence-based or biased? The quality of published reviews of evidence-based practices," Children and Youth Services Review, Elsevier, vol. 30(11), pages 1299-1317, November.
    3. Anonymous, 2013. "Introduction to the Issue," Journal of Wine Economics, Cambridge University Press, vol. 8(3), pages 243-243, December.
    4. David Moher & Alessandro Liberati & Jennifer Tetzlaff & Douglas G Altman & The PRISMA Group, 2009. "Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement," PLOS Medicine, Public Library of Science, vol. 6(7), pages 1-6, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hurmelinna-Laukkanen, Pia & Yang, Jialei, 2022. "Distinguishing between appropriability and appropriation: A systematic review and a renewed conceptual framing," Research Policy, Elsevier, vol. 51(1).
    2. Cristina Ortega-Rodríguez & Ana Licerán-Gutiérrez & Antonio Luis Moreno-Albarracín, 2020. "Transparency as a Key Element in Accountability in Non-Profit Organizations: A Systematic Literature Review," Sustainability, MDPI, vol. 12(14), pages 1-22, July.
    3. Ian Shemilt & Gareth J Hollands & Theresa M Marteau & Ryota Nakamura & Susan A Jebb & Michael P Kelly & Marc Suhrcke & David Ogilvie, 2013. "Economic Instruments for Population Diet and Physical Activity Behaviour Change: A Systematic Scoping Review," PLOS ONE, Public Library of Science, vol. 8(9), pages 1-1, September.
    4. Ranjana Raghunathan, 2022. "Everyday Intimacies and Inter-Ethnic Relationships: Tracing Entanglements of Gender and Race in Multicultural Singapore," Sociological Research Online, , vol. 27(1), pages 77-94, March.
    5. Balint, T. & Lamperti, F. & Mandel, A. & Napoletano, M. & Roventini, A. & Sapio, A., 2017. "Complexity and the Economics of Climate Change: A Survey and a Look Forward," Ecological Economics, Elsevier, vol. 138(C), pages 252-265.
    6. Lamperti, Francesco & Bosetti, Valentina & Roventini, Andrea & Tavoni, Massimo & Treibich, Tania, 2021. "Three green financial policies to address climate risks," Journal of Financial Stability, Elsevier, vol. 54(C).
    7. Songsore, Emmanuel & Buzzelli, Michael, 2014. "Social responses to wind energy development in Ontario: The influence of health risk perceptions and associated concerns," Energy Policy, Elsevier, vol. 69(C), pages 285-296.
    8. Tapsuwan, Sorada & Polyakov, Maksym & Bark, Rosalind & Nolan, Martin, 2015. "Valuing the Barmah–Millewa Forest and in stream river flows: A spatial heteroskedasticity and autocorrelation consistent (SHAC) approach," Ecological Economics, Elsevier, vol. 110(C), pages 98-105.
    9. Omar Al-Ubaydli & John List & Claire Mackevicius & Min Sok Lee & Dana Suskind, 2019. "How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling," Artefactual Field Experiments 00679, The Field Experiments Website.
    10. Nepomuceno, Marcelo Vinhal & Laroche, Michel, 2015. "The impact of materialism and anti-consumption lifestyles on personal debt and account balances," Journal of Business Research, Elsevier, vol. 68(3), pages 654-664.
    11. Bertschek, Irene & Kesler, Reinhold, 2022. "Let the user speak: Is feedback on Facebook a source of firms’ innovation?," Information Economics and Policy, Elsevier, vol. 60(C).
    12. Avelino, Flor & Wittmayer, Julia M. & Pel, Bonno & Weaver, Paul & Dumitru, Adina & Haxeltine, Alex & Kemp, René & Jørgensen, Michael S. & Bauler, Tom & Ruijsink, Saskia & O'Riordan, Tim, 2019. "Transformative social innovation and (dis)empowerment," Technological Forecasting and Social Change, Elsevier, vol. 145(C), pages 195-206.
    13. Gigi Foster, 2020. "The behavioural economics of government responses to COVID-19," Journal of Behavioral Economics for Policy, Society for the Advancement of Behavioral Economics (SABE), vol. 4(S3), pages 11-43, December.
    14. Audoly, Richard & Vogt-Schilb, Adrien & Guivarch, Céline & Pfeiffer, Alexander, 2018. "Pathways toward zero-carbon electricity required for climate stabilization," Applied Energy, Elsevier, vol. 225(C), pages 884-901.
    15. Gerards, Ruud & Welters, Ricardo, 2016. "Impact of financial pressure on unemployed job search, job find success and job quality," ROA Research Memorandum 008, Maastricht University, Research Centre for Education and the Labour Market (ROA).
    16. Cairns, George & Wright, George & Fairbrother, Peter, 2016. "Promoting articulated action from diverse stakeholders in response to public policy scenarios: A case analysis of the use of ‘scenario improvisation’ method," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 97-108.
    17. Vasile-Daniel Păvăloaia & Elena-Mădălina Teodor & Doina Fotache & Magdalena Danileţ, 2019. "Opinion Mining on Social Media Data: Sentiment Analysis of User Preferences," Sustainability, MDPI, vol. 11(16), pages 1-21, August.
    18. Cailong Xu & Ruidong Li & Wenwen Song & Tingting Wu & Shi Sun & Shuixiu Hu & Tianfu Han & Cunxiang Wu, 2021. "Responses of Branch Number and Yield Component of Soybean Cultivars Tested in Different Planting Densities," Agriculture, MDPI, vol. 11(1), pages 1-12, January.
    19. Romaniuk, Jenni & Nenycz-Thiel, Magda, 2016. "Lapsed buyers' durable brand consideration in emerging markets," Journal of Business Research, Elsevier, vol. 69(9), pages 3645-3651.
    20. Caitlin Robinson & Stefan Bouzarovski & Sarah Lindley, 2018. "Underrepresenting neighbourhood vulnerabilities? The measurement of fuel poverty in England," Environment and Planning A, , vol. 50(5), pages 1109-1127, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:sagope:v:10:y:2020:i:3:p:2158244020939530. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.