IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0096920.html
   My bibliography  Save this article

Poor Reliability between Cochrane Reviewers and Blinded External Reviewers When Applying the Cochrane Risk of Bias Tool in Physical Therapy Trials

Author

Listed:
  • Susan Armijo-Olivo
  • Maria Ospina
  • Bruno R da Costa
  • Matthias Egger
  • Humam Saltaji
  • Jorge Fuentes
  • Christine Ha
  • Greta G Cummings

Abstract

Objectives: To test the inter-rater reliability of the RoB tool applied to Physical Therapy (PT) trials by comparing ratings from Cochrane review authors with those of blinded external reviewers. Methods: Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments were conducted independently by 2 reviewers blinded to the RoB ratings reported in the Cochrane reviews. Data on RoB assessments from Cochrane reviews and other characteristics of reviews and trials were extracted. Consensus assessments between the two reviewers were then compared with the RoB ratings from the Cochrane reviews. Agreement between Cochrane and blinded external reviewers was assessed using weighted kappa (κ). Results: In total, 109 trials included in 17 Cochrane reviews were assessed. Inter-rater reliability on the overall RoB assessment between Cochrane review authors and blinded external reviewers was poor (κ = 0.02, 95%CI: −0.06, 0.06]). Inter-rater reliability on individual domains of the RoB tool was poor (median κ = 0.19), ranging from κ = −0.04 (“Other bias”) to κ = 0.62 (“Sequence generation”). There was also no agreement (κ = −0.29, 95%CI: −0.81, 0.35]) in the overall RoB assessment at the meta-analysis level. Conclusions: Risk of bias assessments of RCTs using the RoB tool are not consistent across different research groups. Poor agreement was not only demonstrated at the trial level but also at the meta-analysis level. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to consistently apply the RoB tool and revisions to the tool for different health areas are needed.

Suggested Citation

  • Susan Armijo-Olivo & Maria Ospina & Bruno R da Costa & Matthias Egger & Humam Saltaji & Jorge Fuentes & Christine Ha & Greta G Cummings, 2014. "Poor Reliability between Cochrane Reviewers and Blinded External Reviewers When Applying the Cochrane Risk of Bias Tool in Physical Therapy Trials," PLOS ONE, Public Library of Science, vol. 9(5), pages 1-10, May.
    • Handle: RePEc:plo:pone00:0096920
    DOI: 10.1371/journal.pone.0096920
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0096920
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0096920&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0096920?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Editors The, 2008. "From the Editors," Basic Income Studies, De Gruyter, vol. 2(2), pages 1-3, January.
    2. Editors The, 2008. "From the Editors," Basic Income Studies, De Gruyter, vol. 3(1), pages 1-1, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Anne M Moseley & Prinon Rahman & George A Wells & Joshua R Zadro & Catherine Sherrington & Karine Toupin-April & Lucie Brosseau, 2019. "Agreement between the Cochrane risk of bias tool and Physiotherapy Evidence Database (PEDro) scale: A meta-epidemiological study of randomized controlled trials of physical therapy interventions," PLOS ONE, Public Library of Science, vol. 14(9), pages 1-16, September.
    2. Holger Cramer & Romy Lauche & Hoda Azizi & Gustav Dobos & Jost Langhorst, 2014. "Yoga for Multiple Sclerosis: A Systematic Review and Meta-Analysis," PLOS ONE, Public Library of Science, vol. 9(11), pages 1-11, November.
    3. Maaike M Rademaker & Geerte G J Ramakers & Adriana L Smit & Lotty Hooft & Inge Stegeman, 2020. "The effect of the CONSORT statement on the amount of “unclear” Risk of Bias reporting in Cochrane Systematic Reviews," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-9, July.
    4. Susan Armijo-Olivo & Bruno R da Costa & Greta G Cummings & Christine Ha & Jorge Fuentes & Humam Saltaji & Matthias Egger, 2015. "PEDro or Cochrane to Assess the Quality of Clinical Trials? A Meta-Epidemiological Study," PLOS ONE, Public Library of Science, vol. 10(7), pages 1-14, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rosalie L Tung & Günter K Stahl, 2018. "The tortuous evolution of the role of culture in IB research: What we know, what we don’t know, and where we are headed," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 49(9), pages 1167-1189, December.
    2. Laurent, Catherine E. & Berriet-Solliec, Marielle & Kirsch, Marc & Labarthe, Pierre & Trouve, Aurelie, 2010. "Multifunctionality Of Agriculture, Public Policies And Scientific Evidences: Some Critical Issues Of Contemporary Controversies," APSTRACT: Applied Studies in Agribusiness and Commerce, AGRIMBA, vol. 4(1-2), pages 1-6.
    3. Hsu, Dan K. & Burmeister-Lamp, Katrin & Simmons, Sharon A. & Foo, Maw-Der & Hong, Michelle C. & Pipes, Jesse D., 2019. "“I know I can, but I don't fit”: Perceived fit, self-efficacy, and entrepreneurial intention," Journal of Business Venturing, Elsevier, vol. 34(2), pages 311-326.
    4. Lude, Maximilian & Prügl, Reinhard, 2021. "Experimental studies in family business research," Journal of Family Business Strategy, Elsevier, vol. 12(1).
    5. Batistič, Saša & Černe, Matej & Kaše, Robert & Zupic, Ivan, 2016. "The role of organizational context in fostering employee proactive behavior: The interplay between HR system configurations and relational climates," European Management Journal, Elsevier, vol. 34(5), pages 579-588.
    6. Dan K. Hsu & Johan Wiklund & Richard D. Cotton, 2017. "Success, Failure, and Entrepreneurial Reentry: An Experimental Assessment of the Veracity of Self–Efficacy and Prospect Theory," Entrepreneurship Theory and Practice, , vol. 41(1), pages 19-47, January.
    7. Krueger, Norris & Bogers, Marcel L.A.M. & Labaki, Rania & Basco, Rodrigo, 2021. "Advancing family business science through context theorizing: The case of the Arab world," Journal of Family Business Strategy, Elsevier, vol. 12(1).
    8. Jana Schmutzler & Edward Lorenz, 2018. "Tolerance, agglomeration, and enterprise innovation performance: a multilevel analysis of Latin American regions," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 27(2), pages 243-268.
    9. Choi, James J. & Haisley, Emily & Kurkoski, Jennifer & Massey, Cade, 2017. "Small cues change savings choices," Journal of Economic Behavior & Organization, Elsevier, vol. 142(C), pages 378-395.
    10. Catherine Welch & Eriikka Paavilainen-Mäntymäki & Rebecca Piekkari & Emmanuella Plakoyiannaki, 2022. "Reconciling theory and context: How the case study can set a new agenda for international business research," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 53(1), pages 4-26, February.
    11. Sirola, Nina & Pitesa, Marko, 2018. "The macroeconomic environment and the psychology of work evaluation," Organizational Behavior and Human Decision Processes, Elsevier, vol. 144(C), pages 11-24.
    12. Weber, Ellen & Büttgen, Marion & Bartsch, Silke, 2022. "How to take employees on the digital transformation journey: An experimental study on complementary leadership behaviors in managing organizational change," Journal of Business Research, Elsevier, vol. 143(C), pages 225-238.
    13. Sturt W Manning & Brita Lorentzen & Lynn Welton & Stephen Batiuk & Timothy P Harrison, 2020. "Beyond megadrought and collapse in the Northern Levant: The chronology of Tell Tayinat and two historical inflection episodes, around 4.2ka BP, and following 3.2ka BP," PLOS ONE, Public Library of Science, vol. 15(10), pages 1-38, October.
    14. Milazzo, M.F. & Spina, F. & Primerano, P. & Bart, J.C.J., 2013. "Soy biodiesel pathways: Global prospects," Renewable and Sustainable Energy Reviews, Elsevier, vol. 26(C), pages 579-624.
    15. Ravi KANBUR & Lucas RONCONI, 2018. "Enforcement matters: The effective regulation of labour," International Labour Review, International Labour Organization, vol. 157(3), pages 331-356, September.
    16. Meuleman, Miguel & Wright, Mike, 2011. "Cross-border private equity syndication: Institutional context and learning," Journal of Business Venturing, Elsevier, vol. 26(1), pages 35-48, January.
    17. Ferreira, Manuel Portugal & Li, Dan & Guisinger, Stephen & Serra, Fernando A. Ribeiro, 2009. "Será o ambiente internacional de negócios o contexto efetivo para a pesquisa em negócios internacionais?," RAE - Revista de Administração de Empresas, FGV-EAESP Escola de Administração de Empresas de São Paulo (Brazil), vol. 49(3), July.
    18. Stephanie B Linek & Benedikt Fecher & Sascha Friesike & Marcel Hebing, 2017. "Data sharing as social dilemma: Influence of the researcher’s personality," PLOS ONE, Public Library of Science, vol. 12(8), pages 1-24, August.
    19. Louis Sevitenyi Nkwatoh & Yahya Zakari Abdullahi & Chika Usman Aliyu, 2019. "Past and Current European Monetary Union Crises: Lessons for the Envisaged West African Monetary Union," International Journal of Economics and Financial Issues, Econjournals, vol. 9(4), pages 50-59.
    20. Robert E. Marks, 2010. "Welcome to SAGE Publications," Australian Journal of Management, Australian School of Business, vol. 35(1), pages 3-5, April.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0096920. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.