IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0220565.html
   My bibliography  Save this article

Feasibility and preliminary validity evidence for remote video-based assessment of clinicians in a global health setting

Author

Listed:
  • Katherine A Smith
  • Segolame Setlhare
  • Allan DeCaen
  • Aaron Donoghue
  • Janell L Mensinger
  • Bingqing Zhang
  • Brennan Snow
  • Dikai Zambo
  • Kagiso Ndlovu
  • Ryan Littman-Quinn
  • Farhan Bhanji
  • Peter A Meaney

Abstract

Background: Serious childhood illnesses (SCI), defined as severe pneumonia, severe dehydration, sepsis, and severe malaria, remain major contributors to amenable child mortality worldwide. Inadequate recognition and treatment of SCI are factors that impact child mortality in Botswana. Skills assessments of providers caring for SCI have not been validated in low and middle-income countries. Objective: To establish preliminary inter-rater reliability, validity evidence, and feasibility for an assessment of providers who care for SCI using simulated patients and remote video capture in community clinic settings in Botswana. Methods: This was a pilot study. Four scenarios were developed via a modified Delphi technique and implemented at primary care clinics in Kweneng, Botswana. Sessions were video captured and independently reviewed. Response process and internal structure analysis utilized intra-class correlation (ICC) and Fleiss’ Kappa. A structured log was utilized for feasibility of remote video capture. Results: Eleven subjects participated. Scenarios of Lower Airway Obstruction (ICC = 0.925, 95%CI 0.695–0.998) and Hypovolemic Shock from Severe Dehydration (ICC = 0.892, 95%CI 0.596–0.997) produced excellent ICC among raters while Lower Respiratory Tract Infection (LRTI, ICC = 0, 95%CI -0.034–0.97) and LRTI + Distributive Shock from Sepsis (0.365, 95%CI -0.025–0.967) were poor. Oxygen therapy (0.707), arranging transport (0.706), and fluid administration (0.701) demonstrated substantial task reliability. Conclusions: Initial development of an assessment tool demonstrates many, but not all, criteria for validity evidence. Some scenarios and tasks demonstrate excellent reliability among raters, but others may be limited by manikin design and study implementation. Remote simulation assessment of some skills by clinic-based providers in global health settings is reliable and feasible.

Suggested Citation

  • Katherine A Smith & Segolame Setlhare & Allan DeCaen & Aaron Donoghue & Janell L Mensinger & Bingqing Zhang & Brennan Snow & Dikai Zambo & Kagiso Ndlovu & Ryan Littman-Quinn & Farhan Bhanji & Peter A , 2019. "Feasibility and preliminary validity evidence for remote video-based assessment of clinicians in a global health setting," PLOS ONE, Public Library of Science, vol. 14(8), pages 1-13, August.
    • Handle: RePEc:plo:pone00:0220565
    DOI: 10.1371/journal.pone.0220565
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0220565
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0220565&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0220565?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Graefe, Andreas & Armstrong, J. Scott, 2011. "Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task," International Journal of Forecasting, Elsevier, vol. 27(1), pages 183-195, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Armstrong, J. Scott & Green, Kesten C. & Graefe, Andreas, 2015. "Golden rule of forecasting: Be conservative," Journal of Business Research, Elsevier, vol. 68(8), pages 1717-1731.
    2. Litsiou, Konstantia & Polychronakis, Yiannis & Karami, Azhdar & Nikolopoulos, Konstantinos, 2022. "Relative performance of judgmental methods for forecasting the success of megaprojects," International Journal of Forecasting, Elsevier, vol. 38(3), pages 1185-1196.
    3. Christoph Markmann & Alexander Spickermann & Heiko A. von der Gracht & Alexander Brem, 2021. "Improving the question formulation in Delphi‐like surveys: Analysis of the effects of abstract language and amount of information on response behavior," Futures & Foresight Science, John Wiley & Sons, vol. 3(1), March.
    4. Hanea, A.M. & McBride, M.F. & Burgman, M.A. & Wintle, B.C. & Fidler, F. & Flander, L. & Twardy, C.R. & Manning, B. & Mascaro, S., 2017. "I nvestigate D iscuss E stimate A ggregate for structured expert judgement," International Journal of Forecasting, Elsevier, vol. 33(1), pages 267-279.
    5. Julia A. Minson & Jennifer S. Mueller & Richard P. Larrick, 2018. "The Contingent Wisdom of Dyads: When Discussion Enhances vs. Undermines the Accuracy of Collaborative Judgments," Management Science, INFORMS, vol. 64(9), pages 4177-4192, September.
    6. Pennings, Clint L.P. & van Dalen, Jan & Rook, Laurens, 2019. "Coordinating judgmental forecasting: Coping with intentional biases," Omega, Elsevier, vol. 87(C), pages 46-56.
    7. Christoph Diermann & Arnd Huchzermeier, 2017. "Case Article—Canyon Bicycles: Judgmental Demand Forecasting in Direct Sales," INFORMS Transactions on Education, INFORMS, vol. 17(2), pages 58-62, January.
    8. Bo Cowgill & Eric Zitzewitz, 2015. "Corporate Prediction Markets: Evidence from Google, Ford, and Firm X," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 82(4), pages 1309-1341.
    9. Jukrin Moon & Dongoo Lee & Taesik Lee & Jaemyung Ahn & Jindong Shin & Kyungho Yoon & Dongsik Choi, 2015. "Group Decision Procedure to Model the Dependency Structure of Complex Systems: Framework and Case Study for Critical Infrastructures," Systems Engineering, John Wiley & Sons, vol. 18(4), pages 323-338, July.
    10. Cary Deck & David Porter, 2013. "Prediction Markets In The Laboratory," Journal of Economic Surveys, Wiley Blackwell, vol. 27(3), pages 589-603, July.
    11. Tommaso Ciarli & Alex Coad & Ismael Rafols, 2015. "Quantitative Analysis of Technology Futures: A review of Techniques, Uses and Characteristics," SPRU Working Paper Series 2015-23, SPRU - Science Policy Research Unit, University of Sussex Business School.
    12. Muhammad Ridhuan Tony Lim Abdullah & Saedah Siraj & Zulkipli Ghazali, 2021. "An ISM Approach for Managing Critical Stakeholder Issues Regarding Carbon Capture and Storage (CCS) Deployment in Developing Asian Countries," Sustainability, MDPI, vol. 13(12), pages 1-23, June.
    13. repec:eee:intfor:v:27:y:2011:i:1:p:1-13 is not listed on IDEAS
    14. Apreda, Riccardo & Bonaccorsi, Andrea & dell'Orletta, Felice & Fantoni, Gualtiero, 2019. "Expert forecast and realized outcomes in technology foresight," Technological Forecasting and Social Change, Elsevier, vol. 141(C), pages 277-288.
    15. Jaemyung Ahn & Olivier L. de Weck & Martin Steele, 2014. "Credibility Assessment of Models and Simulations Based on NASA’s Models and Simulation Standard Using the Delphi Method," Systems Engineering, John Wiley & Sons, vol. 17(2), pages 237-248, June.
    16. Griffiths, Frances & Cave, Jonathan & Boardman, Felicity & Ren, Justin & Pawlikowska, Teresa & Ball, Robin & Clarke, Aileen & Cohen, Alan, 2012. "Social networks – The future for health care delivery," Social Science & Medicine, Elsevier, vol. 75(12), pages 2233-2241.
    17. Sung, Ming-Chien & McDonald, David C.J. & Johnson, Johnnie E.V. & Tai, Chung-Ching & Cheah, Eng-Tuck, 2019. "Improving prediction market forecasts by detecting and correcting possible over-reaction to price movements," European Journal of Operational Research, Elsevier, vol. 272(1), pages 389-405.
    18. Rengarajan, Srinath & Moser, Roger & Narayanamurthy, Gopalakrishnan, 2021. "Strategy tools in dynamic environments – An expert-panel study," Technological Forecasting and Social Change, Elsevier, vol. 165(C).
    19. Bolger, Fergus & Rowe, Gene & Belton, Ian & Crawford, Megan M & Hamlin, Iain & Sissons, Aileen & Taylor Browne Lūka, Courtney & Vasilichi, Alexandrina & Wright, George, 2020. "The Simulated Group Response Paradigm: A new approach to the study of opinion change in Delphi and other structured-group techniques," OSF Preprints 4ufzg, Center for Open Science.
    20. von der Gracht, Heiko A. & Hommel, Ulrich & Prokesch, Tobias & Wohlenberg, Holger, 2016. "Testing weighting approaches for forecasting in a Group Wisdom Support System environment," Journal of Business Research, Elsevier, vol. 69(10), pages 4081-4094.
    21. Lian Jian & Rahul Sami, 2012. "Aggregation and Manipulation in Prediction Markets: Effects of Trading Mechanism and Information Distribution," Management Science, INFORMS, vol. 58(1), pages 123-140, January.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0220565. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.