IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0220565.html
   My bibliography  Save this article

Feasibility and preliminary validity evidence for remote video-based assessment of clinicians in a global health setting

Author

Listed:
  • Katherine A Smith
  • Segolame Setlhare
  • Allan DeCaen
  • Aaron Donoghue
  • Janell L Mensinger
  • Bingqing Zhang
  • Brennan Snow
  • Dikai Zambo
  • Kagiso Ndlovu
  • Ryan Littman-Quinn
  • Farhan Bhanji
  • Peter A Meaney

Abstract

Background: Serious childhood illnesses (SCI), defined as severe pneumonia, severe dehydration, sepsis, and severe malaria, remain major contributors to amenable child mortality worldwide. Inadequate recognition and treatment of SCI are factors that impact child mortality in Botswana. Skills assessments of providers caring for SCI have not been validated in low and middle-income countries. Objective: To establish preliminary inter-rater reliability, validity evidence, and feasibility for an assessment of providers who care for SCI using simulated patients and remote video capture in community clinic settings in Botswana. Methods: This was a pilot study. Four scenarios were developed via a modified Delphi technique and implemented at primary care clinics in Kweneng, Botswana. Sessions were video captured and independently reviewed. Response process and internal structure analysis utilized intra-class correlation (ICC) and Fleiss’ Kappa. A structured log was utilized for feasibility of remote video capture. Results: Eleven subjects participated. Scenarios of Lower Airway Obstruction (ICC = 0.925, 95%CI 0.695–0.998) and Hypovolemic Shock from Severe Dehydration (ICC = 0.892, 95%CI 0.596–0.997) produced excellent ICC among raters while Lower Respiratory Tract Infection (LRTI, ICC = 0, 95%CI -0.034–0.97) and LRTI + Distributive Shock from Sepsis (0.365, 95%CI -0.025–0.967) were poor. Oxygen therapy (0.707), arranging transport (0.706), and fluid administration (0.701) demonstrated substantial task reliability. Conclusions: Initial development of an assessment tool demonstrates many, but not all, criteria for validity evidence. Some scenarios and tasks demonstrate excellent reliability among raters, but others may be limited by manikin design and study implementation. Remote simulation assessment of some skills by clinic-based providers in global health settings is reliable and feasible.

Suggested Citation

  • Katherine A Smith & Segolame Setlhare & Allan DeCaen & Aaron Donoghue & Janell L Mensinger & Bingqing Zhang & Brennan Snow & Dikai Zambo & Kagiso Ndlovu & Ryan Littman-Quinn & Farhan Bhanji & Peter A , 2019. "Feasibility and preliminary validity evidence for remote video-based assessment of clinicians in a global health setting," PLOS ONE, Public Library of Science, vol. 14(8), pages 1-13, August.
    • Handle: RePEc:plo:pone00:0220565
    DOI: 10.1371/journal.pone.0220565
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0220565
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0220565&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0220565?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Graefe, Andreas & Armstrong, J. Scott, 2011. "Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task," International Journal of Forecasting, Elsevier, vol. 27(1), pages 183-195, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Prommer, Lisa & Tiberius, Victor & Kraus, Sascha, 2020. "Exploring the future of startup leadership development," Journal of Business Venturing Insights, Elsevier, vol. 14(C).
    2. Armstrong, J. Scott & Green, Kesten C. & Graefe, Andreas, 2015. "Golden rule of forecasting: Be conservative," Journal of Business Research, Elsevier, vol. 68(8), pages 1717-1731.
    3. Litsiou, Konstantia & Polychronakis, Yiannis & Karami, Azhdar & Nikolopoulos, Konstantinos, 2022. "Relative performance of judgmental methods for forecasting the success of megaprojects," International Journal of Forecasting, Elsevier, vol. 38(3), pages 1185-1196.
    4. Keller, Jonas & von der Gracht, Heiko A., 2014. "The influence of information and communication technology (ICT) on future foresight processes — Results from a Delphi survey," Technological Forecasting and Social Change, Elsevier, vol. 85(C), pages 81-92.
    5. Pavel Atanasov & Phillip Rescober & Eric Stone & Samuel A. Swift & Emile Servan-Schreiber & Philip Tetlock & Lyle Ungar & Barbara Mellers, 2017. "Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls," Management Science, INFORMS, vol. 63(3), pages 691-706, March.
    6. Christoph Markmann & Alexander Spickermann & Heiko A. von der Gracht & Alexander Brem, 2021. "Improving the question formulation in Delphi‐like surveys: Analysis of the effects of abstract language and amount of information on response behavior," Futures & Foresight Science, John Wiley & Sons, vol. 3(1), March.
    7. Hanea, A.M. & McBride, M.F. & Burgman, M.A. & Wintle, B.C. & Fidler, F. & Flander, L. & Twardy, C.R. & Manning, B. & Mascaro, S., 2017. "I nvestigate D iscuss E stimate A ggregate for structured expert judgement," International Journal of Forecasting, Elsevier, vol. 33(1), pages 267-279.
    8. Julia A. Minson & Jennifer S. Mueller & Richard P. Larrick, 2018. "The Contingent Wisdom of Dyads: When Discussion Enhances vs. Undermines the Accuracy of Collaborative Judgments," Management Science, INFORMS, vol. 64(9), pages 4177-4192, September.
    9. Pennings, Clint L.P. & van Dalen, Jan & Rook, Laurens, 2019. "Coordinating judgmental forecasting: Coping with intentional biases," Omega, Elsevier, vol. 87(C), pages 46-56.
    10. Christoph Diermann & Arnd Huchzermeier, 2017. "Case Article—Canyon Bicycles: Judgmental Demand Forecasting in Direct Sales," INFORMS Transactions on Education, INFORMS, vol. 17(2), pages 58-62, January.
    11. Bo Cowgill & Eric Zitzewitz, 2015. "Corporate Prediction Markets: Evidence from Google, Ford, and Firm X," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 82(4), pages 1309-1341.
    12. Laura Studen & Victor Tiberius, 2020. "Social Media, Quo Vadis? Prospective Development and Implications," Future Internet, MDPI, vol. 12(9), pages 1-22, August.
    13. Wright, George & Rowe, Gene, 2011. "Group-based judgmental forecasting: An integration of extant knowledge and the development of priorities for a new research agenda," International Journal of Forecasting, Elsevier, vol. 27(1), pages 1-13, January.
    14. Winkler, Jens & Kuklinski, Christian Paul Jian-Wei & Moser, Roger, 2015. "Decision making in emerging markets: The Delphi approach's contribution to coping with uncertainty and equivocality," Journal of Business Research, Elsevier, vol. 68(5), pages 1118-1126.
    15. Jukrin Moon & Dongoo Lee & Taesik Lee & Jaemyung Ahn & Jindong Shin & Kyungho Yoon & Dongsik Choi, 2015. "Group Decision Procedure to Model the Dependency Structure of Complex Systems: Framework and Case Study for Critical Infrastructures," Systems Engineering, John Wiley & Sons, vol. 18(4), pages 323-338, July.
    16. Eksoz, Can & Mansouri, S. Afshin & Bourlakis, Michael, 2014. "Collaborative forecasting in the food supply chain: A conceptual framework," International Journal of Production Economics, Elsevier, vol. 158(C), pages 120-135.
    17. Spickermann, Alexander & Zimmermann, Martin & von der Gracht, Heiko A., 2014. "Surface- and deep-level diversity in panel selection — Exploring diversity effects on response behaviour in foresight," Technological Forecasting and Social Change, Elsevier, vol. 85(C), pages 105-120.
    18. Cary Deck & David Porter, 2013. "Prediction Markets In The Laboratory," Journal of Economic Surveys, Wiley Blackwell, vol. 27(3), pages 589-603, July.
    19. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    20. Tommaso Ciarli & Alex Coad & Ismael Rafols, 2015. "Quantitative Analysis of Technology Futures: A review of Techniques, Uses and Characteristics," SPRU Working Paper Series 2015-23, SPRU - Science Policy Research Unit, University of Sussex Business School.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0220565. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.