IDEAS home Printed from https://ideas.repec.org/a/spr/drugsa/v46y2023i1d10.1007_s40264-022-01254-4.html
   My bibliography  Save this article

Development and Evaluation of the Algorithm CErtaInty Tool (ACE-IT) to Assess Electronic Medical Record and Claims-based Algorithms’ Fit for Purpose for Safety Outcomes

Author

Listed:
  • Sonal Singh

    (UMass Chan Medical School
    UMass Chan Medical School)

  • Julie Beyrer

    (Eli Lilly and Company)

  • Xiaofeng Zhou

    (Pfizer, Global Medical Epidemiology)

  • Joel Swerdel

    (Janssen R&D, LLC)

  • Raymond A. Harvey

    (Janssen R&D, LLC)

  • Kenneth Hornbuckle

    (Eli Lilly and Company)

  • Leo Russo

    (Pfizer, Global Medical Epidemiology)

  • Kanwal Ghauri

    (Reagan-Udall Foundation for the FDA)

  • Ivan H. Abi-Elias

    (UMass Chan Medical School)

  • John S. Cox

    (UMass Chan Medical School)

  • Carla Rodriguez-Watson

    (Reagan-Udall Foundation for the FDA)

Abstract

Introduction Electronic health record (EHR) or medical claims-based algorithms (i.e., operational definitions) can be used to define safety outcomes using real-world data. However, existing tools do not allow researchers and decision-makers to adequately appraise whether a particular algorithm is fit for purpose (FFP) to support regulatory decisions on drug safety surveillance. Our objective was to develop a tool to enable regulatory decision-makers and other stakeholders to appraise whether a given algorithm is FFP for a specific decision context. Methods We drafted a set of 77 generic items informed by regulatory guidance documents, existing instruments, and publications. The outcome of ischemic stroke served as an exemplar to inform the development of draft items. The items were designed to be outcome independent. We conducted a three-round online Delphi panel to develop and refine the tool and achieve consensus on items (> 70% agreement) among panel participants composed of regulators, researchers from pharmaceutical organizations, academic clinicians, methodologists, pharmacoepidemiologists, and cardiologists. We conducted a qualitative analysis of panel responses. Five pairs of reviewers independently evaluated two ischemic stroke algorithm validation studies to test its application. We developed a user guide, with explanation and elaboration for each item, guidance on essential and additional elements for user responses, and an illustrative example of a complete assessment. Furthermore, we conducted a 2-h online stakeholder panel of 16 participants from regulatory agencies, academic institutions, and industry. We solicited input on key factors for an FFP assessment, their general reaction to the Algorithm CErtaInty Tool (ACE-IT), limitations of the tool, and its potential use. Results The expert panel reviewed and made changes to the initial list of 77 items. The panel achieved consensus on 38 items, and the final version of the ACE-IT includes 34 items after removal of duplicate items. Applying the tool to two ischemic stroke algorithms demonstrated challenges in its application and identified shared concepts addressed by more than one item. The ACE-IT was viewed positively by the majority of stakeholders. They identified that the tool could serve as an educational resource as well as an information-sharing platform. The time required to complete the assessment was identified as an important limitation. We consolidated items with shared concepts and added a preliminary screen section and a summary assessment box based on their input. The final version of the ACE-IT is a 34-item tool for assessing whether algorithm validation studies on safety outcomes are FFP. It comprises the domains of internal validity (24 items), external validity (seven items), and ethical conduct and reporting of the validation study (three items). The internal validity domain includes sections on objectives, data sources, population, outcomes, design and setting, statistical methods, reference standard, accuracy, and strengths and limitations. The external validity domain includes items that assess the generalizability to a proposed target study. The domain on ethics and transparency includes items on ethical conduct and reporting of the validation study. Conclusion The ACE-IT supports a structured, transparent, and flexible approach for decision-makers to appraise whether electronic health record or medical claims-based algorithms for safety outcomes are FFP for a specific decision context. Reliability and validity testing using a larger sample of participants in other therapeutic areas and further modifications to reduce the time needed to complete the assessment are needed to fully evaluate its utility for regulatory decision-making.

Suggested Citation

  • Sonal Singh & Julie Beyrer & Xiaofeng Zhou & Joel Swerdel & Raymond A. Harvey & Kenneth Hornbuckle & Leo Russo & Kanwal Ghauri & Ivan H. Abi-Elias & John S. Cox & Carla Rodriguez-Watson, 2023. "Development and Evaluation of the Algorithm CErtaInty Tool (ACE-IT) to Assess Electronic Medical Record and Claims-based Algorithms’ Fit for Purpose for Safety Outcomes," Drug Safety, Springer, vol. 46(1), pages 87-97, January.
  • Handle: RePEc:spr:drugsa:v:46:y:2023:i:1:d:10.1007_s40264-022-01254-4
    DOI: 10.1007/s40264-022-01254-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s40264-022-01254-4
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s40264-022-01254-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Pill, Juri, 1971. "The Delphi method: Substance, context, a critique and an annotated bibliography," Socio-Economic Planning Sciences, Elsevier, vol. 5(1), pages 57-71, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zahra Karbasi & Maliheh Kadivar & Reza Safdari & Leila Shahmoradi & Maryam Zahmatkeshan & Somayyeh Zakerabasali & Shahabeddin Abhari & Azadeh Sayarifard, 2020. "Better monitoring of abused children by designing a child abuse surveillance system: Determining national child abuse minimum data set," International Journal of Health Planning and Management, Wiley Blackwell, vol. 35(4), pages 843-851, July.
    2. Zanakis, Stelios H. & Mandakovic, Tomislav & Gupta, Sushil K. & Sahay, Sundeep & Hong, Sungwan, 1995. "A review of program evaluation and fund allocation methods within the service and government sectors," Socio-Economic Planning Sciences, Elsevier, vol. 29(1), pages 59-79, March.
    3. Saud Alshehri & Yacine Rezgui & Haijiang Li, 2015. "Delphi-based consensus study into a framework of community resilience to disaster," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 75(3), pages 2221-2245, February.
    4. Siti Raudhah M. Yusop & Mohamad Sattar Rasul & Ruhizan Mohammad Yasin & Haida Umiera Hashim, 2023. "Identifying and Validating Vocational Skills Domains and Indicators in Classroom Assessment Practices in TVET," Sustainability, MDPI, vol. 15(6), pages 1-27, March.
    5. McCarl, Bruce A. & Musser, Wesley N., 1985. "Modeling Long Run Risk In Production And Investment Decisions," Regional Research Projects > 1985: S-180 Annual Meeting, March 24-27, 1985, Charleston, South Carolina 271799, Regional Research Projects > S-180: An Economic Analysis of Risk Management Strategies for Agricultural Production Firms.
    6. J. P. Sharma & Sunaina Kanojia & Shikha Sachdeva, 2018. "Comparison of Whistle-blower Protection Mechanism of Select Countries," Indian Journal of Corporate Governance, , vol. 11(1), pages 45-68, June.
    7. C. J. Torrecilla-Salinas & O. Troyer & M. J. Escalona & M. Mejías, 2019. "A Delphi-based expert judgment method applied to the validation of a mature Agile framework for Web development projects," Information Technology and Management, Springer, vol. 20(1), pages 9-40, March.
    8. Hyun Baek & Sun-Kyoung Park, 2015. "Sustainable Development Plan for Korea through Expansion of Green IT: Policy Issues for the Effective Utilization of Big Data," Sustainability, MDPI, vol. 7(2), pages 1-21, January.
    9. Scemama, Pierre & Mongruel, Rémi & Kermagoret, Charlène & Bailly, Denis & Carlier, Antoine & Mao, Patrick Le & Vaschalde, et Diane, 2022. "Guidance for stakeholder consultation to support national ecosystem services assessment: A case study from French marine assessment," Ecosystem Services, Elsevier, vol. 54(C).
    10. O'Loughlin, Rosalyn & Kelly, Alan, 2004. "Equity in resource allocation in the Irish health service: A policy Delphi study," Health Policy, Elsevier, vol. 67(3), pages 271-280, March.
    11. Syamsiah Mokhtar & Ruhizan Muhamad Yasin, 2018. "Design of Teaching Influences the Training Transfer Amongst TVET’s Instructors: Fuzzy Delphi Technique," International Journal of Academic Research in Business and Social Sciences, Human Resource Management Academic Research Society, International Journal of Academic Research in Business and Social Sciences, vol. 8(6), pages 1083-1097, June.
    12. Carmody, M.J. & Hardaker, J. Brian & Powell, Roy A. & Everett, R.E., 1984. "An Economic Evaluation of the Footrot Eradication Program in the New England Region of New South Wales," Review of Marketing and Agricultural Economics, Australian Agricultural and Resource Economics Society, vol. 52(01), pages 1-13, April.
    13. Jakob Wachsmuth, 2015. "Cross-sectoral integration in regional adaptation to climate change via participatory scenario development," Climatic Change, Springer, vol. 132(3), pages 387-400, October.
    14. Payne, Katherine & Nicholls, Stuart G. & McAllister, Marion & MacLeod, Rhona & Ellis, Ian & Donnai, Dian & Davies, Linda M., 2007. "Outcome measures for clinical genetics services: A comparison of genetics healthcare professionals and patients' views," Health Policy, Elsevier, vol. 84(1), pages 112-122, November.
    15. Hardaker, J. B., 1982. "Fundamental Aspects Of Risk And Uncertainty In Agriculture," Agrekon, Agricultural Economics Association of South Africa (AEASA), vol. 21(2), October.
    16. Sara Biscaya & Hisham Elkadi, 2023. "A Catalyst Approach for Smart Ecological Urban Corridors at Disused Waterways," Urban Planning, Cogitatio Press, vol. 8(3), pages 406-424.
    17. Unknown, 1998. "References/Literature Cited," Commodity Costs and Returns Estimation Handbook,, Iowa State University.
    18. Sahar Mirzaee & David Fannon & Matthias Ruth, 2019. "A comparison of preference elicitation methods for multi-criteria design decisions about resilient and sustainable buildings," Environment Systems and Decisions, Springer, vol. 39(4), pages 439-453, December.
    19. Adam Stecyk & Marta Sidorkiewicz & Katarzyna Orfin-Tomaszewska, 2021. "Model of Regional Tourism Competitiveness: Fuzzy Multiple-Criteria Approach (FDM-FAHP-PROMETHE II Framework)," European Research Studies Journal, European Research Studies Journal, vol. 0(3), pages 638-662.
    20. Wesley Buckwalter & Andrew Peterson, 2020. "Public attitudes toward allocating scarce resources in the COVID-19 pandemic," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-20, November.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:drugsa:v:46:y:2023:i:1:d:10.1007_s40264-022-01254-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com/economics/journal/40264 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.