IDEAS home Printed from https://ideas.repec.org/a/spr/pharme/v37y2019i11d10.1007_s40273-019-00844-y.html
   My bibliography  Save this article

TECH-VER: A Verification Checklist to Reduce Errors in Models and Improve Their Credibility

Author

Listed:
  • Nasuh C. Büyükkaramikli

    (Erasmus University Rotterdam)

  • Maureen P. M. H. Rutten-van Mölken

    (Erasmus University Rotterdam)

  • Johan L. Severens

    (Erasmus University Rotterdam)

  • Maiwenn Al

    (Erasmus University Rotterdam)

Abstract

Background In health economic literature, checklists or best practice recommendations on model validation/credibility always declare verification of the programmed model as a fundamental step, such as ‘is the model implemented correctly and does the implementation accurately represent the conceptual model?’ However, to date, little operational guidance for the model verification process has been given. In this study, we aimed to create an operational checklist for model users or reviewers to verify the technical implementation of health economic decision analytical models and document their verification efforts. Methods Literature on model validation, verification, programming errors and credibility was reviewed systematically from scientific databases. An initial beta version of the checklist was developed based on the checklists/tests identified from the literature and from authors’ previous modeling/appraisal experience. Next, the first draft checklist was presented to a number of health economists on several occasions and was tested on different models (built in different software, developed by different stakeholders, including drug manufacturers, consultancies or academia), each time leading to an update of the checklist and culminating in the final version of the TECHnical VERification (TECH-VER) checklist, introduced in this paper. Results The TECH-VER necessitates a model reviewer (preferably independent), an executable and transparent model, its input sources, and detailed documentation (e.g. technical report/scientific paper) in which the conceptual model, its implementation, programmed model inputs, and results are reported. The TECH-VER checklist consists of five domains: (1) input calculations; (2) event-state (patient flow) calculations; (3) result calculations; (4) uncertainty analysis calculations; and (5) other overall checks (e.g. validity or interface). The first four domains reflect the verification of the components of a typical health economic model. For these domains, as a prerequisite of verification tests, the reviewer should identify the relevant calculations in the electronic model and assess the provided justifications for the methods used in the identified calculations. For this purpose, we recommend completeness/consistency checks. Afterwards, the verification tests can be conducted for the calculations in each of these stages by checking the correctness of the implementation of these calculations. For this purpose, the following type of tests are recommended in consecutive order: (i) black-box tests, i.e. checking if model calculations are in line with a priori expectations; (ii) white-box testing, i.e. going through the program code details line by line, or cell by cell (recommended for some crucial calculations and if there are some unexpected results from the black-box tests); and (iii) model replication/parallel programming (recommended only in certain situations, and if the issues related to the identified unexpected results from black-box tests could not be resolved through white-box testing). To reduce the time burden of model verification, we suggest a hierarchical order in tests i–iii, where going to the next step is necessary when the previous step fails. Conclusions The TECH-VER checklist is a comprehensive checklist for the technical verification of decision analytical models, aiming to help identify model implementation errors and their root causes while improving the transparency and efficiency of the verification efforts. In addition to external reviews, we consider that the TECH-VER can be used as an internal training and quality control tool for new health economists, while developing their initial models. It is the authors’ aim that the TECH-VER checklist transforms itself to an open-source living document, with possible future versions, or ‘bolt-on’ extensions for specific applications with additional ‘fit-for-purpose’ tests, as well as ‘tips and tricks’ and some demonstrative error examples. For this reason, the TECH-VER checklist and the list of black-box tests created in this paper and a few model verification examples is uploaded to an open access, online platform (github and the website of the institute), where other users will also be able to upload their original verification efforts and tests.

Suggested Citation

  • Nasuh C. Büyükkaramikli & Maureen P. M. H. Rutten-van Mölken & Johan L. Severens & Maiwenn Al, 2019. "TECH-VER: A Verification Checklist to Reduce Errors in Models and Improve Their Credibility," PharmacoEconomics, Springer, vol. 37(11), pages 1391-1408, November.
  • Handle: RePEc:spr:pharme:v:37:y:2019:i:11:d:10.1007_s40273-019-00844-y
    DOI: 10.1007/s40273-019-00844-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s40273-019-00844-y
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s40273-019-00844-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Michael Willis & Pierre Johansen & Andreas Nilsson & Christian Asseburg, 2017. "Validation of the Economic and Health Outcomes Model of Type 2 Diabetes Mellitus (ECHO-T2DM)," PharmacoEconomics, Springer, vol. 35(3), pages 375-396, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Blog mentions

    As found by EconAcademics.org, the blog aggregator for Economics research:
    1. Rita Faria’s journal round-up for 30th December 2019
      by Rita Faria in The Academic Health Economists' Blog on 2019-12-30 12:00:00

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ali Tafazzoli & Odette S. Reifsnider & Leana Bellanca & Jack Ishak & Marc Carrasco & Pal Rakonczai & Matthew Stargardter & Stephan Linden, 2023. "A European multinational cost-effectiveness analysis of empagliflozin in heart failure with reduced ejection fraction," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 24(9), pages 1441-1454, December.
    2. Nasuh C. Buyukkaramikli & Peter Wigfield & Men Thi Hoang, 2021. "A MEA is a MEA is a MEA? Sequential decision making and the impact of different managed entry agreements at the manufacturer and payer level, using a case study for an oncology drug in England," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 22(1), pages 51-73, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sandra Lopes & Pierre Johansen & Mark Lamotte & Phil McEwan & Anamaria-Vera Olivieri & Volker Foos, 2020. "External Validation of the Core Obesity Model to Assess the Cost-Effectiveness of Weight Management Interventions," PharmacoEconomics, Springer, vol. 38(10), pages 1123-1133, October.
    2. Michael Laxy & Verena Maria Schöning & Christoph Kurz & Rolf Holle & Annette Peters & Christa Meisinger & Wolfgang Rathmann & Kristin Mühlenbruch & Katharina Kähm, 2019. "Performance of the UKPDS Outcomes Model 2 for Predicting Death and Cardiovascular Events in Patients with Type 2 Diabetes Mellitus from a German Population-Based Cohort," PharmacoEconomics, Springer, vol. 37(12), pages 1485-1494, December.
    3. Pierre Johansen & Jonas Håkan-Bloch & Aiden R. Liu & Peter G. Bech & Sofie Persson & Lawrence A. Leiter, 2019. "Cost Effectiveness of Once-Weekly Semaglutide Versus Once-Weekly Dulaglutide in the Treatment of Type 2 Diabetes in Canada," PharmacoEconomics - Open, Springer, vol. 3(4), pages 537-550, December.
    4. Joan Gil & Antoni Sicras-Mainar & Eugenio Zucchelli, 2018. "Uncontrolled diabetes and health care utilisation: panel data evidence from Spain," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 19(6), pages 785-795, July.
    5. Michael Willis & Christian Asseburg & April Slee & Andreas Nilsson & Cheryl Neslusan, 2021. "Macrovascular Risk Equations Based on the CANVAS Program," PharmacoEconomics, Springer, vol. 39(4), pages 447-461, April.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:pharme:v:37:y:2019:i:11:d:10.1007_s40273-019-00844-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.