Harnessing Expert Judgment to Support Clinical Decisions When the Evidence Base Is Weak

Purpose. In the process of developing an evidence-based decision dashboard to support treatment decisions for patients with newly diagnosed prostate cancer, we found that the clinical evidence base is insufficient to provide high-quality comparative outcome data. We therefore sought to determine if clinically acceptable outcome estimates could be created using a modified version of the Sheffield Elicitation Framework (SHELF), a formal method for eliciting judgments regarding probability distributions of expected decision outcomes. Methods. We asked a panel of 3 urologists, 4 radiation oncologists, and 2 medical oncologists to estimate the probabilities of 11 treatment outcomes based on their clinical experience and an annotated evidence summary. The estimates were elicited using a Microsoft Excel spreadsheet containing a self-guided, adapted version of the SHELF Roulette method distributed via email. We created combined outcome estimates by taking the mean values of the panel members’ upper and lower 95% bounds for each outcome. The combined estimates were then distributed via email to the panel for final approval. Results. Eight of the 9 responses were judged to be correct applications of the SHELF method and included in the combined outcome estimates. The final set of outcome estimates was unanimously accepted by the clinician panel members and used to create a decision dashboard suitable for clinical use and evaluation. Conclusions. Many important health care decisions need to be made in situations where the evidence base is inadequate. Use of a formal protocol for eliciting expert judgments is feasible and can be used to promote evidence-based practice by providing a powerful tool to facilitate the combination of professional judgment with research evidence and patient preferences to guide clinical decisions.