Impact of surgical intervention trials on healthcare: A systematic review of assessment methods, healthcare outcomes, and determinants

Background: Frameworks used in research impact evaluation studies vary widely and it remains unclear which methods are most appropriate for evaluating research impact in the field of surgical research. Therefore, we aimed to identify and review the methods used to assess the impact of surgical intervention trials on healthcare and to identify determinants for surgical impact. Methods: We searched journal databases up to March 10, 2020 for papers assessing the impact of surgical effectiveness trials on healthcare. Two researchers independently screened the papers for eligibility and performed a Risk of Bias assessment. Characteristics of both impact papers and trial papers were summarized. Univariate analyses were performed to identify determinants for finding research impact, which was defined as a change in healthcare practice. Results: Sixty-one impact assessments were performed in 37 included impact papers. Some surgical trial papers were evaluated in more than one impact paper, which provides a total of 38 evaluated trial papers. Most impact papers were published after 2010 (n = 29). Medical records (n = 10), administrative databases (n = 22), and physician’s opinion through surveys (n = 5) were used for data collection. Those data were analyzed purely descriptively (n = 3), comparing data before and after publication (n = 29), or through time series analyses (n = 5). Significant healthcare impact was observed 49 times and more often in more recent publications. Having impact was positively associated with using medical records or administrative databases (ref.: surveys), a longer timeframe for impact evaluation and more months between the publication of the trial paper and the impact paper, data collection in North America (ref.: Europe), no economic evaluation of the intervention, finding no significant difference in surgical outcomes, and suggesting de-implementation in the original trial paper. Conclusions and implications: Research impact evaluation receives growing interest, but still a small number of impact papers per year was identified. The analysis showed that characteristics of both surgical trial papers and impact papers were associated with finding research impact. We advise to collect data from either medical records or administrative databases, with an evaluation time frame of at least 4 years since trial publication.