Evidence Following Conditional NICE Technology Appraisal Recommendations: A Critical Analysis of Methods, Quality and Risk of Bias

Background The National Institute for Health and Care Excellence (NICE) may approve health technologies on condition of more evidence generated only in research (OiR) or only with research (OwR). NICE specifies the information needed to comply with its request, although it may not necessarily guarantee good quality and timely evidence for re-appraisal, before reaching a final decision. Aim This study aimed to critically appraise the methods, quality and risk of bias of evidence generated in response to NICE OiR and OwR technology appraisal (TA) and highly specialised technologies (HSTs) recommendations. Methods NICE TAs (between March 2000 and September 2020) and HST evaluations (to October 2023) of medicines were reviewed. Conditional recommendations were analysed to identify the evidence requested by NICE for re-appraisal. The new evidence was analysed for compliance with NICE’s request and assessed using the Cochrane Collaboration’s tools for risk of bias in randomised trials and the ROBINS-I tool for non-randomised evidence. Results NICE made 54 conditional recommendations from TAs (13 OiR and 41 OwR) and five conditional recommendations for HSTs (all OwR). Of these, 16 TAs presented additional evidence for re-appraisal (9 OiR [69%] and 7 OwR [17%]) and three HSTs (3 OwR [60%]). Two of the nine re-appraised TAs with OiR recommendation and four of the seven OwR complied fully with NICE’s request for further evidence, while all three from the HSTs complied. The majority of re-appraised TAs and HSTs included evidence that was deemed to be at serious, high, moderate or unclear risk of bias. Among the 26 randomised controlled trials from TAs assessed, eight were categorised as having low risk of bias in all domains and ten had at least one domain as a high risk of bias. Reporting was unclear for the remainder. Twenty-two non-randomised studies, primarily single-arm studies, were susceptible to biases mostly due to the selection of participants and to confounding. Two HSTs provided evidence from randomised controlled trials which were classified as unclear or high risk of bias. All non-randomised evidence from HSTs were categorised as moderate or serious risk of bias. Conclusions There is widespread non-compliance with agreed data requests and important variation in the quality of evidence submitted in response to NICE conditional approval recommendations. Quality standards ought to be stipulated in respect to evidence contributing to re-appraisals following NICE conditional approval recommendations.