“We measure what we can measure”: Struggles in defining and evaluating institutional review board quality

There has been a persistent lack of clarity regarding how to define and measure the quality of Institutional Review Boards (IRBs). To address this challenge, we interviewed 43 individuals designated as IRB Stakeholders, including leaders in research ethics oversight, policymakers, investigators, research sponsors, and patient advocates, about their views regarding key features of IRB quality and how those features could be measured. We also interviewed 20 U.S. IRB directors (or individuals in similar roles) to learn how their institutions currently define and measure IRB quality and to assess satisfaction with those approaches. We analyzed the interviews, all of which were conducted in 2018, using a modified grounded theory approach. Individuals in the Stakeholder group struggled both to define IRB quality and identify appropriate measures. Those in the Director group gave less abstract and more bounded accounts, offering definitions of quality based on what their institutions currently measure. In identifying core definitional elements of IRB quality, both groups discussed efficiency, compliance, board and staff qualifications, and research facilitation. However, in an important omission by Directors, only Stakeholders named participant protection and thoughtful review as essential elements of IRB quality, despite the centrality of these factors to the very purpose of IRBs. Directors in our sample were largely satisfied with their institutions' current approaches to quality measurement, which included audits of internal processes and regulatory compliance, efficiency tracking, and feedback from board members and researchers. In addition to fleshing out what it means for IRB discretion to be exercised reasonably, adopting proposed metrics related to participant protection outcomes could help IRBs refocus on their core mission and prevent them from falling further into the broader trend of ‘audit culture.’