Best-Worst Scaling with many items

Best-worst scaling (BWS) has become so useful that practitioners feel pressure to include ever more items in their experiments. Researchers wanting more items and enough observations of each item by each respondent to support individual respondent-level utility models may greatly increase the burden on respondents, resulting in respondent fatigue and potentially in lower quality responses. Wirth and Wolfrath (2012) proposed two methods for creating BWS designs that allow for large numbers of items and respondent-level utility estimation, Sparse and Express BWS. This study aims to uncover the recommended approach when the goal is recovering individual respondent-level utilities and intends to do so by comparing the relative ability of Sparse and Express BWS to capture the utilities that would have resulted from a full BWS experiment, one with at least three observations of each item by each respondent. The current study repeats previous comparisons of Sparse and Express BWS using a new empirical data set. It also extends previous findings by collecting enough observations from each respondent for both a full experiment and one of the proposed methods, Express BWS and Sparse BWS. The results replicate and extend previous findings regarding the superior ability of the Sparse BWS methodology, relative to Express, to reproduce “known” utilities or utilities that result from a full BWS design.