Extracting the Wisdom from the Crowd: A Comparison of Approaches to Aggregating Collective Intelligence

To benefit from crowdsourcing, companies are increasingly required to employ mechanisms for aggregating the multiple opinions generated in this process. Scholars have raised concerns, however, about the currently most popular method used for this purpose - majority-voting. We conduct two studies to compare different aggregation methods and measure their performance. While the first study aims to generate a general understanding of the aggregating approaches by asking general knowledge questions, the second study is employed in the context of idea evaluation. Moreover, by differentiating between different levels of question difficulty or idea quality respectively, we find that the average-confidence approach i) provides the highest percentage of correctly identified answers across different categories of general knowledge questions and ii) is better suited to identify high quality ideas. Our findings both extend the existing literature on aggregation approaches used for collective intelligence and offer practical insights. Since we use a crowd on a commercial crowdsourcing platform, our results offer valuable insights for companies using or planning to use a crowd for collective intelligence.\\