IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v68y2022i5p3949-3965.html
   My bibliography  Save this article

The Crowd Classification Problem: Social Dynamics of Binary-Choice Accuracy

Author

Listed:
  • Joshua Aaron Becker

    (UCL School of Management, University College London, London E145AA, United Kingdom)

  • Douglas Guilbeault

    (Haas School of Business, University of California Berkeley, Berkeley, California 94720)

  • Edward Bishop Smith

    (Kellogg School of Management, Northwestern University, Evanston, Illinois 60208)

Abstract

Decades of research suggest that information exchange in groups and organizations can reliably improve judgment accuracy in tasks such as financial forecasting, market research, and medical decision making. However, we show that improving the accuracy of numeric estimates does not necessarily improve the accuracy of decisions. For binary-choice judgments, also known as classification tasks—for example, yes/no or build/buy decisions—social influence is most likely to grow the majority vote share, regardless of the accuracy of that opinion. As a result, initially, inaccurate groups become increasingly inaccurate after information exchange, even as they signal stronger support. We term this dynamic the “crowd classification problem.” Using both a novel data set and a reanalysis of three previous data sets, we study this process in two types of information exchange: (1) when people share votes only, and (2) when people form and exchange numeric estimates prior to voting. Surprisingly, when people exchange numeric estimates prior to voting, the binary-choice vote can become less accurate, even as the average numeric estimate becomes more accurate. Our findings recommend against voting as a form of decision making when groups are optimizing for accuracy. For those cases where voting is required, we discuss strategies for managing communication to avoid the crowd classification problem. We close with a discussion of how our results contribute to a broader contingency theory of collective intelligence.

Suggested Citation

  • Joshua Aaron Becker & Douglas Guilbeault & Edward Bishop Smith, 2022. "The Crowd Classification Problem: Social Dynamics of Binary-Choice Accuracy," Management Science, INFORMS, vol. 68(5), pages 3949-3965, May.
  • Handle: RePEc:inm:ormnsc:v:68:y:2022:i:5:p:3949-3965
    DOI: 10.1287/mnsc.2021.4127
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mnsc.2021.4127
    Download Restriction: no

    File URL: https://libkey.io/10.1287/mnsc.2021.4127?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Albert E. Mannes, 2009. "Are We Wise About the Wisdom of Crowds? The Use of Group Judgments in Belief Revision," Management Science, INFORMS, vol. 55(8), pages 1267-1279, August.
    2. Felipe A. Csaszar & J. P. Eggers, 2013. "Organizational Decision Making: An Information Aggregation View," Management Science, INFORMS, vol. 59(10), pages 2257-2277, October.
    3. Oliver Schnusenberg, 2006. "The stock market behaviour prior and subsequent to new highs," Applied Financial Economics, Taylor & Francis Journals, vol. 16(6), pages 429-438.
    4. Raj Aggarwal & Brian M. Lucey, 2007. "Psychological barriers in gold prices?," Review of Financial Economics, John Wiley & Sons, vol. 16(2), pages 217-230.
    5. Pavel Atanasov & Phillip Rescober & Eric Stone & Samuel A. Swift & Emile Servan-Schreiber & Philip Tetlock & Lyle Ungar & Barbara Mellers, 2017. "Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls," Management Science, INFORMS, vol. 63(3), pages 691-706, March.
    6. Nofer, Michael & Hinz, Oliver, 2014. "Are Crowds on the Internet Wiser than Experts? The Case of a Stock Prediction Community," Publications of Darmstadt Technical University, Institute for Business Studies (BWL) 69935, Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL).
    7. Joaquin Navajas & Tamara Niella & Gerry Garbulsky & Bahador Bahrami & Mariano Sigman, 2018. "Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds," Nature Human Behaviour, Nature, vol. 2(2), pages 126-132, February.
    8. Asa B. Palley & Jack B. Soll, 2019. "Extracting the Wisdom of Crowds When Information Is Shared," Management Science, INFORMS, vol. 67(5), pages 2291-2309, May.
    9. Ashton, Robert H., 1986. "Combining the judgments of experts: How many and which ones?," Organizational Behavior and Human Decision Processes, Elsevier, vol. 38(3), pages 405-414, December.
    10. Scott E. Page, 2007. "Prologue to The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies," Introductory Chapters, in: The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies, Princeton University Press.
    11. Jesse Shore & Ethan Bernstein & David Lazer, 2015. "Facts and Figuring: An Experimental Investigation of Network Structure and Performance in Information and Solution Spaces," Organization Science, INFORMS, vol. 26(5), pages 1432-1446, October.
    12. Payam Aminpour & Steven A. Gray & Antonie J. Jetter & Joshua E. Introne & Alison Singer & Robert Arlinghaus, 2020. "Wisdom of stakeholder crowds in complex social–ecological systems," Nature Sustainability, Nature, vol. 3(3), pages 191-199, March.
    13. Keith Burghardt & William Rand & Michelle Girvan, 2019. "Inferring models of opinion dynamics from aggregated jury data," PLOS ONE, Public Library of Science, vol. 14(7), pages 1-15, July.
    14. Alessandro Arlotto & Stephen E. Chick & Noah Gans, 2014. "Optimal Hiring and Retention Policies for Heterogeneous Workers Who Learn," Management Science, INFORMS, vol. 60(1), pages 110-129, January.
    15. Gabriel Madirolas & Gonzalo G de Polavieja, 2015. "Improving Collective Estimations Using Resistance to Social Influence," PLOS Computational Biology, Public Library of Science, vol. 11(11), pages 1-16, November.
    16. Zhi Da & Xing Huang, 2020. "Harnessing the Wisdom of Crowds," Management Science, INFORMS, vol. 66(5), pages 1847-1867, May.
    17. Julia A. Minson & Jennifer S. Mueller & Richard P. Larrick, 2018. "The Contingent Wisdom of Dyads: When Discussion Enhances vs. Undermines the Accuracy of Collaborative Judgments," Management Science, INFORMS, vol. 64(9), pages 4177-4192, September.
    18. Norman Dalkey & Olaf Helmer, 1963. "An Experimental Application of the DELPHI Method to the Use of Experts," Management Science, INFORMS, vol. 9(3), pages 458-467, April.
    19. Jansen, W. Jos & Jin, Xiaowen & de Winter, Jasper M., 2016. "Forecasting and nowcasting real GDP: Comparing statistical models and subjective forecasts," International Journal of Forecasting, Elsevier, vol. 32(2), pages 411-436.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Joshua Becker & Douglas Guilbeault & Ned Smith, 2021. "The Crowd Classification Problem: Social Dynamics of Binary Choice Accuracy," Papers 2104.11300, arXiv.org.
    2. Joshua Becker & Abdullah Almaatouq & EmH{o}ke-'Agnes Horv'at, 2020. "Network Structures of Collective Intelligence: The Contingent Benefits of Group Discussion," Papers 2009.07202, arXiv.org, revised Mar 2021.
    3. Vincenz Frey & Arnout van de Rijt, 2021. "Social Influence Undermines the Wisdom of the Crowd in Sequential Decision Making," Management Science, INFORMS, vol. 67(7), pages 4273-4286, July.
    4. Boris Maciejovsky & David V. Budescu, 2020. "Too Much Trust in Group Decisions: Uncovering Hidden Profiles by Groups and Markets," Organization Science, INFORMS, vol. 31(6), pages 1497-1514, November.
    5. Jon Atwell & Marlon Twyman II, 2023. "Metawisdom of the Crowd: How Choice Within Aided Decision Making Can Make Crowd Wisdom Robust," Papers 2308.15451, arXiv.org.
    6. Christian Ganser & Marc Keuschnigg, 2018. "Social Influence Strengthens Crowd Wisdom Under Voting," Advances in Complex Systems (ACS), World Scientific Publishing Co. Pte. Ltd., vol. 21(06n07), pages 1-23, September.
    7. Acosta, Hernando & Wu, Dongrui & Forrest, Barrie M., 2010. "Fuzzy experts on recreational vessels, a risk modelling approach for marine invasions," Ecological Modelling, Elsevier, vol. 221(5), pages 850-863.
    8. repec:cup:judgdm:v:14:y:2019:i:3:p:349-363 is not listed on IDEAS
    9. Mavrodiev, Pavlin & Schweitzer, Frank, 2021. "The ambigous role of social influence on the wisdom of crowds: An analytic approach," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 567(C).
    10. Keppo, Jussi & Satopää, Ville A., 2024. "Bayesian herd detection for dynamic data," International Journal of Forecasting, Elsevier, vol. 40(1), pages 285-301.
    11. Tang, Ming & Liao, Huchang, 2024. "Group efficiency and individual fairness tradeoff in making wise decisions," Omega, Elsevier, vol. 124(C).
    12. Rowe, Gene & Wright, George, 1999. "The Delphi technique as a forecasting tool: issues and analysis," International Journal of Forecasting, Elsevier, vol. 15(4), pages 353-375, October.
    13. Jacqueline N. Lane & Misha Teplitskiy & Gary Gray & Hardeep Ranu & Michael Menietti & Eva C. Guinan & Karim R. Lakhani, 2022. "Conservatism Gets Funded? A Field Experiment on the Role of Negative Information in Novel Project Evaluation," Management Science, INFORMS, vol. 68(6), pages 4478-4495, June.
    14. Shardul S. Phadnis, 2019. "Effectiveness of Delphi‐ and scenario planning‐like processes in enabling organizational adaptation: A simulation‐based comparison," Futures & Foresight Science, John Wiley & Sons, vol. 1(2), June.
    15. Satopää, Ville A. & Salikhov, Marat & Tetlock, Philip E. & Mellers, Barbara, 2023. "Decomposing the effects of crowd-wisdom aggregators: The bias–information–noise (BIN) model," International Journal of Forecasting, Elsevier, vol. 39(1), pages 470-485.
    16. Soll, Jack B. & Mannes, Albert E., 2011. "Judgmental aggregation strategies depend on whether the self is involved," International Journal of Forecasting, Elsevier, vol. 27(1), pages 81-102.
    17. Haeussler, Carolin & Vieth, Sabrina, 2022. "A question worth a million: The expert, the crowd, or myself? An investigation of problem solving," Research Policy, Elsevier, vol. 51(3).
    18. Soll, Jack B. & Mannes, Albert E., 2011. "Judgmental aggregation strategies depend on whether the self is involved," International Journal of Forecasting, Elsevier, vol. 27(1), pages 81-102, January.
    19. Joséphine Riemens & Andrée-Anne Lemieux & Samir Lamouri & Léonore Garnier, 2021. "A Delphi-Régnier Study Addressing the Challenges of Textile Recycling in Europe for the Fashion and Apparel Industry," Sustainability, MDPI, vol. 13(21), pages 1-30, October.
    20. Thomas Schultze & Andreas Mojzisch & Stefan Schulz-Hardt, 2019. "Why dyads heed advice less than individuals do," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 14(3), pages 349-363, May.
    21. Philipp Ecken & Richard Pibernik, 2016. "Hit or Miss: What Leads Experts to Take Advice for Long-Term Judgments?," Management Science, INFORMS, vol. 62(7), pages 2002-2021, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:68:y:2022:i:5:p:3949-3965. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.