IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2104.11300.html
   My bibliography  Save this paper

The Crowd Classification Problem: Social Dynamics of Binary Choice Accuracy

Author

Listed:
  • Joshua Becker
  • Douglas Guilbeault
  • Ned Smith

Abstract

Decades of research suggest that information exchange in groups and organizations can reliably improve judgment accuracy in tasks such as financial forecasting, market research, and medical decision-making. However, we show that improving the accuracy of numeric estimates does not necessarily improve the accuracy of decisions. For binary choice judgments, also known as classification tasks--e.g. yes/no or build/buy decisions--social influence is most likely to grow the majority vote share, regardless of the accuracy of that opinion. As a result, initially inaccurate groups become increasingly inaccurate after information exchange even as they signal stronger support. We term this dynamic the "crowd classification problem." Using both a novel dataset as well as a reanalysis of three previous datasets, we study this process in two types of information exchange: (1) when people share votes only, and (2) when people form and exchange numeric estimates prior to voting. Surprisingly, when people exchange numeric estimates prior to voting, the binary choice vote can become less accurate even as the average numeric estimate becomes more accurate. Our findings recommend against voting as a form of decision-making when groups are optimizing for accuracy. For those cases where voting is required, we discuss strategies for managing communication to avoid the crowd classification problem. We close with a discussion of how our results contribute to a broader contingency theory of collective intelligence.

Suggested Citation

  • Joshua Becker & Douglas Guilbeault & Ned Smith, 2021. "The Crowd Classification Problem: Social Dynamics of Binary Choice Accuracy," Papers 2104.11300, arXiv.org.
  • Handle: RePEc:arx:papers:2104.11300
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2104.11300
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Albert E. Mannes, 2009. "Are We Wise About the Wisdom of Crowds? The Use of Group Judgments in Belief Revision," Management Science, INFORMS, vol. 55(8), pages 1267-1279, August.
    2. Felipe A. Csaszar & J. P. Eggers, 2013. "Organizational Decision Making: An Information Aggregation View," Management Science, INFORMS, vol. 59(10), pages 2257-2277, October.
    3. Oliver Schnusenberg, 2006. "The stock market behaviour prior and subsequent to new highs," Applied Financial Economics, Taylor & Francis Journals, vol. 16(6), pages 429-438.
    4. Raj Aggarwal & Brian M. Lucey, 2007. "Psychological barriers in gold prices?," Review of Financial Economics, John Wiley & Sons, vol. 16(2), pages 217-230.
    5. Pavel Atanasov & Phillip Rescober & Eric Stone & Samuel A. Swift & Emile Servan-Schreiber & Philip Tetlock & Lyle Ungar & Barbara Mellers, 2017. "Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls," Management Science, INFORMS, vol. 63(3), pages 691-706, March.
    6. Nofer, Michael & Hinz, Oliver, 2014. "Are Crowds on the Internet Wiser than Experts? The Case of a Stock Prediction Community," Publications of Darmstadt Technical University, Institute for Business Studies (BWL) 69935, Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL).
    7. Abdullah Almaatouq & Alejandro Noriega-Campero & Abdulrahman Alotaibi & P. M. Krafft & Mehdi Moussaid & Alex Pentland, 2020. "Adaptive social networks promote the wisdom of crowds," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 117(21), pages 11379-11386, May.
    8. Joaquin Navajas & Tamara Niella & Gerry Garbulsky & Bahador Bahrami & Mariano Sigman, 2018. "Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds," Nature Human Behaviour, Nature, vol. 2(2), pages 126-132, February.
    9. Asa B. Palley & Jack B. Soll, 2019. "Extracting the Wisdom of Crowds When Information Is Shared," Management Science, INFORMS, vol. 67(5), pages 2291-2309, May.
    10. Ashton, Robert H., 1986. "Combining the judgments of experts: How many and which ones?," Organizational Behavior and Human Decision Processes, Elsevier, vol. 38(3), pages 405-414, December.
    11. Scott E. Page, 2007. "Prologue to The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies," Introductory Chapters, in: The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies, Princeton University Press.
    12. Jesse Shore & Ethan Bernstein & David Lazer, 2015. "Facts and Figuring: An Experimental Investigation of Network Structure and Performance in Information and Solution Spaces," Organization Science, INFORMS, vol. 26(5), pages 1432-1446, October.
    13. Keith Burghardt & William Rand & Michelle Girvan, 2019. "Inferring models of opinion dynamics from aggregated jury data," PLOS ONE, Public Library of Science, vol. 14(7), pages 1-15, July.
    14. Alessandro Arlotto & Stephen E. Chick & Noah Gans, 2014. "Optimal Hiring and Retention Policies for Heterogeneous Workers Who Learn," Management Science, INFORMS, vol. 60(1), pages 110-129, January.
    15. Gabriel Madirolas & Gonzalo G de Polavieja, 2015. "Improving Collective Estimations Using Resistance to Social Influence," PLOS Computational Biology, Public Library of Science, vol. 11(11), pages 1-16, November.
    16. Zhi Da & Xing Huang, 2020. "Harnessing the Wisdom of Crowds," Management Science, INFORMS, vol. 66(5), pages 1847-1867, May.
    17. Julia A. Minson & Jennifer S. Mueller & Richard P. Larrick, 2018. "The Contingent Wisdom of Dyads: When Discussion Enhances vs. Undermines the Accuracy of Collaborative Judgments," Management Science, INFORMS, vol. 64(9), pages 4177-4192, September.
    18. Norman Dalkey & Olaf Helmer, 1963. "An Experimental Application of the DELPHI Method to the Use of Experts," Management Science, INFORMS, vol. 9(3), pages 458-467, April.
    19. Jansen, W. Jos & Jin, Xiaowen & de Winter, Jasper M., 2016. "Forecasting and nowcasting real GDP: Comparing statistical models and subjective forecasts," International Journal of Forecasting, Elsevier, vol. 32(2), pages 411-436.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jon Atwell & Marlon Twyman II, 2023. "Metawisdom of the Crowd: How Choice Within Aided Decision Making Can Make Crowd Wisdom Robust," Papers 2308.15451, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Joshua Aaron Becker & Douglas Guilbeault & Edward Bishop Smith, 2022. "The Crowd Classification Problem: Social Dynamics of Binary-Choice Accuracy," Management Science, INFORMS, vol. 68(5), pages 3949-3965, May.
    2. Joshua Becker & Abdullah Almaatouq & EmH{o}ke-'Agnes Horv'at, 2020. "Network Structures of Collective Intelligence: The Contingent Benefits of Group Discussion," Papers 2009.07202, arXiv.org, revised Mar 2021.
    3. Vincenz Frey & Arnout van de Rijt, 2021. "Social Influence Undermines the Wisdom of the Crowd in Sequential Decision Making," Management Science, INFORMS, vol. 67(7), pages 4273-4286, July.
    4. Boris Maciejovsky & David V. Budescu, 2020. "Too Much Trust in Group Decisions: Uncovering Hidden Profiles by Groups and Markets," Organization Science, INFORMS, vol. 31(6), pages 1497-1514, November.
    5. Jon Atwell & Marlon Twyman II, 2023. "Metawisdom of the Crowd: How Choice Within Aided Decision Making Can Make Crowd Wisdom Robust," Papers 2308.15451, arXiv.org.
    6. Christian Ganser & Marc Keuschnigg, 2018. "Social Influence Strengthens Crowd Wisdom Under Voting," Advances in Complex Systems (ACS), World Scientific Publishing Co. Pte. Ltd., vol. 21(06n07), pages 1-23, September.
    7. Acosta, Hernando & Wu, Dongrui & Forrest, Barrie M., 2010. "Fuzzy experts on recreational vessels, a risk modelling approach for marine invasions," Ecological Modelling, Elsevier, vol. 221(5), pages 850-863.
    8. repec:cup:judgdm:v:14:y:2019:i:3:p:349-363 is not listed on IDEAS
    9. Mavrodiev, Pavlin & Schweitzer, Frank, 2021. "The ambigous role of social influence on the wisdom of crowds: An analytic approach," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 567(C).
    10. Keppo, Jussi & Satopää, Ville A., 2024. "Bayesian herd detection for dynamic data," International Journal of Forecasting, Elsevier, vol. 40(1), pages 285-301.
    11. Tang, Ming & Liao, Huchang, 2024. "Group efficiency and individual fairness tradeoff in making wise decisions," Omega, Elsevier, vol. 124(C).
    12. Rowe, Gene & Wright, George, 1999. "The Delphi technique as a forecasting tool: issues and analysis," International Journal of Forecasting, Elsevier, vol. 15(4), pages 353-375, October.
    13. Jacqueline N. Lane & Misha Teplitskiy & Gary Gray & Hardeep Ranu & Michael Menietti & Eva C. Guinan & Karim R. Lakhani, 2022. "Conservatism Gets Funded? A Field Experiment on the Role of Negative Information in Novel Project Evaluation," Management Science, INFORMS, vol. 68(6), pages 4478-4495, June.
    14. Shardul S. Phadnis, 2019. "Effectiveness of Delphi‐ and scenario planning‐like processes in enabling organizational adaptation: A simulation‐based comparison," Futures & Foresight Science, John Wiley & Sons, vol. 1(2), June.
    15. Satopää, Ville A. & Salikhov, Marat & Tetlock, Philip E. & Mellers, Barbara, 2023. "Decomposing the effects of crowd-wisdom aggregators: The bias–information–noise (BIN) model," International Journal of Forecasting, Elsevier, vol. 39(1), pages 470-485.
    16. Soll, Jack B. & Mannes, Albert E., 2011. "Judgmental aggregation strategies depend on whether the self is involved," International Journal of Forecasting, Elsevier, vol. 27(1), pages 81-102.
    17. Soll, Jack B. & Mannes, Albert E., 2011. "Judgmental aggregation strategies depend on whether the self is involved," International Journal of Forecasting, Elsevier, vol. 27(1), pages 81-102, January.
    18. Joséphine Riemens & Andrée-Anne Lemieux & Samir Lamouri & Léonore Garnier, 2021. "A Delphi-Régnier Study Addressing the Challenges of Textile Recycling in Europe for the Fashion and Apparel Industry," Sustainability, MDPI, vol. 13(21), pages 1-30, October.
    19. Thomas Schultze & Andreas Mojzisch & Stefan Schulz-Hardt, 2019. "Why dyads heed advice less than individuals do," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 14(3), pages 349-363, May.
    20. Philipp Ecken & Richard Pibernik, 2016. "Hit or Miss: What Leads Experts to Take Advice for Long-Term Judgments?," Management Science, INFORMS, vol. 62(7), pages 2002-2021, July.
    21. Jens Witkowski & Rupert Freeman & Jennifer Wortman Vaughan & David M. Pennock & Andreas Krause, 2023. "Incentive-Compatible Forecasting Competitions," Management Science, INFORMS, vol. 69(3), pages 1354-1374, March.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2104.11300. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.