IDEAS home Printed from https://ideas.repec.org/a/eee/soceco/v96y2022ics2214804321001488.html
   My bibliography  Save this article

Is natural language processing the cheap charlie of analyzing cheap talk? A horse race between classifiers on experimental communication data

Author

Listed:
  • Tebbe, Eva
  • Wegener, Benjamin

Abstract

We conduct a horse race between various classification algorithms in order to assess whether some algorithms might be more appropriate for classifying experimental communication data than others. We use data reported by various experimental studies involving digital written communication. The effectiveness criterion for comparing the algorithms is based on the agreement between human message-codings and classifications generated by the respective text classification approaches. Our results show that Gradient Boosting Machines are a good choice for separating empty talk from relevant talk messages. This holds (1) independent of the training set size, (2) when the ratio of empty talk to relevant talk is low, and (3) when there are not only two, but three message classes.

Suggested Citation

  • Tebbe, Eva & Wegener, Benjamin, 2022. "Is natural language processing the cheap charlie of analyzing cheap talk? A horse race between classifiers on experimental communication data," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 96(C).
  • Handle: RePEc:eee:soceco:v:96:y:2022:i:c:s2214804321001488
    DOI: 10.1016/j.socec.2021.101808
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S2214804321001488
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.socec.2021.101808?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Jingnan Chen & Daniel Houser, 2017. "Promises and lies: can observers detect deception in written messages," Experimental Economics, Springer;Economic Science Association, vol. 20(2), pages 396-419, June.
    2. Luca Corazzini & Sebastian Kube & Michel André Maréchal & Antonio Nicolò, 2014. "Elections and Deceptions: An Experimental Study on the Behavioral Effects of Democracy," American Journal of Political Science, John Wiley & Sons, vol. 58(3), pages 579-592, July.
    3. Cary Deck & Maroš Servátka & Steven Tucker, 2013. "An examination of the effect of messages on cooperation under double-blind and single-blind payoff procedures," Experimental Economics, Springer;Economic Science Association, vol. 16(4), pages 597-607, December.
    4. Stephen Hansen & Michael McMahon & Andrea Prat, 2018. "Transparency and Deliberation Within the FOMC: A Computational Linguistics Approach," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 133(2), pages 801-870.
    5. Schniter, Eric & Sheremeta, Roman M. & Sznycer, Daniel, 2013. "Building and rebuilding trust with promises and apologies," Journal of Economic Behavior & Organization, Elsevier, vol. 94(C), pages 242-256.
    6. Kuhn, Max, 2008. "Building Predictive Models in R Using the caret Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 28(i05).
    7. Cason, Timothy N. & Lau, Sau-Him Paul & Mui, Vai-Lam, 2019. "Prior interaction, identity, and cooperation in the Inter-group Prisoner's Dilemma," Journal of Economic Behavior & Organization, Elsevier, vol. 166(C), pages 613-629.
    8. Stefan P. Penczynski, 2019. "Using machine learning for communication classification," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 1002-1029, December.
    9. Moellers, Claudia & Normann, Hans-Theo & Snyder, Christopher M., 2017. "Communication in vertical markets: Experimental evidence," International Journal of Industrial Organization, Elsevier, vol. 50(C), pages 214-258.
    10. Christoph Vanberg, 2008. "Why Do People Keep Their Promises? An Experimental Test of Two Explanations -super-1," Econometrica, Econometric Society, vol. 76(6), pages 1467-1480, November.
    11. Timothy Cason & Vai-Lam Mui, 2007. "Communication and coordination in the laboratory collective resistance game," Experimental Economics, Springer;Economic Science Association, vol. 10(3), pages 251-267, September.
    12. Qiyan Ong & Yohanes Riyanto & Steven Sheffrin, 2012. "How does voice matter? Evidence from the ultimatum game," Experimental Economics, Springer;Economic Science Association, vol. 15(4), pages 604-621, December.
    13. Wang, Siyu & Houser, Daniel, 2019. "Demanding or deferring? An experimental analysis of the economic value of communication with attitude," Games and Economic Behavior, Elsevier, vol. 115(C), pages 381-395.
    14. Nick Feltovich & Yasuyo Hamaguchi, 2018. "The Effect of Whistle‐Blowing Incentives on Collusion: An Experimental Study of Leniency Programs," Southern Economic Journal, John Wiley & Sons, vol. 84(4), pages 1024-1049, April.
    15. Morton, Rebecca B. & Ou, Kai & Qin, Xiangdong, 2020. "Reducing the detrimental effect of identity voting: An experiment on intergroup coordination in China," Journal of Economic Behavior & Organization, Elsevier, vol. 174(C), pages 320-331.
    16. Antinyan, Armenak & Corazzini, Luca & D'Agostino, Elena & Pavesi, Filippo, 2023. "Watch your words: An experimental study on communication and the opportunity cost of delegation," Journal of Economic Behavior & Organization, Elsevier, vol. 214(C), pages 216-232.
    17. Joseph Farrell, 1987. "Cheap Talk, Coordination, and Entry," RAND Journal of Economics, The RAND Corporation, vol. 18(1), pages 34-39, Spring.
    18. Charness, Gary & Dufwenberg, Martin, 2003. "Promises & Partnership," Research Papers in Economics 2003:3, Stockholm University, Department of Economics.
    19. Özkes, Ali & Hanaki, Nobuyuki, 2020. "Talkin' Bout Cooperation," Department for Strategy and Innovation Working Paper Series 08/2020, WU Vienna University of Economics and Business.
    20. Isaac, R Mark & Walker, James M, 1988. "Communication and Free-Riding Behavior: The Voluntary Contribution Mechanism," Economic Inquiry, Western Economic Association International, vol. 26(4), pages 585-608, October.
    21. Gary Charness & Martin Dufwenberg, 2006. "Promises and Partnership," Econometrica, Econometric Society, vol. 74(6), pages 1579-1601, November.
    22. Yan Chen & Grace Jeon & Yong-Mi Kim, 2014. "A day without a search engine: an experimental study of online and offline searches," Experimental Economics, Springer;Economic Science Association, vol. 17(4), pages 512-536, December.
    23. Nielsen, Kirby & Bhattacharya, Puja & Kagel, John H. & Sengupta, Arjun, 2019. "Teams promise but do not deliver," Games and Economic Behavior, Elsevier, vol. 117(C), pages 420-432.
    24. Jiabin Wu, 2018. "Indirect higher order beliefs and cooperation," Experimental Economics, Springer;Economic Science Association, vol. 21(4), pages 858-876, December.
    25. Fischer, Christian & Normann, Hans-Theo, 2019. "Collusion and bargaining in asymmetric Cournot duopoly—An experiment," European Economic Review, Elsevier, vol. 111(C), pages 360-379.
    26. Huseyn Ismayilov & Jan Potters, 2016. "Why do promises affect trustworthiness, or do they?," Experimental Economics, Springer;Economic Science Association, vol. 19(2), pages 382-393, June.
    27. Papke, Leslie E & Wooldridge, Jeffrey M, 1996. "Econometric Methods for Fractional Response Variables with an Application to 401(K) Plan Participation Rates," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 11(6), pages 619-632, Nov.-Dec..
    28. Erte Xiao & Daniel Houser, 2005. "Emotion expression in human punishment behavior," Experimental 0504003, University Library of Munich, Germany, revised 18 May 2005.
    29. Daniel Houser & Erte Xiao, 2011. "Classification of natural language messages using a coordination game," Experimental Economics, Springer;Economic Science Association, vol. 14(1), pages 1-14, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Benjamin Wegener, 2021. "How to Analyze Communication Data from Laboratory Experiments Without Being a Machine Learning Specialist," Journal of Economics and Behavioral Studies, AMH International, vol. 13(1), pages 32-56.
    2. Wang, Siyu & Houser, Daniel, 2019. "Demanding or deferring? An experimental analysis of the economic value of communication with attitude," Games and Economic Behavior, Elsevier, vol. 115(C), pages 381-395.
    3. Zakaria Babutsidze & Nobuyuki Hanaki & Adam Zylbersztejn, 2021. "Nonverbal content and trust: An experiment on digital communication," Economic Inquiry, Western Economic Association International, vol. 59(4), pages 1517-1532, October.
    4. Antinyan, Armenak & Corazzini, Luca & D'Agostino, Elena & Pavesi, Filippo, 2023. "Watch your words: An experimental study on communication and the opportunity cost of delegation," Journal of Economic Behavior & Organization, Elsevier, vol. 214(C), pages 216-232.
    5. Daniel Houser & Erte Xiao, 2011. "Classification of natural language messages using a coordination game," Experimental Economics, Springer;Economic Science Association, vol. 14(1), pages 1-14, March.
    6. Jérôme Hergueux & Nicolas Jacquemet & Stéphane Luchini & Jason F. Shogren, 2022. "Leveraging the Honor Code: Public Goods Contributions under Oath," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 81(3), pages 591-616, March.
    7. Zakaria Babutsidze & Nobuyuki Hanaki & Adam Zylbersztejn, 2020. "Nonverbal content and swift trust: An experiment on digital communication," Working Papers 2008, Groupe d'Analyse et de Théorie Economique Lyon St-Étienne (GATE Lyon St-Étienne), Université de Lyon.
    8. Eric Schniter & Roman M. Sheremeta, 2014. "Predictable and Predictive Emotions: Explaining Cheap Signals and Trust Re-Extension," Working Papers 14-07, Chapman University, Economic Science Institute.
    9. Jingnan Chen & Daniel Houser, 2017. "Promises and lies: can observers detect deception in written messages," Experimental Economics, Springer;Economic Science Association, vol. 20(2), pages 396-419, June.
    10. Nobuyuki Hanaki & Ali I. Ozkes, 2023. "Strategic environment effect and communication," Experimental Economics, Springer;Economic Science Association, vol. 26(3), pages 588-621, July.
    11. Zakaria Babutsidze & Nobuyuki Hanaki & Adam Zylbersztejn, 2019. "Digital Communication and Swift Trust," Post-Print halshs-02409314, HAL.
    12. Miettinen, Topi, 2013. "Promises and conventions – An approach to pre-play agreements," Games and Economic Behavior, Elsevier, vol. 80(C), pages 68-84.
    13. Andres, Maximilian & Bruttel, Lisa & Friedrichsen, Jana, 2023. "How communication makes the difference between a cartel and tacit collusion: A machine learning approach," European Economic Review, Elsevier, vol. 152(C).
    14. Andres, Maximilian & Bruttel, Lisa & Friedrichsen, Jana, 2021. "How do sanctions work? The choice between cartel formation and tacit collusion," VfS Annual Conference 2021 (Virtual Conference): Climate Economics 242372, Verein für Socialpolitik / German Economic Association.
    15. Giovanni Di Bartolomeo & Martin Dufwenberg & Stefano Papa, 2023. "Promises and partner-switch," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 9(1), pages 77-89, June.
    16. Hu, Youxin & Kagel, John & Yang, Huanxing & Zhang, Lan, 2020. "The effects of pre-play communication in a coordination game with incomplete information," Journal of Economic Behavior & Organization, Elsevier, vol. 176(C), pages 403-415.
    17. Koessler, Ann-Kathrin, 2022. "Pledges and how social influence shapes their effectiveness," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 98(C).
    18. Davies, Elwyn & Fafchamps, Marcel, 2017. "Pledging, Praising and Shaming: Experimental Labour Markets in Ghana," IZA Discussion Papers 10520, Institute of Labor Economics (IZA).
    19. Konstantinos Georgalos & John Hey, 2020. "Testing for the emergence of spontaneous order," Experimental Economics, Springer;Economic Science Association, vol. 23(3), pages 912-932, September.
    20. Gawn, Glynis & Innes, Robert, 2018. "Language and lies," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 74(C), pages 167-176.

    More about this item

    Keywords

    Laboratory experiments; Communication; Cheap talk; Classification of natural language messages; Machine learning;
    All these keywords.

    JEL classification:

    • B49 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Other
    • C88 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Other Computer Software
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:soceco:v:96:y:2022:i:c:s2214804321001488. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/inca/620175 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.