IDEAS home Printed from https://ideas.repec.org/a/kap/mktlet/v35y2024i4d10.1007_s11002-024-09729-3.html
   My bibliography  Save this article

The Turing test of online reviews: Can we tell the difference between human-written and GPT-4-written online reviews?

Author

Listed:
  • Balázs Kovács

    (Yale University)

Abstract

Online reviews serve as a guide for consumer choice. With advancements in large language models (LLMs) and generative AI, the fast and inexpensive creation of human-like text may threaten the feedback function of online reviews if neither readers nor platforms can differentiate between human-written and AI-generated content. In two experiments, we found that humans cannot recognize AI-written reviews. Even with monetary incentives for accuracy, both Type I and Type II errors were common: human reviews were often mistaken for AI-generated reviews, and even more frequently, AI-generated reviews were mistaken for human reviews. This held true across various ratings, emotional tones, review lengths, and participants’ genders, education levels, and AI expertise. Younger participants were somewhat better at distinguishing between human and AI reviews. An additional study revealed that current AI detectors were also fooled by AI-generated reviews. We discuss the implications of our findings on trust erosion, manipulation, regulation, consumer behavior, AI detection, market structure, innovation, and review platforms.

Suggested Citation

  • Balázs Kovács, 2024. "The Turing test of online reviews: Can we tell the difference between human-written and GPT-4-written online reviews?," Marketing Letters, Springer, vol. 35(4), pages 651-666, December.
    • Handle: RePEc:kap:mktlet:v:35:y:2024:i:4:d:10.1007_s11002-024-09729-3
    DOI: 10.1007/s11002-024-09729-3
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11002-024-09729-3
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s11002-024-09729-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Xinxin Li & Lorin M. Hitt, 2008. "Self-Selection and Information Role of Online Product Reviews," Information Systems Research, INFORMS, vol. 19(4), pages 456-474, December.
    2. Dellarocas, Chrysanthos, 2003. "The Digitization of Word-of-mouth: Promise and Challenges of Online Feedback Mechanisms," Working papers 4296-03, Massachusetts Institute of Technology (MIT), Sloan School of Management.
    3. Uttara M. Ananthakrishnan & Beibei Li & Michael D. Smith, 2020. "A Tangled Web: Should Online Review Portals Display Fraudulent Reviews?," Information Systems Research, INFORMS, vol. 31(3), pages 950-971, September.
    4. Paul A. Pavlou & David Gefen, 2004. "Building Effective Online Marketplaces with Institution-Based Trust," Information Systems Research, INFORMS, vol. 15(1), pages 37-59, March.
    5. Chrysanthos Dellarocas, 2003. "The Digitization of Word of Mouth: Promise and Challenges of Online Feedback Mechanisms," Management Science, INFORMS, vol. 49(10), pages 1407-1424, October.
    6. Dina Mayzlin & Yaniv Dover & Judith Chevalier, 2014. "Promotional Reviews: An Empirical Investigation of Online Review Manipulation," American Economic Review, American Economic Association, vol. 104(8), pages 2421-2455, August.
    7. Nikolay Archak & Anindya Ghose & Panagiotis G. Ipeirotis, 2011. "Deriving the Pricing Power of Product Features by Mining Consumer Reviews," Management Science, INFORMS, vol. 57(8), pages 1485-1509, August.
    8. Sherry He & Brett Hollenbeck & Davide Proserpio, 2022. "The Market for Fake Reviews," Marketing Science, INFORMS, vol. 41(5), pages 896-921, September.
    9. Yi Zhao & Sha Yang & Vishal Narayan & Ying Zhao, 2013. "Modeling Consumer Learning from Online Product Reviews," Marketing Science, INFORMS, vol. 32(1), pages 153-169, May.
    10. Balázs Kovács & Glenn R. Carroll & David W. Lehman, 2014. "Authenticity and Consumer Value Ratings: Empirical Tests from the Restaurant Domain," Organization Science, INFORMS, vol. 25(2), pages 458-478, April.
    11. Zhang, Tao & Li, Gang & Cheng, T.C.E. & Lai, Kin Keung, 2017. "Welfare economics of review information: Implications for the online selling platform owner," International Journal of Production Economics, Elsevier, vol. 184(C), pages 69-79.
    12. Paul A. Pavlou & Angelika Dimoka, 2006. "The Nature and Role of Feedback Text Comments in Online Marketplaces: Implications for Trust Building, Price Premiums, and Seller Differentiation," Information Systems Research, INFORMS, vol. 17(4), pages 392-414, December.
    13. Oded Netzer & Ronen Feldman & Jacob Goldenberg & Moshe Fresko, 2012. "Mine Your Own Business: Market-Structure Surveillance Through Text Mining," Marketing Science, INFORMS, vol. 31(3), pages 521-543, May.
    14. Steven Tadelis, 2016. "Reputation and Feedback Systems in Online Platform Markets," Annual Review of Economics, Annual Reviews, vol. 8(1), pages 321-340, October.
    15. Michael Luca & Georgios Zervas, 2016. "Fake It Till You Make It: Reputation, Competition, and Yelp Review Fraud," Management Science, INFORMS, vol. 62(12), pages 3412-3427, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dominik Gutt & Jürgen Neumann & Steffen Zimmermann & Dennis Kundisch & Jianqing Chen, 2018. "Design of Review Systems - A Strategic Instrument to shape Online Review Behavior and Economic Outcomes," Working Papers Dissertations 42, Paderborn University, Faculty of Business Administration and Economics.
    2. Hui, Xiang & Klein, Tobias & Stahl, Konrad, 2022. "Learning from Online Ratings," CEPR Discussion Papers 17006, C.E.P.R. Discussion Papers.
    3. Marios Kokkodis & Theodoros Lappas & Gerald C. Kane, 2022. "Optional purchase verification in e‐commerce platforms: More representative product ratings and higher quality reviews," Production and Operations Management, Production and Operations Management Society, vol. 31(7), pages 2943-2961, July.
    4. Krügel, Jan Philipp & Paetzel, Fabian, 2024. "The impact of fraud on reputation systems," Games and Economic Behavior, Elsevier, vol. 144(C), pages 329-354.
    5. Tao Lu & May Yuan & Chong (Alex) Wang & Xiaoquan (Michael) Zhang, 2022. "Histogram Distortion Bias in Consumer Choices," Management Science, INFORMS, vol. 68(12), pages 8963-8978, December.
    6. Heeseung Andrew Lee & Angela Aerry Choi & Tianshu Sun & Wonseok Oh, 2021. "Reviewing Before Reading? An Empirical Investigation of Book-Consumption Patterns and Their Effects on Reviews and Sales," Information Systems Research, INFORMS, vol. 32(4), pages 1368-1389, December.
    7. Dominik Gutt & Philipp Herrmann & Mohammad S. Rahman, 2018. "Crowd-Driven Competitive Intelligence: Understanding the Relationship Between Local Market Competition and Online Rating Distributions," Working Papers Dissertations 41, Paderborn University, Faculty of Business Administration and Economics.
    8. Dominik Gutt & Philipp Herrmann & Mohammad S. Rahman, 2019. "Crowd-Driven Competitive Intelligence: Understanding the Relationship Between Local Market Competition and Online Rating Distributions," Information Systems Research, INFORMS, vol. 30(3), pages 980-994, September.
    9. Jorge Mejia & Shawn Mankad & Anandasivam Gopal, 2019. "A for Effort? Using the Crowd to Identify Moral Hazard in New York City Restaurant Hygiene Inspections," Information Systems Research, INFORMS, vol. 30(4), pages 1363-1386, December.
    10. Lingfang (Ivy) Li & Steven Tadelis & Xiaolan Zhou, 2020. "Buying reputation as a signal of quality: Evidence from an online marketplace," RAND Journal of Economics, RAND Corporation, vol. 51(4), pages 965-988, December.
    11. Gesche, Tobias, 2018. "Reference Price Shifts and Customer Antagonism: Evidence from Reviews for Online Auctions," VfS Annual Conference 2018 (Freiburg, Breisgau): Digital Economy 181650, Verein für Socialpolitik / German Economic Association.
    12. Juan Feng & Xin Li & Xiaoquan (Michael) Zhang, 2019. "Online Product Reviews-Triggered Dynamic Pricing: Theory and Evidence," Information Systems Research, INFORMS, vol. 30(4), pages 1107-1123, December.
    13. Jürgen Neumann & Dominik Gutt & Dennis Kundisch, 2018. "The Traveling Reviewer Problem – Exploring the Relationship between Offline Locations and Online Rating Behavior," Working Papers Dissertations 44, Paderborn University, Faculty of Business Administration and Economics.
    14. King, Robert Allen & Racherla, Pradeep & Bush, Victoria D., 2014. "What We Know and Don't Know About Online Word-of-Mouth: A Review and Synthesis of the Literature," Journal of Interactive Marketing, Elsevier, vol. 28(3), pages 167-183.
    15. Colmekcioglu, Nazan & Marvi, Reza & Foroudi, Pantea & Okumus, Fevzi, 2022. "Generation, susceptibility, and response regarding negativity: An in-depth analysis on negative online reviews," Journal of Business Research, Elsevier, vol. 153(C), pages 235-250.
    16. Warut Khern-am-nuai & Karthik Kannan & Hossein Ghasemkhani, 2018. "Extrinsic versus Intrinsic Rewards for Contributing Reviews in an Online Platform," Information Systems Research, INFORMS, vol. 29(4), pages 871-892, December.
    17. Marios Kokkodis, 2021. "Dynamic, Multidimensional, and Skillset-Specific Reputation Systems for Online Work," Information Systems Research, INFORMS, vol. 32(3), pages 688-712, September.
    18. Nachiketa Sahoo & Chrysanthos Dellarocas & Shuba Srinivasan, 2018. "The Impact of Online Product Reviews on Product Returns," Information Systems Research, INFORMS, vol. 29(3), pages 723-738, September.
    19. Yili (Kevin) Hong & Paul A. Pavlou, 2014. "Product Fit Uncertainty in Online Markets: Nature, Effects, and Antecedents," Information Systems Research, INFORMS, vol. 25(2), pages 328-344, June.
    20. Ishita Chakraborty & Minkyung Kim & K. Sudhir, 2019. "Attribute Sentiment Scoring With Online Text Reviews : Accounting for Language Structure and Attribute Self-Selection," Cowles Foundation Discussion Papers 2176R2, Cowles Foundation for Research in Economics, Yale University, revised Jun 2021.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:kap:mktlet:v:35:y:2024:i:4:d:10.1007_s11002-024-09729-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.