IDEAS home Printed from https://ideas.repec.org/a/spr/custns/v11y2024i1d10.1007_s40547-024-00143-4.html
   My bibliography  Save this article

Sentiment Analysis in the Age of Generative AI

Author

Listed:
  • Jan Ole Krugmann

    (Technical University of Munich (TUM), TUM School of Management)

  • Jochen Hartmann

    (Technical University of Munich (TUM), TUM School of Management)

Abstract

In the rapidly advancing age of Generative AI, Large Language Models (LLMs) such as ChatGPT stand at the forefront of disrupting marketing practice and research. This paper presents a comprehensive exploration of LLMs’ proficiency in sentiment analysis, a core task in marketing research for understanding consumer emotions, opinions, and perceptions. We benchmark the performance of three state-of-the-art LLMs, i.e., GPT-3.5, GPT-4, and Llama 2, against established, high-performing transfer learning models. Despite their zero-shot nature, our research reveals that LLMs can not only compete with but in some cases also surpass traditional transfer learning methods in terms of sentiment classification accuracy. We investigate the influence of textual data characteristics and analytical procedures on classification accuracy, shedding light on how data origin, text complexity, and prompting techniques impact LLM performance. We find that linguistic features such as the presence of lengthy, content-laden words improve classification performance, while other features such as single-sentence reviews and less structured social media text documents reduce performance. Further, we explore the explainability of sentiment classifications generated by LLMs. The findings indicate that LLMs, especially Llama 2, offer remarkable classification explanations, highlighting their advanced human-like reasoning capabilities. Collectively, this paper enriches the current understanding of sentiment analysis, providing valuable insights and guidance for the selection of suitable methods by marketing researchers and practitioners in the age of Generative AI.

Suggested Citation

  • Jan Ole Krugmann & Jochen Hartmann, 2024. "Sentiment Analysis in the Age of Generative AI," Customer Needs and Solutions, Springer;Institute for Sustainable Innovation and Growth (iSIG), vol. 11(1), pages 1-19, December.
  • Handle: RePEc:spr:custns:v:11:y:2024:i:1:d:10.1007_s40547-024-00143-4
    DOI: 10.1007/s40547-024-00143-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s40547-024-00143-4
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s40547-024-00143-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Artem Timoshenko & John R. Hauser, 2019. "Identifying Customer Needs from User-Generated Content," Marketing Science, INFORMS, vol. 38(1), pages 1-20, January.
    2. Jonah Berger & Grant Packard & Reihane Boghrati & Ming Hsu & Ashlee Humphreys & Andrea Luangrath & Sarah Moore & Gideon Nave & Christopher Olivola & Matthew Rocklage, 2022. "Marketing insights from text analysis," Marketing Letters, Springer, vol. 33(3), pages 365-377, September.
    3. Martin Reisenbichler & Thomas Reutterer & David A. Schweidel & Daniel Dan, 2022. "Frontiers: Supporting Content Marketing with Natural Language Generation," Marketing Science, INFORMS, vol. 41(3), pages 441-452, May.
    4. Arun Rai, 2020. "Explainable AI: from black box to glass box," Journal of the Academy of Marketing Science, Springer, vol. 48(1), pages 137-141, January.
    5. Jianghao Wang & Yichun Fan & Juan Palacios & Yuchen Chai & Nicolas Guetta-Jeanrenaud & Nick Obradovich & Chenghu Zhou & Siqi Zheng, 2022. "Global evidence of expressed sentiment alterations during the COVID-19 pandemic," Nature Human Behaviour, Nature, vol. 6(3), pages 349-358, March.
    6. Peiyao Li & Noah Castelo & Zsolt Katona & Miklos Sarvary, 2024. "Frontiers: Determining the Validity of Large Language Models for Automated Perceptual Analysis," Marketing Science, INFORMS, vol. 43(2), pages 254-266, March.
    7. Hartmann, Jochen & Heitmann, Mark & Siebert, Christian & Schamp, Christina, 2023. "More than a Feeling: Accuracy and Application of Sentiment Analysis," International Journal of Research in Marketing, Elsevier, vol. 40(1), pages 75-87.
    8. Hartmann, Jochen & Huppertz, Juliana & Schamp, Christina & Heitmann, Mark, 2019. "Comparing automated text classification methods," International Journal of Research in Marketing, Elsevier, vol. 36(1), pages 20-38.
    9. Matthijs Meire & Kelly Hewett & Michel Ballings & V. Kumar & Dirk van den Poel, 2019. "The Role of Marketer-Generated Content in Customer Engagement Marketing," Post-Print hal-02509303, HAL.
    10. Hansen, Nele & Kupfer, Ann-Kristin & Hennig-Thurau, Thorsten, 2018. "Brand crises in the digital age: The short- and long-term effects of social media firestorms on consumers and brands," International Journal of Research in Marketing, Elsevier, vol. 35(4), pages 557-574.
    11. Qiang Zhang & Wenbo Wang & Yuxin Chen, 2020. "Frontiers: In-Consumption Social Listening with Moment-to-Moment Unstructured Data: The Case of Movie Appreciation and Live Comments," Marketing Science, INFORMS, vol. 39(2), pages 285-295, March.
    12. Jonah Berger & Grant Packard & Reihane Boghrati & Ming Hsu & Ashlee Humphreys & Andrea Luangrath & Sarah Moore & Gideon Nave & Christopher Olivola & Matthew Rocklage, 2022. "Correction to: Marketing insights from text analysis," Marketing Letters, Springer, vol. 33(3), pages 379-379, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hartmann, Jochen & Heitmann, Mark & Siebert, Christian & Schamp, Christina, 2023. "More than a Feeling: Accuracy and Application of Sentiment Analysis," International Journal of Research in Marketing, Elsevier, vol. 40(1), pages 75-87.
    2. Ming-Hui Huang & Roland T. Rust, 2021. "A strategic framework for artificial intelligence in marketing," Journal of the Academy of Marketing Science, Springer, vol. 49(1), pages 30-50, January.
    3. Alantari, Huwail J. & Currim, Imran S. & Deng, Yiting & Singh, Sameer, 2022. "An empirical comparison of machine learning methods for text-based sentiment analysis of online consumer reviews," International Journal of Research in Marketing, Elsevier, vol. 39(1), pages 1-19.
    4. Yoganathan, Vignesh & Osburg, Victoria-Sophie, 2024. "The mind in the machine: Estimating mind perception's effect on user satisfaction with voice-based conversational agents," Journal of Business Research, Elsevier, vol. 175(C).
    5. Uttara Ananthakrishnan & Davide Proserpio & Siddhartha Sharma, 2023. "I Hear You: Does Quality Improve with Customer Voice?," Marketing Science, INFORMS, vol. 42(6), pages 1143-1161, November.
    6. Morimura, Fumikazu & Sakagawa, Yuji, 2023. "The intermediating role of big data analytics capability between responsive and proactive market orientations and firm performance in the retail industry," Journal of Retailing and Consumer Services, Elsevier, vol. 71(C).
    7. Jonah Berger & Grant Packard & Reihane Boghrati & Ming Hsu & Ashlee Humphreys & Andrea Luangrath & Sarah Moore & Gideon Nave & Christopher Olivola & Matthew Rocklage, 2022. "Marketing insights from text analysis," Marketing Letters, Springer, vol. 33(3), pages 365-377, September.
    8. Fernando, Angeline Gautami & Aw, Eugene Cheng-Xi, 2023. "What do consumers want? A methodological framework to identify determinant product attributes from consumers’ online questions," Journal of Retailing and Consumer Services, Elsevier, vol. 73(C).
    9. Qian, Yang & Ling, Haifeng & Meng, Xiangrui & Jiang, Yuanchun & Chai, Yidong & Liu, Yezheng, 2024. "Voice of the Professional: Acquiring competitive intelligence from large-scale professional generated contents," Journal of Business Research, Elsevier, vol. 180(C).
    10. Christopher Gerling & Stefan Lessmann, 2024. "Leveraging AI and NLP for Bank Marketing: A Systematic Review and Gap Analysis," Papers 2411.14463, arXiv.org.
    11. Jürgensmeier, Lukas & Skiera, Bernd, 2024. "Generative AI for scalable feedback to multimodal exercises," International Journal of Research in Marketing, Elsevier, vol. 41(3), pages 468-488.
    12. Hermann, Erik & Puntoni, Stefano, 2024. "Artificial intelligence and consumer behavior: From predictive to generative AI," Journal of Business Research, Elsevier, vol. 180(C).
    13. de Haan, Evert & Padigar, Manjunath & El Kihal, Siham & Kübler, Raoul & Wieringa, Jaap E., 2024. "Unstructured data research in business: Toward a structured approach," Journal of Business Research, Elsevier, vol. 177(C).
    14. Erik Hermann, 2022. "Leveraging Artificial Intelligence in Marketing for Social Good—An Ethical Perspective," Journal of Business Ethics, Springer, vol. 179(1), pages 43-61, August.
    15. Peiyao Li & Noah Castelo & Zsolt Katona & Miklos Sarvary, 2024. "Frontiers: Determining the Validity of Large Language Models for Automated Perceptual Analysis," Marketing Science, INFORMS, vol. 43(2), pages 254-266, March.
    16. Chen, Changdong, 2024. "How consumers respond to service failures caused by algorithmic mistakes: The role of algorithmic interpretability," Journal of Business Research, Elsevier, vol. 176(C).
    17. Bernhardt, Lea & Dewenter, Ralf & Thomas, Tobias, 2023. "Measuring partisan media bias in US newscasts from 2001 to 2012," European Journal of Political Economy, Elsevier, vol. 78(C).
    18. Robertson, Jeandri & Ferreira, Caitlin & Botha, Elsamari & Oosthuizen, Kim, 2024. "Game changers: A generative AI prompt protocol to enhance human-AI knowledge co-construction," Business Horizons, Elsevier, vol. 67(5), pages 499-510.
    19. Yucheng Zhang & Zhiling Wang & Lin Xiao & Lijun Wang & Pei Huang, 2023. "Discovering the evolution of online reviews: A bibliometric review," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-22, December.
    20. Leah Warfield Smith & Randall Lee Rose & Alex R. Zablah & Heath McCullough & Mohammad “Mike” Saljoughian, 2023. "Examining post-purchase consumer responses to product automation," Journal of the Academy of Marketing Science, Springer, vol. 51(3), pages 530-550, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:custns:v:11:y:2024:i:1:d:10.1007_s40547-024-00143-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.