IDEAS home Printed from https://ideas.repec.org/a/gam/jftint/v15y2023i12p375-d1286429.html
   My bibliography  Save this article

A Structured Narrative Prompt for Prompting Narratives from Large Language Models: Sentiment Assessment of ChatGPT-Generated Narratives and Real Tweets

Author

Listed:
  • Christopher J. Lynch

    (Virginia, Modeling, Analysis, and Simulation Center, Old Dominion University, 1030 University Blvd., Suffolk, VA 23435, USA)

  • Erik J. Jensen

    (Computational Modeling and Simulation Engineering Department, Old Dominion University, Norfolk, VA 23508, USA)

  • Virginia Zamponi

    (Virginia, Modeling, Analysis, and Simulation Center, Old Dominion University, 1030 University Blvd., Suffolk, VA 23435, USA)

  • Kevin O’Brien

    (Virginia, Modeling, Analysis, and Simulation Center, Old Dominion University, 1030 University Blvd., Suffolk, VA 23435, USA)

  • Erika Frydenlund

    (Virginia, Modeling, Analysis, and Simulation Center, Old Dominion University, 1030 University Blvd., Suffolk, VA 23435, USA)

  • Ross Gore

    (Virginia, Modeling, Analysis, and Simulation Center, Old Dominion University, 1030 University Blvd., Suffolk, VA 23435, USA)

Abstract

Large language models (LLMs) excel in providing natural language responses that sound authoritative, reflect knowledge of the context area, and can present from a range of varied perspectives. Agent-based models and simulations consist of simulated agents that interact within a simulated environment to explore societal, social, and ethical, among other, problems. Simulated agents generate large volumes of data and discerning useful and relevant content is an onerous task. LLMs can help in communicating agents’ perspectives on key life events by providing natural language narratives. However, these narratives should be factual, transparent, and reproducible. Therefore, we present a structured narrative prompt for sending queries to LLMs, we experiment with the narrative generation process using OpenAI’s ChatGPT, and we assess statistically significant differences across 11 Positive and Negative Affect Schedule (PANAS) sentiment levels between the generated narratives and real tweets using chi-squared tests and Fisher’s exact tests. The narrative prompt structure effectively yields narratives with the desired components from ChatGPT. In four out of forty-four categories, ChatGPT generated narratives which have sentiment scores that were not discernibly different, in terms of statistical significance (alpha level α = 0.05 ), from the sentiment expressed in real tweets. Three outcomes are provided: (1) a list of benefits and challenges for LLMs in narrative generation; (2) a structured prompt for requesting narratives of an LLM chatbot based on simulated agents’ information; (3) an assessment of statistical significance in the sentiment prevalence of the generated narratives compared to real tweets. This indicates significant promise in the utilization of LLMs for helping to connect a simulated agent’s experiences with real people.

Suggested Citation

  • Christopher J. Lynch & Erik J. Jensen & Virginia Zamponi & Kevin O’Brien & Erika Frydenlund & Ross Gore, 2023. "A Structured Narrative Prompt for Prompting Narratives from Large Language Models: Sentiment Assessment of ChatGPT-Generated Narratives and Real Tweets," Future Internet, MDPI, vol. 15(12), pages 1-36, November.
    • Handle: RePEc:gam:jftint:v:15:y:2023:i:12:p:375-:d:1286429
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1999-5903/15/12/375/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1999-5903/15/12/375/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Robert Axelrod, 1997. "Advancing the Art of Simulation in the Social Sciences," Working Papers 97-05-048, Santa Fe Institute.
    2. Spyros Makridakis & Fotios Petropoulos & Yanfei Kang, 2023. "Large Language Models: Their Success and Impact," Forecasting, MDPI, vol. 5(3), pages 1-14, August.
    3. Nicole Gross, 2023. "What ChatGPT Tells Us about Gender: A Cautionary Tale about Performativity and Gender Biases in AI," Social Sciences, MDPI, vol. 12(8), pages 1-15, August.
    4. Keiki Takadama & Tetsuro Kawai & Yuhsuke Koyama, 2008. "Micro- and Macro-Level Validation in Agent-Based Simulation: Reproduction of Human-Like Behaviors and Thinking in a Sequential Bargaining Game," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 11(2), pages 1-9.
    5. Edisa Lozić & Benjamin Štular, 2023. "Fluent but Not Factual: A Comparative Analysis of ChatGPT and Other AI Chatbots’ Proficiency and Originality in Scientific Writing for Humanities," Future Internet, MDPI, vol. 15(10), pages 1-26, October.
    6. Zoltán Szabó & Vilmos Bilicki, 2023. "A New Approach to Web Application Security: Utilizing GPT Language Models for Source Code Inspection," Future Internet, MDPI, vol. 15(10), pages 1-27, September.
    7. Daniel Kornhauser & Uri Wilensky & William Rand, 2009. "Design Guidelines for Agent Based Model Visualization," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 12(2), pages 1-1.
    8. Eva A. M. van Dis & Johan Bollen & Willem Zuidema & Robert van Rooij & Claudi L. Bockting, 2023. "ChatGPT: five priorities for research," Nature, Nature, vol. 614(7947), pages 224-226, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chiarello, Filippo & Giordano, Vito & Spada, Irene & Barandoni, Simone & Fantoni, Gualtiero, 2024. "Future applications of generative large language models: A data-driven case study on ChatGPT," Technovation, Elsevier, vol. 133(C).
    2. Evangelos Katsamakas & Oleg V. Pavlov & Ryan Saklad, 2024. "Artificial intelligence and the transformation of higher education institutions," Papers 2402.08143, arXiv.org.
    3. Celiktutan, Begum & Klesse, Anne-Kathrin & Tuk, Mirjam A., 2024. "Acceptability lies in the eye of the beholder: Self-other biases in GenAI collaborations," International Journal of Research in Marketing, Elsevier, vol. 41(3), pages 496-512.
    4. Giannoccaro, Ilaria, 2015. "Adaptive supply chains in industrial districts: A complexity science approach focused on learning," International Journal of Production Economics, Elsevier, vol. 170(PB), pages 576-589.
    5. Flaminio Squazzoni, 2010. "The impact of agent-based models in the social sciences after 15 years of incursions," History of Economic Ideas, Fabrizio Serra Editore, Pisa - Roma, vol. 18(2), pages 197-234.
    6. Pietro Terna, 2000. "Sum: A Surprising (Un)Realistic Market - Building A Simple Stock Market Structure With Swarm," Computing in Economics and Finance 2000 173, Society for Computational Economics.
    7. Emilio Ferrara, 2024. "GenAI against humanity: nefarious applications of generative artificial intelligence and large language models," Journal of Computational Social Science, Springer, vol. 7(1), pages 549-569, April.
    8. Newbery, Robert & Lean, Jonathan & Moizer, Jonathan & Haddoud, Mohamed, 2018. "Entrepreneurial identity formation during the initial entrepreneurial experience: The influence of simulation feedback and existing identity," Journal of Business Research, Elsevier, vol. 85(C), pages 51-59.
    9. Lilian N. Alessa & Melinda Laituri & C. Michael Barton, 2006. "An "All Hands" Call to the Social Science Community: Establishing a Community Framework for Complexity Modeling Using Agent Based Models and Cyberinfrastructure," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 9(4), pages 1-6.
    10. Shahad Al-Khalifa & Fatima Alhumaidhi & Hind Alotaibi & Hend S. Al-Khalifa, 2023. "ChatGPT across Arabic Twitter: A Study of Topics, Sentiments, and Sarcasm," Data, MDPI, vol. 8(11), pages 1-19, November.
    11. Meysam Alizadeh & Maël Kubli & Zeynab Samei & Shirin Dehghani & Mohammadmasiha Zahedivafa & Juan D. Bermeo & Maria Korobeynikova & Fabrizio Gilardi, 2025. "Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning," Journal of Computational Social Science, Springer, vol. 8(1), pages 1-25, February.
    12. Healey, Mark P. & Bleda, Mercedes & Querbes, Adrien, 2021. "Opportunity evaluation in teams: A social cognitive model," Journal of Business Venturing, Elsevier, vol. 36(4).
    13. Nur Ashfaraliana Abd Hadi & Faizah Mohamad & Elia Md Johar & Zaemah Abdul Kadir, 2024. "Exploring the Acceptance of ChatGPT as an Assisting Tool in Academic Writing among ESL Undergraduate Students," International Journal of Research and Innovation in Social Science, International Journal of Research and Innovation in Social Science (IJRISS), vol. 8(10), pages 2886-2901, October.
    14. Edoardo Mollona & David Hales, 2006. "Knowledge-Based Jobs and the Boundaries of Firms Agent-based Simulation of Firms Learning and Workforce Skill Set Dynamics," Computational Economics, Springer;Society for Computational Economics, vol. 27(1), pages 35-62, February.
    15. Shu-Heng Chen & Bin-Tzong Chie & Ying-Fang Kao & Ragupathy Venkatachalam, 2019. "Agent-Based Modeling of a Non-tâtonnement Process for the Scarf Economy: The Role of Learning," Computational Economics, Springer;Society for Computational Economics, vol. 54(1), pages 305-341, June.
    16. Nigel Gilbert & Pietro Terna, 2000. "How to build and use agent-based models in social science," Mind & Society: Cognitive Studies in Economics and Social Sciences, Springer;Fondazione Rosselli, vol. 1(1), pages 57-72, March.
    17. Shu-Heng Chen & Chia-Hsuan Yeh, 1999. "Evolving Traders and the Faculty of the Business School: A New Architecture of the Artificial Stock Market," Computing in Economics and Finance 1999 613, Society for Computational Economics.
    18. Carlo Bianchi & Pasquale Cirillo & Mauro Gallegati & Pietro Vagliasindi, 2007. "Validating and Calibrating Agent-Based Models: A Case Study," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 245-264, October.
    19. Federico Cosenz & Guido Noto, 2016. "Applying System Dynamics Modelling to Strategic Management: A Literature Review," Systems Research and Behavioral Science, Wiley Blackwell, vol. 33(6), pages 703-741, November.
    20. Johanna Habib & François de Corbière, 2014. "Proposition et illustration d'un design de recherche combinant étude de cas et simulation multi-agents pour explorer les processus émergents en Systèmes d'information," Post-Print hal-01130779, HAL.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jftint:v:15:y:2023:i:12:p:375-:d:1286429. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.