IDEAS home Printed from https://ideas.repec.org/a/spr/jcsosc/v8y2025i1d10.1007_s42001-024-00338-8.html
   My bibliography  Save this article

In generative AI we trust: can chatbots effectively verify political information?

Author

Listed:
  • Elizaveta Kuznetsova

    (Weizenbaum Institute for the Networked Society)

  • Mykola Makhortykh

    (University of Bern)

  • Victoria Vziatysheva

    (University of Bern)

  • Martha Stolze

    (Weizenbaum Institute for the Networked Society)

  • Ani Baghumyan

    (University of Bern)

  • Aleksandra Urman

    (University of Zurich)

Abstract

This article presents a comparative analysis of the potential of two large language model (LLM)-based chatbots—ChatGPT and Bing Chat (recently rebranded to Microsoft Copilot)—to detect veracity of political information. We use AI auditing methodology to investigate how chatbots evaluate true, false, and borderline statements on five topics: COVID-19, Russian aggression against Ukraine, the Holocaust, climate change, and LGBTQ + -related debates. We compare how the chatbots respond in high- and low-resource languages by using prompts in English, Russian, and Ukrainian. Furthermore, we explore chatbots’ ability to evaluate statements according to political communication concepts of disinformation, misinformation, and conspiracy theory, using definition-oriented prompts. We also systematically test how such evaluations are influenced by source attribution. The results show high potential of ChatGPT for the baseline veracity evaluation task, with 72% of the cases evaluated in accordance with the baseline on average across languages without pre-training. Bing Chat evaluated 67% of the cases in accordance with the baseline. We observe significant disparities in how chatbots evaluate prompts in high- and low-resource languages and how they adapt their evaluations to political communication concepts with ChatGPT providing more nuanced outputs than Bing Chat. These findings highlight the potential of LLM-based chatbots in tackling different forms of false information in online environments, but also point to the substantial variation in terms of how such potential is realized due to specific factors (e.g. language of the prompt or the topic).

Suggested Citation

  • Elizaveta Kuznetsova & Mykola Makhortykh & Victoria Vziatysheva & Martha Stolze & Ani Baghumyan & Aleksandra Urman, 2025. "In generative AI we trust: can chatbots effectively verify political information?," Journal of Computational Social Science, Springer, vol. 8(1), pages 1-31, February.
    • Handle: RePEc:spr:jcsosc:v:8:y:2025:i:1:d:10.1007_s42001-024-00338-8
    DOI: 10.1007/s42001-024-00338-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s42001-024-00338-8
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s42001-024-00338-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Bradshaw, Samantha, 2019. "Disinformation optimised: gaming search engine algorithms to amplify junk news," Internet Policy Review: Journal on Internet Regulation, Alexander von Humboldt Institute for Internet and Society (HIIG), Berlin, vol. 8(4), pages 1-24.
    2. Sergei A. Samoilenko & John Cook, 2024. "Developing an Ad Hominem typology for classifying climate misinformation," Climate Policy, Taylor & Francis Journals, vol. 24(1), pages 138-151, January.
    3. Antonio A. Arechar & Jennifer Allen & Adam J. Berinsky & Rocky Cole & Ziv Epstein & Kiran Garimella & Andrew Gully & Jackson G. Lu & Robert M. Ross & Michael N. Stagnaro & Yunhao Zhang & Gordon Pennyc, 2023. "Understanding and combatting misinformation across 16 countries on six continents," Nature Human Behaviour, Nature, vol. 7(9), pages 1502-1513, September.
    4. Alberto Barchetti & Emma Neybert & Susan Powell Mantel & Frank R. Kardes, 2022. "The Half-Truth Effect and Its Implications for Sustainability," Sustainability, MDPI, vol. 14(11), pages 1-19, June.
    5. Cheuk Hang Au & Kevin K. W. Ho & Dickson K.W. Chiu, 2022. "The Role of Online Misinformation and Fake News in Ideological Polarization: Barriers, Catalysts, and Implications," Information Systems Frontiers, Springer, vol. 24(4), pages 1331-1354, August.
    6. Antonio A. Arechar & Jennifer Allen & Adam J. Berinsky & Rocky Cole & Ziv Epstein & Kiran Garimella & Andrew Gully & Jackson G. Lu & Robert M. Ross & Michael N. Stagnaro & Yunhao Zhang & Gordon Pennyc, 2023. "Author Correction: Understanding and combatting misinformation across 16 countries on six continents," Nature Human Behaviour, Nature, vol. 7(10), pages 1797-1797, October.
    7. Julian Unkel & Alexander Haas, 2017. "The effects of credibility cues on the selection of search engine results," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(8), pages 1850-1862, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tiziana Assenza & Alberto Cardaci & Stefanie Huber, 2024. "Fake News: Susceptibility, Awareness, and Solutions," ECONtribute Policy Brief Series 065, University of Bonn and University of Cologne, Germany.
    2. Wonchan Choi, 2020. "Older adultsʼ credibility assessment of online health information: An exploratory study using an extended typology of web credibility," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 71(11), pages 1295-1307, November.
    3. Maram Hasanain & Tamer Elsayed, 2022. "Studying effectiveness of Web search for fact checking," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 73(5), pages 738-751, May.
    4. Tobia Spampatti & Ulf J. J. Hahnel & Evelina Trutnevyte & Tobias Brosch, 2024. "Psychological inoculation strategies to fight climate disinformation across 12 countries," Nature Human Behaviour, Nature, vol. 8(2), pages 380-398, February.
    5. Georg Pardi & Steffen Gottschling & Yvonne Kammerer, 2024. "The influence of knowledge type and source reputation on preferences for website or video search results," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 75(5), pages 521-537, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:jcsosc:v:8:y:2025:i:1:d:10.1007_s42001-024-00338-8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.