IDEAS home Printed from https://ideas.repec.org/a/gam/jftint/v16y2024i12p432-d1525868.html
   My bibliography  Save this article

Using Multimodal Foundation Models for Detecting Fake Images on the Internet with Explanations

Author

Listed:
  • Vishnu S. Pendyala

    (Department of Applied Data Science, San Jose State University, San Jose, CA 95192, USA)

  • Ashwin Chintalapati

    (Department of Computer Science, Purdue University, West Lafayette, IN 47907, USA)

Abstract

Generative AI and multimodal foundation models have fueled a proliferation of fake content on the Internet. This paper investigates if foundation models help detect and thereby contain the spread of fake images. The task of detecting fake images is a formidable challenge owing to its visual nature and intricate analysis. This paper details experiments using four multimodal foundation models, Llava, CLIP, Moondream2, and Gemini 1.5 Flash, to detect fake images. Explainable AI techniques such as Local Interpretable Model-Agnostic Explanations (LIME) and removal-based explanations are used to gain insights into the detection process. The dataset used comprised real images and fake images generated by a generative artificial intelligence tool called MidJourney. Results show that the models can achieve up to a 69% accuracy rate in detecting fake images in an intuitively explainable way, as confirmed by multiple techniques and metrics.

Suggested Citation

  • Vishnu S. Pendyala & Ashwin Chintalapati, 2024. "Using Multimodal Foundation Models for Detecting Fake Images on the Internet with Explanations," Future Internet, MDPI, vol. 16(12), pages 1-16, November.
    • Handle: RePEc:gam:jftint:v:16:y:2024:i:12:p:432-:d:1525868
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1999-5903/16/12/432/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1999-5903/16/12/432/
    Download Restriction: no
    ---><---

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jftint:v:16:y:2024:i:12:p:432-:d:1525868. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.