IDEAS home Printed from https://ideas.repec.org/a/gam/jftint/v13y2021i11p298-d685122.html
   My bibliography  Save this article

Face Swapping Consistency Transfer with Neural Identity Carrier

Author

Listed:
  • Kunlin Liu

    (School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China)

  • Ping Wang

    (School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China)

  • Wenbo Zhou

    (School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China)

  • Zhenyu Zhang

    (Tencent Youtu, Shanghai 200233, China)

  • Yanhao Ge

    (Tencent Youtu, Shanghai 200233, China)

  • Honggu Liu

    (School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China)

  • Weiming Zhang

    (School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China)

  • Nenghai Yu

    (School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China)

Abstract

Deepfake aims to swap a face of an image with someone else’s likeness in a reasonable manner. Existing methods usually perform deepfake frame by frame, thus ignoring video consistency and producing incoherent results. To address such a problem, we propose a novel framework Neural Identity Carrier (NICe), which learns identity transformation from an arbitrary face-swapping proxy via a U - N e t . By modeling the incoherence between frames as noise, NICe naturally suppresses its disturbance and preserves primary identity information. Concretely, NICe inputs the original frame and learns transformation supervised by swapped pseudo labels. As the temporal incoherence has an uncertain or stochastic pattern, NICe can filter out such outliers and well maintain the target content by uncertainty prediction. With the predicted temporally stable appearance, NICe enhances its details by constraining 3D geometry consistency, making NICe learn fine-grained facial structure across the poses. In this way, NICe guarantees the temporal stableness of deepfake approaches and predicts detailed results against over-smoothness. Extensive experiments on benchmarks demonstrate that NICe significantly improves the quality of existing deepfake methods on video-level. Besides, data generated by our methods can benefit video-level deepfake detection methods.

Suggested Citation

  • Kunlin Liu & Ping Wang & Wenbo Zhou & Zhenyu Zhang & Yanhao Ge & Honggu Liu & Weiming Zhang & Nenghai Yu, 2021. "Face Swapping Consistency Transfer with Neural Identity Carrier," Future Internet, MDPI, vol. 13(11), pages 1-14, November.
  • Handle: RePEc:gam:jftint:v:13:y:2021:i:11:p:298-:d:685122
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1999-5903/13/11/298/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/1999-5903/13/11/298/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Chaminda Hewage & Erhan Ekmekcioglu, 2020. "Multimedia Quality of Experience (QoE): Current Status and Future Direction," Future Internet, MDPI, vol. 12(7), pages 1-3, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.

      Corrections

      All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jftint:v:13:y:2021:i:11:p:298-:d:685122. See general information about how to correct material in RePEc.

      If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

      If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

      If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

      For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

      Please note that corrections may take a couple of weeks to filter through the various RePEc services.

      IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.