IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v15y2024i1d10.1038_s41467-023-44673-2.html
   My bibliography  Save this article

Encoding of multi-modal emotional information via personalized skin-integrated wireless facial interface

Author

Listed:
  • Jin Pyo Lee

    (Ulsan National Institute of Science and Technology
    Nanyang Technological University)

  • Hanhyeok Jang

    (Ulsan National Institute of Science and Technology)

  • Yeonwoo Jang

    (Ulsan National Institute of Science and Technology)

  • Hyeonseo Song

    (Ulsan National Institute of Science and Technology)

  • Suwoo Lee

    (Ulsan National Institute of Science and Technology)

  • Pooi See Lee

    (Nanyang Technological University)

  • Jiyun Kim

    (Ulsan National Institute of Science and Technology
    Ulsan National Institute of Science and Technology)

Abstract

Human affects such as emotions, moods, feelings are increasingly being considered as key parameter to enhance the interaction of human with diverse machines and systems. However, their intrinsically abstract and ambiguous nature make it challenging to accurately extract and exploit the emotional information. Here, we develop a multi-modal human emotion recognition system which can efficiently utilize comprehensive emotional information by combining verbal and non-verbal expression data. This system is composed of personalized skin-integrated facial interface (PSiFI) system that is self-powered, facile, stretchable, transparent, featuring a first bidirectional triboelectric strain and vibration sensor enabling us to sense and combine the verbal and non-verbal expression data for the first time. It is fully integrated with a data processing circuit for wireless data transfer allowing real-time emotion recognition to be performed. With the help of machine learning, various human emotion recognition tasks are done accurately in real time even while wearing mask and demonstrated digital concierge application in VR environment.

Suggested Citation

  • Jin Pyo Lee & Hanhyeok Jang & Yeonwoo Jang & Hyeonseo Song & Suwoo Lee & Pooi See Lee & Jiyun Kim, 2024. "Encoding of multi-modal emotional information via personalized skin-integrated wireless facial interface," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
  • Handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-023-44673-2
    DOI: 10.1038/s41467-023-44673-2
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-023-44673-2
    File Function: Abstract
    Download Restriction: no

    File URL: https://libkey.io/10.1038/s41467-023-44673-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Faren Huo & Yeying Zhao & Chunlei Chai & Fei Fang, 2023. "A user experience map design method based on emotional quantification of in-vehicle HMI," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-10, December.
    2. Ryan Yung & Catheryn Khoo-Lattimore & Leigh Ellen Potter, 2021. "Virtual reality and tourism marketing: conceptualizing a framework on presence, emotion, and intention," Current Issues in Tourism, Taylor & Francis Journals, vol. 24(11), pages 1505-1525, June.
    3. Siyoung Lee & Junsoo Kim & Inyeol Yun & Geun Yeol Bae & Daegun Kim & Sangsik Park & Il-Min Yi & Wonkyu Moon & Yoonyoung Chung & Kilwon Cho, 2019. "An ultrathin conformable vibration-responsive electronic skin for quantitative vocal recognition," Nature Communications, Nature, vol. 10(1), pages 1-11, December.
    4. Yijia Lu & Han Tian & Jia Cheng & Fei Zhu & Bin Liu & Shanshan Wei & Linhong Ji & Zhong Lin Wang, 2022. "Decoding lip language using triboelectric sensors with deep learning," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    5. Qilin Hua & Junlu Sun & Haitao Liu & Rongrong Bao & Ruomeng Yu & Junyi Zhai & Caofeng Pan & Zhong Lin Wang, 2018. "Skin-inspired highly stretchable and conformable matrix networks for multifunctional sensing," Nature Communications, Nature, vol. 9(1), pages 1-11, December.
    6. Domenico Consoli, 2010. "A New Concept of Marketing: The Emotional Marketing," BRAND. Broad Research in Accounting, Negotiation, and Distribution, EduSoft Publishing, vol. 1(1), pages 52-59, September.
    7. Chuah, Stephanie Hui-Wen & Yu, Joanne, 2021. "The future of service: The power of emotion in human-robot interaction," Journal of Retailing and Consumer Services, Elsevier, vol. 61(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sijia Xu & Jie-Xiang Yu & Hongshuang Guo & Shu Tian & You Long & Jing Yang & Lei Zhang, 2023. "Force-induced ion generation in zwitterionic hydrogels for a sensitive silent-speech sensor," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    2. Maggie Wang, Yazhu & Matook, Sabine & Dennis, Alan R., 2024. "Unintended consequences of humanoid service robots: A case study of public service organizations," Journal of Business Research, Elsevier, vol. 174(C).
    3. You Wang & Ming Zhang & Ruifen Hu & Guang Li & Nan Li, 2020. "Silent Speech Recognition for BCI - A Review," Biomedical Journal of Scientific & Technical Research, Biomedical Research Network+, LLC, vol. 27(2), pages 20625-20627, April.
    4. Hengtian Zhu & Huan Yang & Siqi Xu & Yuanyuan Ma & Shugeng Zhu & Zhengyi Mao & Weiwei Chen & Zizhong Hu & Rongrong Pan & Yurui Xu & Yifeng Xiong & Ye Chen & Yanqing Lu & Xinghai Ning & Dechen Jiang & , 2024. "Frequency-encoded eye tracking smart contact lens for human–machine interaction," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    5. Song, Christina Soyoung & Kim, Youn-Kyung, 2022. "The role of the human-robot interaction in consumers’ acceptance of humanoid retail service robots," Journal of Business Research, Elsevier, vol. 146(C), pages 489-503.
    6. Subhodeep Mukherjee & Manish Mohan Baral & Chittipaka Venkataiah & Surya Kant Pal & Ramji Nagariya, 2021. "Service robots are an option for contactless services due to the COVID-19 pandemic in the hotels," DECISION: Official Journal of the Indian Institute of Management Calcutta, Springer;Indian Institute of Management Calcutta, vol. 48(4), pages 445-460, December.
    7. Shuo Li & Yong Zhang & Xiaoping Liang & Haomin Wang & Haojie Lu & Mengjia Zhu & Huimin Wang & Mingchao Zhang & Xinping Qiu & Yafeng Song & Yingying Zhang, 2022. "Humidity-sensitive chemoelectric flexible sensors based on metal-air redox reaction for health management," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    8. María Dolores Aledo‐Ruiz & Eva Martínez‐Caro & José Manuel Santos‐Jaén, 2022. "The influence of corporate social responsibility on students' emotional appeal in the HEIs: The mediating effect of reputation and corporate image," Corporate Social Responsibility and Environmental Management, John Wiley & Sons, vol. 29(3), pages 578-592, May.
    9. Prentice, Catherine & Nguyen, Mai, 2021. "Robotic service quality – Scale development and validation," Journal of Retailing and Consumer Services, Elsevier, vol. 62(C).
    10. Xueguang Lu & Feilong Zhang & Liguo Zhu & Shan Peng & Jiazhen Yan & Qiwu Shi & Kefan Chen & Xue Chang & Hongfu Zhu & Cheng Zhang & Wanxia Huang & Qiang Cheng, 2024. "A terahertz meta-sensor array for 2D strain mapping," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    11. Minglu Wang & Shanshan Liu & Lingling Hu & Jong-Yoon Lee, 2023. "A Study of Metaverse Exhibition Sustainability on the Perspective of the Experience Economy," Sustainability, MDPI, vol. 15(12), pages 1-14, June.
    12. Linlin Li & Shufang Zhao & Wenhao Ran & Zhexin Li & Yongxu Yan & Bowen Zhong & Zheng Lou & Lili Wang & Guozhen Shen, 2022. "Dual sensing signal decoupling based on tellurium anisotropy for VR interaction and neuro-reflex system application," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    13. Zhao, Lin-Chuan & Zhou, Teng & Chang, Si-Deng & Zou, Hong-Xiang & Gao, Qiu-Hua & Wu, Zhi-Yuan & Yan, Ge & Wei, Ke-Xiang & Yeatman, Eric M. & Meng, Guang & Zhang, Wen-Ming, 2024. "A disposable cup inspired smart floor for trajectory recognition and human-interactive sensing," Applied Energy, Elsevier, vol. 357(C).
    14. Vishwakarma, Laxmi Pandit & Singh, Rajesh Kr & Mishra, Ruchi & Demirkol, Denizhan & Daim, Tugrul, 2024. "The adoption of social robots in service operations: A comprehensive review," Technology in Society, Elsevier, vol. 76(C).
    15. Subero-Navarro, à urea & Pelegrín-Borondo, Jorge & Reinares-Lara, Eva & Olarte-Pascual, Cristina, 2022. "Proposal for modeling social robot acceptance by retail customers: CAN model + technophobia," Journal of Retailing and Consumer Services, Elsevier, vol. 64(C).
    16. Xun Zhao & Yihao Zhou & William Kwak & Aaron Li & Shaolei Wang & Marklin Dallenger & Songyue Chen & Yuqi Zhang & Allison Lium & Jun Chen, 2024. "A reconfigurable and conformal liquid sensor for ambulatory cardiac monitoring," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    17. Jiayue Zhang & Yikui Gao & Di Liu & Jing-Shan Zhao & Jie Wang, 2023. "Discharge domains regulation and dynamic processes of direct-current triboelectric nanogenerator," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    18. Zhang, Yaqiong & Wang, Shifu, 2023. "The influence of anthropomorphic appearance of artificial intelligence products on consumer behavior and brand evaluation under different product types," Journal of Retailing and Consumer Services, Elsevier, vol. 74(C).
    19. Yijia Lu & Han Tian & Jia Cheng & Fei Zhu & Bin Liu & Shanshan Wei & Linhong Ji & Zhong Lin Wang, 2022. "Decoding lip language using triboelectric sensors with deep learning," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    20. Li, Sixian & Peluso, Alessandro M. & Duan, Jinyun, 2023. "Why do we prefer humans to artificial intelligence in telemarketing? A mind perception explanation," Journal of Retailing and Consumer Services, Elsevier, vol. 70(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-023-44673-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.