IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v13y2022i1d10.1038_s41467-022-29083-0.html
   My bibliography  Save this article

Decoding lip language using triboelectric sensors with deep learning

Author

Listed:
  • Yijia Lu

    (Tsinghua University)

  • Han Tian

    (Tsinghua University)

  • Jia Cheng

    (Tsinghua University)

  • Fei Zhu

    (Chinese Academy of Sciences)

  • Bin Liu

    (Tsinghua University)

  • Shanshan Wei

    (Tsinghua University)

  • Linhong Ji

    (Tsinghua University)

  • Zhong Lin Wang

    (Chinese Academy of Sciences
    University of Chinese Academy of Sciences
    Georgia Institute of Technology)

Abstract

Lip language is an effective method of voice-off communication in daily life for people with vocal cord lesions and laryngeal and lingual injuries without occupying the hands. Collection and interpretation of lip language is challenging. Here, we propose the concept of a novel lip-language decoding system with self-powered, low-cost, contact and flexible triboelectric sensors and a well-trained dilated recurrent neural network model based on prototype learning. The structural principle and electrical properties of the flexible sensors are measured and analysed. Lip motions for selected vowels, words, phrases, silent speech and voice speech are collected and compared. The prototype learning model reaches a test accuracy of 94.5% in training 20 classes with 100 samples each. The applications, such as identity recognition to unlock a gate, directional control of a toy car and lip-motion to speech conversion, work well and demonstrate great feasibility and potential. Our work presents a promising way to help people lacking a voice live a convenient life with barrier-free communication and boost their happiness, enriches the diversity of lip-language translation systems and will have potential value in many applications.

Suggested Citation

  • Yijia Lu & Han Tian & Jia Cheng & Fei Zhu & Bin Liu & Shanshan Wei & Linhong Ji & Zhong Lin Wang, 2022. "Decoding lip language using triboelectric sensors with deep learning," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
  • Handle: RePEc:nat:natcom:v:13:y:2022:i:1:d:10.1038_s41467-022-29083-0
    DOI: 10.1038/s41467-022-29083-0
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-022-29083-0
    File Function: Abstract
    Download Restriction: no

    File URL: https://libkey.io/10.1038/s41467-022-29083-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Qiongfeng Shi & Zixuan Zhang & Tianyiyi He & Zhongda Sun & Bingjie Wang & Yuqin Feng & Xuechuan Shan & Budiman Salam & Chengkuo Lee, 2020. "Deep learning enabled smart mats as a scalable floor monitoring system," Nature Communications, Nature, vol. 11(1), pages 1-11, December.
    2. Tao Jin & Zhongda Sun & Long Li & Quan Zhang & Minglu Zhu & Zixuan Zhang & Guangjie Yuan & Tao Chen & Yingzhong Tian & Xuyan Hou & Chengkuo Lee, 2020. "Triboelectric nanogenerator sensors for soft robotics aiming at digital twin applications," Nature Communications, Nature, vol. 11(1), pages 1-12, December.
    3. Qilin Hua & Junlu Sun & Haitao Liu & Rongrong Bao & Ruomeng Yu & Junyi Zhai & Caofeng Pan & Zhong Lin Wang, 2018. "Skin-inspired highly stretchable and conformable matrix networks for multifunctional sensing," Nature Communications, Nature, vol. 9(1), pages 1-11, December.
    4. Yang Zou & Puchuan Tan & Bojing Shi & Han Ouyang & Dongjie Jiang & Zhuo Liu & Hu Li & Min Yu & Chan Wang & Xuecheng Qu & Luming Zhao & Yubo Fan & Zhong Lin Wang & Zhou Li, 2019. "A bionic stretchable nanogenerator for underwater sensing and energy harvesting," Nature Communications, Nature, vol. 10(1), pages 1-10, December.
    5. Younghoon Lee & Seung Hee Cha & Yong-Woo Kim & Dukhyun Choi & Jeong-Yun Sun, 2018. "Transparent and attachable ionic communicators based on self-cleanable triboelectric nanogenerators," Nature Communications, Nature, vol. 9(1), pages 1-8, December.
    6. Minglu Zhu & Zhongda Sun & Tao Chen & Chengkuo Lee, 2021. "Low cost exoskeleton manipulator using bidirectional triboelectric sensors enhanced multiple degree of freedom sensory system," Nature Communications, Nature, vol. 12(1), pages 1-16, December.
    7. Subramanian Sundaram & Petr Kellnhofer & Yunzhu Li & Jun-Yan Zhu & Antonio Torralba & Wojciech Matusik, 2019. "Learning the signatures of the human grasp using a scalable tactile glove," Nature, Nature, vol. 569(7758), pages 698-702, May.
    8. Feng Wen & Zixuan Zhang & Tianyiyi He & Chengkuo Lee, 2021. "AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove," Nature Communications, Nature, vol. 12(1), pages 1-13, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sijia Xu & Jie-Xiang Yu & Hongshuang Guo & Shu Tian & You Long & Jing Yang & Lei Zhang, 2023. "Force-induced ion generation in zwitterionic hydrogels for a sensitive silent-speech sensor," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    2. Taemin Kim & Yejee Shin & Kyowon Kang & Kiho Kim & Gwanho Kim & Yunsu Byeon & Hwayeon Kim & Yuyan Gao & Jeong Ryong Lee & Geonhui Son & Taeseong Kim & Yohan Jun & Jihyun Kim & Jinyoung Lee & Seyun Um , 2022. "Ultrathin crystalline-silicon-based strain gauges with deep learning algorithms for silent speech interfaces," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    3. Bincy, Thanjan Shaji & Prasanna, Asokan Poorani Sathya & Balaji, A. Sakthi & Sivasankar, K. Janani & Thiruvadigal, D. John & Anithkumar, Monunith & Kim, Sang-Jae, 2024. "Computational analysis of starch for sustainable power generation towards integrated wearable IoT," Applied Energy, Elsevier, vol. 370(C).
    4. Hang Zhang & Sankaran Sundaresan & Michael A. Webb, 2024. "Thermodynamic driving forces in contact electrification between polymeric materials," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    5. Yuxiang Shi & Peng Yang & Rui Lei & Zhaoqi Liu & Xuanyi Dong & Xinglin Tao & Xiangcheng Chu & Zhong Lin Wang & Xiangyu Chen, 2023. "Eye tracking and eye expression decoding based on transparent, flexible and ultra-persistent electrostatic interface," Nature Communications, Nature, vol. 14(1), pages 1-12, December.
    6. Yang, Ying & Zhang, Wei & Lin, Hongyi & Liu, Yang & Qu, Xiaobo, 2024. "Applying masked language model for transport mode choice behavior prediction," Transportation Research Part A: Policy and Practice, Elsevier, vol. 184(C).
    7. Jin Pyo Lee & Hanhyeok Jang & Yeonwoo Jang & Hyeonseo Song & Suwoo Lee & Pooi See Lee & Jiyun Kim, 2024. "Encoding of multi-modal emotional information via personalized skin-integrated wireless facial interface," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    8. Jiayue Zhang & Yikui Gao & Di Liu & Jing-Shan Zhao & Jie Wang, 2023. "Discharge domains regulation and dynamic processes of direct-current triboelectric nanogenerator," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    9. Hira Hameed & Muhammad Usman & Ahsen Tahir & Amir Hussain & Hasan Abbas & Tie Jun Cui & Muhammad Ali Imran & Qammer H. Abbasi, 2022. "Pushing the limits of remote RF sensing by reading lips under the face mask," Nature Communications, Nature, vol. 13(1), pages 1-9, December.
    10. Zhao, Lin-Chuan & Zhou, Teng & Chang, Si-Deng & Zou, Hong-Xiang & Gao, Qiu-Hua & Wu, Zhi-Yuan & Yan, Ge & Wei, Ke-Xiang & Yeatman, Eric M. & Meng, Guang & Zhang, Wen-Ming, 2024. "A disposable cup inspired smart floor for trajectory recognition and human-interactive sensing," Applied Energy, Elsevier, vol. 357(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhongda Sun & Minglu Zhu & Xuechuan Shan & Chengkuo Lee, 2022. "Augmented tactile-perception and haptic-feedback rings as human-machine interfaces aiming for immersive interactions," Nature Communications, Nature, vol. 13(1), pages 1-13, December.
    2. Min Chen & Jingyu Ouyang & Aijia Jian & Jia Liu & Pan Li & Yixue Hao & Yuchen Gong & Jiayu Hu & Jing Zhou & Rui Wang & Jiaxi Wang & Long Hu & Yuwei Wang & Ju Ouyang & Jing Zhang & Chong Hou & Lei Wei , 2022. "Imperceptible, designable, and scalable braided electronic cord," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    3. Yunlong Xu & Zhongda Sun & Zhiqing Bai & Hua Shen & Run Wen & Fumei Wang & Guangbiao Xu & Chengkuo Lee, 2024. "Bionic e-skin with precise multi-directional droplet sliding sensing for enhanced robotic perception," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    4. Rui Chen & Tao Luo & Jincheng Wang & Renpeng Wang & Chen Zhang & Yu Xie & Lifeng Qin & Haimin Yao & Wei Zhou, 2023. "Nonlinearity synergy: An elegant strategy for realizing high-sensitivity and wide-linear-range pressure sensing," Nature Communications, Nature, vol. 14(1), pages 1-9, December.
    5. Chunpeng Jiang & Wenqiang Xu & Yutong Li & Zhenjun Yu & Longchun Wang & Xiaotong Hu & Zhengyi Xie & Qingkun Liu & Bin Yang & Xiaolin Wang & Wenxin Du & Tutian Tang & Dongzhe Zheng & Siqiong Yao & Cewu, 2024. "Capturing forceful interaction with deformable objects using a deep learning-powered stretchable tactile array," Nature Communications, Nature, vol. 15(1), pages 1-14, December.
    6. Wenbo Liu & Youning Duo & Jiaqi Liu & Feiyang Yuan & Lei Li & Luchen Li & Gang Wang & Bohan Chen & Siqi Wang & Hui Yang & Yuchen Liu & Yanru Mo & Yun Wang & Bin Fang & Fuchun Sun & Xilun Ding & Chi Zh, 2022. "Touchless interactive teaching of soft robots through flexible bimodal sensory interfaces," Nature Communications, Nature, vol. 13(1), pages 1-14, December.
    7. Zhao, Kun & Song, Zhenhua & Sun, Wanru & Gao, Wei & Guo, Junhong & Zhang, Kewei, 2024. "Flexible neodymium iron boron/polyvinyl chloride (Nd2Fe14B/PVC) composite film based hybrid nanogenerator for efficient mechanical energy harvesting," Energy, Elsevier, vol. 300(C).
    8. Shijing Zhang & Yingxiang Liu & Jie Deng & Xiang Gao & Jing Li & Weiyi Wang & Mingxin Xun & Xuefeng Ma & Qingbing Chang & Junkao Liu & Weishan Chen & Jie Zhao, 2023. "Piezo robotic hand for motion manipulation from micro to macro," Nature Communications, Nature, vol. 14(1), pages 1-12, December.
    9. Hengtian Zhu & Huan Yang & Siqi Xu & Yuanyuan Ma & Shugeng Zhu & Zhengyi Mao & Weiwei Chen & Zizhong Hu & Rongrong Pan & Yurui Xu & Yifeng Xiong & Ye Chen & Yanqing Lu & Xinghai Ning & Dechen Jiang & , 2024. "Frequency-encoded eye tracking smart contact lens for human–machine interaction," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    10. Shuo Li & Yong Zhang & Xiaoping Liang & Haomin Wang & Haojie Lu & Mengjia Zhu & Huimin Wang & Mingchao Zhang & Xinping Qiu & Yafeng Song & Yingying Zhang, 2022. "Humidity-sensitive chemoelectric flexible sensors based on metal-air redox reaction for health management," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    11. Hongfa Zhao & Minyi Xu & Mingrui Shu & Jie An & Wenbo Ding & Xiangyu Liu & Siyuan Wang & Cong Zhao & Hongyong Yu & Hao Wang & Chuan Wang & Xianping Fu & Xinxiang Pan & Guangming Xie & Zhong Lin Wang, 2022. "Underwater wireless communication via TENG-generated Maxwell’s displacement current," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    12. Xueguang Lu & Feilong Zhang & Liguo Zhu & Shan Peng & Jiazhen Yan & Qiwu Shi & Kefan Chen & Xue Chang & Hongfu Zhu & Cheng Zhang & Wanxia Huang & Qiang Cheng, 2024. "A terahertz meta-sensor array for 2D strain mapping," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    13. Wenxi Huang & Qiongling Ding & Hao Wang & Zixuan Wu & Yibing Luo & Wenxiong Shi & Le Yang & Yujie Liang & Chuan Liu & Jin Wu, 2023. "Design of stretchable and self-powered sensing device for portable and remote trace biomarkers detection," Nature Communications, Nature, vol. 14(1), pages 1-13, December.
    14. Linlin Li & Shufang Zhao & Wenhao Ran & Zhexin Li & Yongxu Yan & Bowen Zhong & Zheng Lou & Lili Wang & Guozhen Shen, 2022. "Dual sensing signal decoupling based on tellurium anisotropy for VR interaction and neuro-reflex system application," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    15. Zhao, Lin-Chuan & Zhou, Teng & Chang, Si-Deng & Zou, Hong-Xiang & Gao, Qiu-Hua & Wu, Zhi-Yuan & Yan, Ge & Wei, Ke-Xiang & Yeatman, Eric M. & Meng, Guang & Zhang, Wen-Ming, 2024. "A disposable cup inspired smart floor for trajectory recognition and human-interactive sensing," Applied Energy, Elsevier, vol. 357(C).
    16. Ali Matin Nazar & King-James Idala Egbe & Azam Abdollahi & Mohammad Amin Hariri-Ardebili, 2021. "Triboelectric Nanogenerators for Energy Harvesting in Ocean: A Review on Application and Hybridization," Energies, MDPI, vol. 14(18), pages 1-33, September.
    17. Jin Pyo Lee & Hanhyeok Jang & Yeonwoo Jang & Hyeonseo Song & Suwoo Lee & Pooi See Lee & Jiyun Kim, 2024. "Encoding of multi-modal emotional information via personalized skin-integrated wireless facial interface," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    18. Haojie Lu & Yong Zhang & Mengjia Zhu & Shuo Li & Huarun Liang & Peng Bi & Shuai Wang & Haomin Wang & Linli Gan & Xun-En Wu & Yingying Zhang, 2024. "Intelligent perceptual textiles based on ionic-conductive and strong silk fibers," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    19. Sixing Xiong & Kenjiro Fukuda & Kyohei Nakano & Shinyoung Lee & Yutaro Sumi & Masahito Takakuwa & Daishi Inoue & Daisuke Hashizume & Baocai Du & Tomoyuki Yokota & Yinhua Zhou & Keisuke Tajima & Takao , 2024. "Waterproof and ultraflexible organic photovoltaics with improved interface adhesion," Nature Communications, Nature, vol. 15(1), pages 1-10, December.
    20. Bujingda Zheng & Yunchao Xie & Shichen Xu & Andrew C. Meng & Shaoyun Wang & Yuchao Wu & Shuhong Yang & Caixia Wan & Guoliang Huang & James M. Tour & Jian Lin, 2024. "Programmed multimaterial assembly by synergized 3D printing and freeform laser induction," Nature Communications, Nature, vol. 15(1), pages 1-12, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:13:y:2022:i:1:d:10.1038_s41467-022-29083-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.