IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v10y2022i4p618-d751424.html
   My bibliography  Save this article

Continuous Hybrid BCI Control for Robotic Arm Using Noninvasive Electroencephalogram, Computer Vision, and Eye Tracking

Author

Listed:
  • Baoguo Xu

    (The State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China)

  • Wenlong Li

    (The State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China)

  • Deping Liu

    (The State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China)

  • Kun Zhang

    (The State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China)

  • Minmin Miao

    (School of Information Engineering, Huzhou University, Huzhou 313000, China)

  • Guozheng Xu

    (School of Automation and Artificial Intelligence, Nanjing University of Posts and Telecommunications, Nanjing 210003, China)

  • Aiguo Song

    (The State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China)

Abstract

The controlling of robotic arms based on brain–computer interface (BCI) can revolutionize the quality of life and living conditions for individuals with physical disabilities. Invasive electroencephalography (EEG)-based BCI has been able to control multiple degrees of freedom (DOFs) robotic arms in three dimensions. However, it is still hard to control a multi-DOF robotic arm to reach and grasp the desired target accurately in complex three-dimensional (3D) space by a noninvasive system mainly due to the limitation of EEG decoding performance. In this study, we propose a noninvasive EEG-based BCI for a robotic arm control system that enables users to complete multitarget reach and grasp tasks and avoid obstacles by hybrid control. The results obtained from seven subjects demonstrated that motor imagery (MI) training could modulate brain rhythms, and six of them completed the online tasks using the hybrid-control-based robotic arm system. The proposed system shows effective performance due to the combination of MI-based EEG, computer vision, gaze detection, and partially autonomous guidance, which drastically improve the accuracy of online tasks and reduce the brain burden caused by long-term mental activities.

Suggested Citation

  • Baoguo Xu & Wenlong Li & Deping Liu & Kun Zhang & Minmin Miao & Guozheng Xu & Aiguo Song, 2022. "Continuous Hybrid BCI Control for Robotic Arm Using Noninvasive Electroencephalogram, Computer Vision, and Eye Tracking," Mathematics, MDPI, vol. 10(4), pages 1-20, February.
    • Handle: RePEc:gam:jmathe:v:10:y:2022:i:4:p:618-:d:751424
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/10/4/618/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/10/4/618/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Leigh R. Hochberg & Daniel Bacher & Beata Jarosiewicz & Nicolas Y. Masse & John D. Simeral & Joern Vogel & Sami Haddadin & Jie Liu & Sydney S. Cash & Patrick van der Smagt & John P. Donoghue, 2012. "Reach and grasp by people with tetraplegia using a neurally controlled robotic arm," Nature, Nature, vol. 485(7398), pages 372-375, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yujie Li & Longzhao Huang & Jiahui Chen & Xiwen Wang & Benying Tan, 2023. "Appearance-Based Gaze Estimation Method Using Static Transformer Temporal Differential Network," Mathematics, MDPI, vol. 11(3), pages 1-18, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Andrey Eliseyev & Tetiana Aksenova, 2016. "Penalized Multi-Way Partial Least Squares for Smooth Trajectory Decoding from Electrocorticographic (ECoG) Recording," PLOS ONE, Public Library of Science, vol. 11(5), pages 1-19, May.
    2. Ujwal Chaudhary & Bin Xia & Stefano Silvoni & Leonardo G Cohen & Niels Birbaumer, 2017. "Brain–Computer Interface–Based Communication in the Completely Locked-In State," PLOS Biology, Public Library of Science, vol. 15(1), pages 1-25, January.
    3. Keundong Lee & Angelique C. Paulk & Yun Goo Ro & Daniel R. Cleary & Karen J. Tonsfeldt & Yoav Kfir & John S. Pezaris & Youngbin Tchoe & Jihwan Lee & Andrew M. Bourhis & Ritwik Vatsyayan & Joel R. Mart, 2024. "Flexible, scalable, high channel count stereo-electrode for recording in the human brain," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    4. Fan Li & Jazlyn Gallego & Natasha N. Tirko & Jenna Greaser & Derek Bashe & Rudra Patel & Eric Shaker & Grace E. Valkenburg & Alanoud S. Alsubhi & Steven Wellman & Vanshika Singh & Camila Garcia Padill, 2024. "Low-intensity pulsed ultrasound stimulation (LIPUS) modulates microglial activation following intracortical microelectrode implantation," Nature Communications, Nature, vol. 15(1), pages 1-21, December.
    5. Xiao-yu Sun & Bin Ye, 2023. "The functional differentiation of brain–computer interfaces (BCIs) and its ethical implications," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-9, December.
    6. David Balderas & Pedro Ponce & Diego Lopez-Bernal & Arturo Molina, 2021. "Education 4.0: Teaching the Basis of Motor Imagery Classification Algorithms for Brain-Computer Interfaces," Future Internet, MDPI, vol. 13(8), pages 1-27, August.
    7. Eric A Pohlmeyer & Babak Mahmoudi & Shijia Geng & Noeline W Prins & Justin C Sanchez, 2014. "Using Reinforcement Learning to Provide Stable Brain-Machine Interface Control Despite Neural Input Reorganization," PLOS ONE, Public Library of Science, vol. 9(1), pages 1-12, January.
    8. Josh Merel & David Carlson & Liam Paninski & John P Cunningham, 2016. "Neuroprosthetic Decoder Training as Imitation Learning," PLOS Computational Biology, Public Library of Science, vol. 12(5), pages 1-24, May.
    9. Betz, Ulrich A.K. & Arora, Loukik & Assal, Reem A. & Azevedo, Hatylas & Baldwin, Jeremy & Becker, Michael S. & Bostock, Stefan & Cheng, Vinton & Egle, Tobias & Ferrari, Nicola & Schneider-Futschik, El, 2023. "Game changers in science and technology - now and beyond," Technological Forecasting and Social Change, Elsevier, vol. 193(C).
    10. Andrés Úbeda & Enrique Hortal & Eduardo Iáñez & Carlos Perez-Vidal & Jose M Azorín, 2015. "Assessing Movement Factors in Upper Limb Kinematics Decoding from EEG Signals," PLOS ONE, Public Library of Science, vol. 10(5), pages 1-12, May.
    11. Fuji Ren & Yanwei Bao, 2020. "A Review on Human-Computer Interaction and Intelligent Robots," International Journal of Information Technology & Decision Making (IJITDM), World Scientific Publishing Co. Pte. Ltd., vol. 19(01), pages 5-47, February.
    12. Han-Lin Hsieh & Maryam M Shanechi, 2018. "Optimizing the learning rate for adaptive estimation of neural encoding models," PLOS Computational Biology, Public Library of Science, vol. 14(5), pages 1-34, May.
    13. Elisa Donati & Giacomo Valle, 2024. "Neuromorphic hardware for somatosensory neuroprostheses," Nature Communications, Nature, vol. 15(1), pages 1-18, December.
    14. Javier M Antelis & Luis Montesano & Ander Ramos-Murguialday & Niels Birbaumer & Javier Minguez, 2013. "On the Usage of Linear Regression Models to Reconstruct Limb Kinematics from Low Frequency EEG Signals," PLOS ONE, Public Library of Science, vol. 8(4), pages 1-14, April.
    15. Xiangjing Wang & Chunsheng Chen & Li Zhu & Kailu Shi & Baocheng Peng & Yixin Zhu & Huiwu Mao & Haotian Long & Shuo Ke & Chuanyu Fu & Ying Zhu & Changjin Wan & Qing Wan, 2023. "Vertically integrated spiking cone photoreceptor arrays for color perception," Nature Communications, Nature, vol. 14(1), pages 1-9, December.
    16. Jonathan A Michaels & Benjamin Dann & Hansjörg Scherberger, 2016. "Neural Population Dynamics during Reaching Are Better Explained by a Dynamical System than Representational Tuning," PLOS Computational Biology, Public Library of Science, vol. 12(11), pages 1-22, November.
    17. Hong Gi Yeom & June Sic Kim & Chun Kee Chung, 2014. "High-Accuracy Brain-Machine Interfaces Using Feedback Information," PLOS ONE, Public Library of Science, vol. 9(7), pages 1-7, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:10:y:2022:i:4:p:618-:d:751424. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.