IDEAS home Printed from https://ideas.repec.org/a/gam/jftint/v17y2025i3p127-d1611056.html
   My bibliography  Save this article

Deep Neural Network-Based Modeling of Multimodal Human–Computer Interaction in Aircraft Cockpits

Author

Listed:
  • Li Wang

    (School of Electronic & Electrical Engineering, Baoji University of Arts and Sciences, Baoji 721016, China)

  • Heming Zhang

    (School of Optoelectronic Engineering, Xi’an Technological University, Xi’an 710000, China)

  • Changyuan Wang

    (School of Computer Science, Xi’an Technological University, Xi’an 710021, China)

Abstract

Improving the performance of human–computer interaction systems is an essential indicator of aircraft intelligence. To address the limitations of single-modal interaction methods, a multimodal interaction model based on gaze and EEG target selection is proposed using deep learning technology. This model consists of two parts: target classification and intention recognition. The target classification model based on long short-term memory networks is established and trained by combining the eye movement information of the operator. The intention recognition model based on transformers is constructed and trained by combining the operator’s EEG information. In the application scenario of the aircraft radar page system, the highest accuracy of the target classification model is 98%. The intention recognition rate obtained by training the 32-channel EEG information in the intention recognition model is 98.5%, which is higher than other compared models. In addition, we validated the model on a simulated flight platform, and the experimental results show that the proposed multimodal interaction framework outperforms the single gaze interaction in terms of performance.

Suggested Citation

  • Li Wang & Heming Zhang & Changyuan Wang, 2025. "Deep Neural Network-Based Modeling of Multimodal Human–Computer Interaction in Aircraft Cockpits," Future Internet, MDPI, vol. 17(3), pages 1-15, March.
  • Handle: RePEc:gam:jftint:v:17:y:2025:i:3:p:127-:d:1611056
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1999-5903/17/3/127/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/1999-5903/17/3/127/
    Download Restriction: no
    ---><---

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jftint:v:17:y:2025:i:3:p:127-:d:1611056. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.