IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v15y2024i1d10.1038_s41467-024-53873-3.html
   My bibliography  Save this article

Using large language models to accelerate communication for eye gaze typing users with ALS

Author

Listed:
  • Shanqing Cai

    (Google)

  • Subhashini Venugopalan

    (Google)

  • Katie Seaver

    (Google)

  • Xiang Xiao

    (Google)

  • Katrin Tomanek

    (Google)

  • Sri Jalasutram

    (Google)

  • Meredith Ringel Morris

    (Google)

  • Shaun Kane

    (Google)

  • Ajit Narayanan

    (Google)

  • Robert L. MacDonald

    (Google)

  • Emily Kornman

    (Team Gleason Foundation)

  • Daniel Vance

    (Team Gleason Foundation)

  • Blair Casey

    (Team Gleason Foundation)

  • Steve M. Gleason

    (Team Gleason Foundation)

  • Philip Q. Nelson

    (Google)

  • Michael P. Brenner

    (Google
    Harvard University)

Abstract

Accelerating text input in augmentative and alternative communication (AAC) is a long-standing area of research with bearings on the quality of life in individuals with profound motor impairments. Recent advances in large language models (LLMs) pose opportunities for re-thinking strategies for enhanced text entry in AAC. In this paper, we present SpeakFaster, consisting of an LLM-powered user interface for text entry in a highly-abbreviated form, saving 57% more motor actions than traditional predictive keyboards in offline simulation. A pilot study on a mobile device with 19 non-AAC participants demonstrated motor savings in line with simulation and relatively small changes in typing speed. Lab and field testing on two eye-gaze AAC users with amyotrophic lateral sclerosis demonstrated text-entry rates 29–60% above baselines, due to significant saving of expensive keystrokes based on LLM predictions. These findings form a foundation for further exploration of LLM-assisted text entry in AAC and other user interfaces.

Suggested Citation

  • Shanqing Cai & Subhashini Venugopalan & Katie Seaver & Xiang Xiao & Katrin Tomanek & Sri Jalasutram & Meredith Ringel Morris & Shaun Kane & Ajit Narayanan & Robert L. MacDonald & Emily Kornman & Danie, 2024. "Using large language models to accelerate communication for eye gaze typing users with ALS," Nature Communications, Nature, vol. 15(1), pages 1-18, December.
    • Handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-53873-3
    DOI: 10.1038/s41467-024-53873-3
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-024-53873-3
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-024-53873-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Francis R. Willett & Donald T. Avansino & Leigh R. Hochberg & Jaimie M. Henderson & Krishna V. Shenoy, 2021. "High-performance brain-to-text communication via handwriting," Nature, Nature, vol. 593(7858), pages 249-254, May.
    2. Francis R. Willett & Erin M. Kunz & Chaofei Fan & Donald T. Avansino & Guy H. Wilson & Eun Young Choi & Foram Kamdar & Matthew F. Glasser & Leigh R. Hochberg & Shaul Druckmann & Krishna V. Shenoy & Ja, 2023. "A high-performance speech neuroprosthesis," Nature, Nature, vol. 620(7976), pages 1031-1036, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:prg:jnlelg:v:preprint:id:499 is not listed on IDEAS
    2. Jeffrey D. Laurence-Chasen & Callum F. Ross & Fritzie I. Arce-McShane & Nicholas G. Hatsopoulos, 2023. "Robust cortical encoding of 3D tongue shape during feeding in macaques," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    3. Zhouheng Wang & Nanlin Shi & Yingchao Zhang & Ning Zheng & Haicheng Li & Yang Jiao & Jiahui Cheng & Yutong Wang & Xiaoqing Zhang & Ying Chen & Yihao Chen & Heling Wang & Tao Xie & Yijun Wang & Yinji M, 2023. "Conformal in-ear bioelectronics for visual and auditory brain-computer interfaces," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    4. Joshua Kosnoff & Kai Yu & Chang Liu & Bin He, 2024. "Transcranial focused ultrasound to V5 enhances human visual motion brain-computer interface by modulating feature-based attention," Nature Communications, Nature, vol. 15(1), pages 1-18, December.
    5. Robert Burgan, 2023. "Once More about Human Nature, Enhancement and Substitution [Ešte raz o ľudskej prirodzenosti, zdokonaľovaní a substitúcii]," E-LOGOS, Prague University of Economics and Business, vol. 2023(2), pages 4-55.
    6. Xiner Wang & Guo Bai & Jizhi Liang & Qianyang Xie & Zhaohan Chen & Erda Zhou & Meng Li & Xiaoling Wei & Liuyang Sun & Zhiyuan Zhang & Chi Yang & Tiger H. Tao & Zhitao Zhou, 2024. "Gustatory interface for operative assessment and taste decoding in patients with tongue cancer," Nature Communications, Nature, vol. 15(1), pages 1-10, December.
    7. Sarah K. Wandelt & David A. Bjånes & Kelsie Pejsa & Brian Lee & Charles Liu & Richard A. Andersen, 2024. "Representation of internal speech by single neurons in human supramarginal gyrus," Nature Human Behaviour, Nature, vol. 8(6), pages 1136-1149, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-53873-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.