IDEAS home Printed from https://ideas.repec.org/a/gam/jijerp/v18y2021i12p6216-d571157.html
   My bibliography  Save this article

Analysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution

Author

Listed:
  • Mooseop Kim

    (Human Enhancement & Assistive Technology Research Section, Artificial Intelligence Research Lab., Electronics Telecommunications Research Institute (ETRI), Daejeon 34129, Korea)

  • YunKyung Park

    (Human Enhancement & Assistive Technology Research Section, Artificial Intelligence Research Lab., Electronics Telecommunications Research Institute (ETRI), Daejeon 34129, Korea)

  • KyeongDeok Moon

    (Human Enhancement & Assistive Technology Research Section, Artificial Intelligence Research Lab., Electronics Telecommunications Research Institute (ETRI), Daejeon 34129, Korea)

  • Chi Yoon Jeong

    (Human Enhancement & Assistive Technology Research Section, Artificial Intelligence Research Lab., Electronics Telecommunications Research Institute (ETRI), Daejeon 34129, Korea)

Abstract

Visual-auditory sensory substitution has demonstrated great potential to help visually impaired and blind groups to recognize objects and to perform basic navigational tasks. However, the high latency between visual information acquisition and auditory transduction may contribute to the lack of the successful adoption of such aid technologies in the blind community; thus far, substitution methods have remained only laboratory-scale research or pilot demonstrations. This high latency for data conversion leads to challenges in perceiving fast-moving objects or rapid environmental changes. To reduce this latency, prior analysis of auditory sensitivity is necessary. However, existing auditory sensitivity analyses are subjective because they were conducted using human behavioral analysis. Therefore, in this study, we propose a cross-modal generative adversarial network-based evaluation method to find an optimal auditory sensitivity to reduce transmission latency in visual-auditory sensory substitution, which is related to the perception of visual information. We further conducted a human-based assessment to evaluate the effectiveness of the proposed model-based analysis in human behavioral experiments. We conducted experiments with three participant groups, including sighted users (SU), congenitally blind (CB) and late-blind (LB) individuals. Experimental results from the proposed model showed that the temporal length of the auditory signal for sensory substitution could be reduced by 50%. This result indicates the possibility of improving the performance of the conventional vOICe method by up to two times. We confirmed that our experimental results are consistent with human assessment through behavioral experiments. Analyzing auditory sensitivity with deep learning models has the potential to improve the efficiency of sensory substitution.

Suggested Citation

  • Mooseop Kim & YunKyung Park & KyeongDeok Moon & Chi Yoon Jeong, 2021. "Analysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution," IJERPH, MDPI, vol. 18(12), pages 1-22, June.
  • Handle: RePEc:gam:jijerp:v:18:y:2021:i:12:p:6216-:d:571157
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1660-4601/18/12/6216/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/1660-4601/18/12/6216/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Frédéric Gougoux & Franco Lepore & Maryse Lassonde & Patrice Voss & Robert J. Zatorre & Pascal Belin, 2004. "Pitch discrimination in the early blind," Nature, Nature, vol. 430(6997), pages 309-309, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hisham E. Bilal Salih & Kazunori Takeda & Hideyuki Kobayashi & Toshibumi Kakizawa & Masayuki Kawamoto & Keiichi Zempo, 2022. "Use of Auditory Cues and Other Strategies as Sources of Spatial Information for People with Visual Impairment When Navigating Unfamiliar Environments," IJERPH, MDPI, vol. 19(6), pages 1-16, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.

      Corrections

      All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jijerp:v:18:y:2021:i:12:p:6216-:d:571157. See general information about how to correct material in RePEc.

      If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

      If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

      If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

      For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

      Please note that corrections may take a couple of weeks to filter through the various RePEc services.

      IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.