IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v15y2024i1d10.1038_s41467-024-52563-4.html
   My bibliography  Save this article

Birdlike broadband neuromorphic visual sensor arrays for fusion imaging

Author

Listed:
  • Pengshan Xie

    (City University of Hong Kong)

  • Yunchao Xu

    (Central South University
    Shanghai Institute of Microsystem and Information Technology)

  • Jingwen Wang

    (Central South University
    Shanghai Institute of Microsystem and Information Technology)

  • Dengji Li

    (City University of Hong Kong)

  • Yuxuan Zhang

    (City University of Hong Kong)

  • Zixin Zeng

    (City University of Hong Kong)

  • Boxiang Gao

    (City University of Hong Kong)

  • Quan Quan

    (City University of Hong Kong)

  • Bowen Li

    (City University of Hong Kong)

  • You Meng

    (City University of Hong Kong
    City University of Hong Kong)

  • Weijun Wang

    (City University of Hong Kong)

  • Yezhan Li

    (City University of Hong Kong)

  • Yan Yan

    (City University of Hong Kong)

  • Yi Shen

    (City University of Hong Kong)

  • Jia Sun

    (Central South University
    Shanghai Institute of Microsystem and Information Technology)

  • Johnny C. Ho

    (City University of Hong Kong
    City University of Hong Kong
    Kyushu University)

Abstract

Wearable visual bionic devices, fueled by advancements in artificial intelligence, are making remarkable progress. However, traditional silicon vision chips often grapple with high energy losses and challenges in emulating complex biological behaviors. In this study, we constructed a van der Waals P3HT/GaAs nanowires P-N junction by carefully directing the arrangement of organic molecules. Combined with a Schottky junction, this facilitated multi-faceted birdlike visual enhancement, including broadband non-volatile storage, low-light perception, and a near-zero power consumption operating mode in both individual devices and 5 × 5 arrays on arbitrary substrates. Specifically, we realized over 5 bits of in-memory sensing and computing with both negative and positive photoconductivity. When paired with two imaging modes (visible and UV), our reservoir computing system demonstrated up to 94% accuracy for color recognition. It achieved motion and UV grayscale information extraction (displayed with sunscreen), leading to fusion visual imaging. This work provides a promising co-design of material and device for a broadband and highly biomimetic optoelectronic neuromorphic system.

Suggested Citation

  • Pengshan Xie & Yunchao Xu & Jingwen Wang & Dengji Li & Yuxuan Zhang & Zixin Zeng & Boxiang Gao & Quan Quan & Bowen Li & You Meng & Weijun Wang & Yezhan Li & Yan Yan & Yi Shen & Jia Sun & Johnny C. Ho, 2024. "Birdlike broadband neuromorphic visual sensor arrays for fusion imaging," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
  • Handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-52563-4
    DOI: 10.1038/s41467-024-52563-4
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-024-52563-4
    File Function: Abstract
    Download Restriction: no

    File URL: https://libkey.io/10.1038/s41467-024-52563-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Kaushik Roy & Akhilesh Jaiswal & Priyadarshini Panda, 2019. "Towards spike-based machine intelligence with neuromorphic computing," Nature, Nature, vol. 575(7784), pages 607-617, November.
    2. Chao Du & Fuxi Cai & Mohammed A. Zidan & Wen Ma & Seung Hwan Lee & Wei D. Lu, 2017. "Reservoir computing using dynamic memristors for temporal information processing," Nature Communications, Nature, vol. 8(1), pages 1-10, December.
    3. Seongin Hong & Nicolò Zagni & Sooho Choo & Na Liu & Seungho Baek & Arindam Bala & Hocheon Yoo & Byung Ha Kang & Hyun Jae Kim & Hyung Joong Yun & Muhammad Ashraful Alam & Sunkook Kim, 2021. "Highly sensitive active pixel image sensor array driven by large-area bilayer MoS2 transistor circuitry," Nature Communications, Nature, vol. 12(1), pages 1-11, December.
    4. Xiangjing Wang & Chunsheng Chen & Li Zhu & Kailu Shi & Baocheng Peng & Yixin Zhu & Huiwu Mao & Haotian Long & Shuo Ke & Chuanyu Fu & Ying Zhu & Changjin Wan & Qing Wan, 2023. "Vertically integrated spiking cone photoreceptor arrays for color perception," Nature Communications, Nature, vol. 14(1), pages 1-9, December.
    5. He-Shan Zhang & Xue-Mei Dong & Zi-Cheng Zhang & Ze-Pu Zhang & Chao-Yi Ban & Zhe Zhou & Cheng Song & Shi-Qi Yan & Qian Xin & Ju-Qing Liu & Yin-Xiang Li & Wei Huang, 2022. "Co-assembled perylene/graphene oxide photosensitive heterobilayer for efficient neuromorphics," Nature Communications, Nature, vol. 13(1), pages 1-9, December.
    6. Tianshuo Qiu & Qiang An & Jianqi Wang & Jiafu Wang & Cheng-Wei Qiu & Shiyong Li & Hao Lv & Ming Cai & Jianyi Wang & Lin Cong & Shaobo Qu, 2024. "Vision-driven metasurfaces for perception enhancement," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    7. Daniel J. Gauthier & Erik Bollt & Aaron Griffith & Wendson A. S. Barbosa, 2021. "Next generation reservoir computing," Nature Communications, Nature, vol. 12(1), pages 1-8, December.
    8. Cynthia Tedore & Dan-Eric Nilsson, 2019. "Avian UV vision enhances leaf surface contrasts in forest environments," Nature Communications, Nature, vol. 10(1), pages 1-12, December.
    9. Xiaosong Wu & Shaocong Wang & Wei Huang & Yu Dong & Zhongrui Wang & Weiguo Huang, 2023. "Wearable in-sensor reservoir computing using optoelectronic polymers with through-space charge-transport characteristics for multi-task learning," Nature Communications, Nature, vol. 14(1), pages 1-12, December.
    10. Guangdong Zhou & Jie Li & Qunliang Song & Lidan Wang & Zhijun Ren & Bai Sun & Xiaofang Hu & Wenhua Wang & Gaobo Xu & Xiaodie Chen & Lan Cheng & Feichi Zhou & Shukai Duan, 2023. "Full hardware implementation of neuromorphic visual system based on multimodal optoelectronic resistive memory arrays for versatile image processing," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    11. Shuzhi Liu & Jianmin Zeng & Zhixin Wu & Han Hu & Ao Xu & Xiaohe Huang & Weilin Chen & Qilai Chen & Zhe Yu & Yinyu Zhao & Rong Wang & Tingting Han & Chao Li & Pingqi Gao & Hyunwoo Kim & Seung Jae Baik , 2023. "An ultrasmall organic synapse for neuromorphic computing," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    12. Leilei Gu & Swapnadeep Poddar & Yuanjing Lin & Zhenghao Long & Daquan Zhang & Qianpeng Zhang & Lei Shu & Xiao Qiu & Matthew Kam & Ali Javey & Zhiyong Fan, 2020. "A biomimetic eye with a hemispherical perovskite nanowire array retina," Nature, Nature, vol. 581(7808), pages 278-282, May.
    13. Ting Jiang & Yiru Wang & Yingshuang Zheng & Le Wang & Xiang He & Liqiang Li & Yunfeng Deng & Huanli Dong & Hongkun Tian & Yanhou Geng & Linghai Xie & Yong Lei & Haifeng Ling & Deyang Ji & Wenping Hu, 2023. "Tetrachromatic vision-inspired neuromorphic sensors with ultraweak ultraviolet detection," Nature Communications, Nature, vol. 14(1), pages 1-9, December.
    14. Sung Yun Son & Giwon Lee & Hongyu Wang & Stephanie Samson & Qingshan Wei & Yong Zhu & Wei You, 2022. "Integrating charge mobility, stability and stretchability within conjugated polymer films for stretchable multifunctional sensors," Nature Communications, Nature, vol. 13(1), pages 1-11, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ruomin Zhu & Sam Lilak & Alon Loeffler & Joseph Lizier & Adam Stieg & James Gimzewski & Zdenka Kuncic, 2023. "Online dynamical learning and sequence memory with neuromorphic nanowire networks," Nature Communications, Nature, vol. 14(1), pages 1-12, December.
    2. Tao Guo & Shasha Li & Y. Norman Zhou & Wei D. Lu & Yong Yan & Yimin A. Wu, 2024. "Interspecies-chimera machine vision with polarimetry for real-time navigation and anti-glare pattern recognition," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    3. Yang, J. & Primo, E. & Aleja, D. & Criado, R. & Boccaletti, S. & Alfaro-Bittner, K., 2022. "Implementing and morphing Boolean gates with adaptive synchronization: The case of spiking neurons," Chaos, Solitons & Fractals, Elsevier, vol. 162(C).
    4. Fanfan Li & Dingwei Li & Chuanqing Wang & Guolei Liu & Rui Wang & Huihui Ren & Yingjie Tang & Yan Wang & Yitong Chen & Kun Liang & Qi Huang & Mohamad Sawan & Min Qiu & Hong Wang & Bowen Zhu, 2024. "An artificial visual neuron with multiplexed rate and time-to-first-spike coding," Nature Communications, Nature, vol. 15(1), pages 1-11, December.
    5. Pei-Yu Huang & Bi-Yi Jiang & Hong-Ji Chen & Jia-Yi Xu & Kang Wang & Cheng-Yi Zhu & Xin-Yan Hu & Dong Li & Liang Zhen & Fei-Chi Zhou & Jing-Kai Qin & Cheng-Yan Xu, 2023. "Neuro-inspired optical sensor array for high-accuracy static image recognition and dynamic trace extraction," Nature Communications, Nature, vol. 14(1), pages 1-9, December.
    6. Zhiyuan Li & Zhongshao Li & Wei Tang & Jiaping Yao & Zhipeng Dou & Junjie Gong & Yongfei Li & Beining Zhang & Yunxiao Dong & Jian Xia & Lin Sun & Peng Jiang & Xun Cao & Rui Yang & Xiangshui Miao & Ron, 2024. "Crossmodal sensory neurons based on high-performance flexible memristors for human-machine in-sensor computing system," Nature Communications, Nature, vol. 15(1), pages 1-11, December.
    7. Pengzhan Li & Mingzhen Zhang & Qingli Zhou & Qinghua Zhang & Donggang Xie & Ge Li & Zhuohui Liu & Zheng Wang & Erjia Guo & Meng He & Can Wang & Lin Gu & Guozhen Yang & Kuijuan Jin & Chen Ge, 2024. "Reconfigurable optoelectronic transistors for multimodal recognition," Nature Communications, Nature, vol. 15(1), pages 1-11, December.
    8. Chengyu Wang & Yangshuang Bian & Kai Liu & Mingcong Qin & Fan Zhang & Mingliang Zhu & Wenkang Shi & Mingchao Shao & Shengcong Shang & Jiaxin Hong & Zhiheng Zhu & Zhiyuan Zhao & Yunqi Liu & Yunlong Guo, 2024. "Strain-insensitive viscoelastic perovskite film for intrinsically stretchable neuromorphic vision-adaptive transistors," Nature Communications, Nature, vol. 15(1), pages 1-10, December.
    9. Changsong Gao & Di Liu & Chenhui Xu & Weidong Xie & Xianghong Zhang & Junhua Bai & Zhixian Lin & Cheng Zhang & Yuanyuan Hu & Tailiang Guo & Huipeng Chen, 2024. "Toward grouped-reservoir computing: organic neuromorphic vertical transistor with distributed reservoir states for efficient recognition and prediction," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    10. Min Yan & Can Huang & Peter Bienstman & Peter Tino & Wei Lin & Jie Sun, 2024. "Emerging opportunities and challenges for the future of reservoir computing," Nature Communications, Nature, vol. 15(1), pages 1-18, December.
    11. Gianluca Fabiani & Nikolaos Evangelou & Tianqi Cui & Juan M. Bello-Rivas & Cristina P. Martin-Linares & Constantinos Siettos & Ioannis G. Kevrekidis, 2024. "Task-oriented machine learning surrogates for tipping points of agent-based models," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    12. Zequn Lin & Zhaofan Lu & Zengru Di & Ying Tang, 2024. "Learning noise-induced transitions by multi-scaling reservoir computing," Nature Communications, Nature, vol. 15(1), pages 1-10, December.
    13. Shaomei Lin & Weifeng Yang & Xubin Zhu & Yubin Lan & Kerui Li & Qinghong Zhang & Yaogang Li & Chengyi Hou & Hongzhi Wang, 2024. "Triboelectric micro-flexure-sensitive fiber electronics," Nature Communications, Nature, vol. 15(1), pages 1-10, December.
    14. Zhiwei Chen & Wenjie Li & Zhen Fan & Shuai Dong & Yihong Chen & Minghui Qin & Min Zeng & Xubing Lu & Guofu Zhou & Xingsen Gao & Jun-Ming Liu, 2023. "All-ferroelectric implementation of reservoir computing," Nature Communications, Nature, vol. 14(1), pages 1-12, December.
    15. Rohit Abraham John & Yiğit Demirağ & Yevhen Shynkarenko & Yuliia Berezovska & Natacha Ohannessian & Melika Payvand & Peng Zeng & Maryna I. Bodnarchuk & Frank Krumeich & Gökhan Kara & Ivan Shorubalko &, 2022. "Reconfigurable halide perovskite nanocrystal memristors for neuromorphic computing," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    16. Xiangpeng Liang & Yanan Zhong & Jianshi Tang & Zhengwu Liu & Peng Yao & Keyang Sun & Qingtian Zhang & Bin Gao & Hadi Heidari & He Qian & Huaqiang Wu, 2022. "Rotating neurons for all-analog implementation of cyclic reservoir computing," Nature Communications, Nature, vol. 13(1), pages 1-11, December.
    17. Hongwei Tan & Sebastiaan van Dijken, 2023. "Dynamic machine vision with retinomorphic photomemristor-reservoir computing," Nature Communications, Nature, vol. 14(1), pages 1-9, December.
    18. Jongmin Lee & Bum Ho Jeong & Eswaran Kamaraj & Dohyung Kim & Hakjun Kim & Sanghyuk Park & Hui Joon Park, 2023. "Light-enhanced molecular polarity enabling multispectral color-cognitive memristor for neuromorphic visual system," Nature Communications, Nature, vol. 14(1), pages 1-19, December.
    19. Lina Jaurigue & Kathy Lüdge, 2022. "Connecting reservoir computing with statistical forecasting and deep neural networks," Nature Communications, Nature, vol. 13(1), pages 1-3, December.
    20. Bo Dai & Liang Zhang & Chenglong Zhao & Hunter Bachman & Ryan Becker & John Mai & Ziao Jiao & Wei Li & Lulu Zheng & Xinjun Wan & Tony Jun Huang & Songlin Zhuang & Dawei Zhang, 2021. "Biomimetic apposition compound eye fabricated using microfluidic-assisted 3D printing," Nature Communications, Nature, vol. 12(1), pages 1-11, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-52563-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.