IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v14y2023i1d10.1038_s41467-023-37886-y.html
   My bibliography  Save this article

Dynamic machine vision with retinomorphic photomemristor-reservoir computing

Author

Listed:
  • Hongwei Tan

    (Aalto University School of Science)

  • Sebastiaan van Dijken

    (Aalto University School of Science)

Abstract

Dynamic machine vision requires recognizing the past and predicting the future of a moving object based on present vision. Current machine vision systems accomplish this by processing numerous image frames or using complex algorithms. Here, we report motion recognition and prediction in recurrent photomemristor networks. In our system, a retinomorphic photomemristor array, working as dynamic vision reservoir, embeds past motion frames as hidden states into the present frame through inherent dynamic memory. The informative present frame facilitates accurate recognition of past and prediction of future motions with machine learning algorithms. This in-sensor motion processing capability eliminates redundant data flows and promotes real-time perception of moving objects for dynamic machine vision.

Suggested Citation

  • Hongwei Tan & Sebastiaan van Dijken, 2023. "Dynamic machine vision with retinomorphic photomemristor-reservoir computing," Nature Communications, Nature, vol. 14(1), pages 1-9, December.
    • Handle: RePEc:nat:natcom:v:14:y:2023:i:1:d:10.1038_s41467-023-37886-y
    DOI: 10.1038/s41467-023-37886-y
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-023-37886-y
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-023-37886-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Chao Du & Fuxi Cai & Mohammed A. Zidan & Wen Ma & Seung Hwan Lee & Wei D. Lu, 2017. "Reservoir computing using dynamic memristors for temporal information processing," Nature Communications, Nature, vol. 8(1), pages 1-10, December.
    2. Hongwei Tan & Yifan Zhou & Quanzheng Tao & Johanna Rosen & Sebastiaan van Dijken, 2021. "Bioinspired multisensory neural network with crossmodal integration and recognition," Nature Communications, Nature, vol. 12(1), pages 1-9, December.
    3. Steven J. Luck & Edward K. Vogel, 1997. "The capacity of visual working memory for features and conjunctions," Nature, Nature, vol. 390(6657), pages 279-281, November.
    4. Hongwei Tan & Quanzheng Tao & Ishan Pande & Sayani Majumdar & Fu Liu & Yifan Zhou & Per O. Å. Persson & Johanna Rosen & Sebastiaan van Dijken, 2020. "Tactile sensory coding and learning with bio-inspired optoelectronic spiking afferent nerves," Nature Communications, Nature, vol. 11(1), pages 1-9, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Pengzhan Li & Mingzhen Zhang & Qingli Zhou & Qinghua Zhang & Donggang Xie & Ge Li & Zhuohui Liu & Zheng Wang & Erjia Guo & Meng He & Can Wang & Lin Gu & Guozhen Yang & Kuijuan Jin & Chen Ge, 2024. "Reconfigurable optoelectronic transistors for multimodal recognition," Nature Communications, Nature, vol. 15(1), pages 1-11, December.
    2. Changsong Gao & Di Liu & Chenhui Xu & Weidong Xie & Xianghong Zhang & Junhua Bai & Zhixian Lin & Cheng Zhang & Yuanyuan Hu & Tailiang Guo & Huipeng Chen, 2024. "Toward grouped-reservoir computing: organic neuromorphic vertical transistor with distributed reservoir states for efficient recognition and prediction," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    3. Klinshov, Vladimir V. & Kovalchuk, Andrey V. & Soloviev, Igor A. & Maslennikov, Oleg V. & Franović, Igor & Perc, Matjaž, 2024. "Extending dynamic memory of spiking neuron networks," Chaos, Solitons & Fractals, Elsevier, vol. 182(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhongfang Zhang & Xiaolong Zhao & Xumeng Zhang & Xiaohu Hou & Xiaolan Ma & Shuangzhu Tang & Ying Zhang & Guangwei Xu & Qi Liu & Shibing Long, 2022. "In-sensor reservoir computing system for latent fingerprint recognition with deep ultraviolet photo-synapses and memristor array," Nature Communications, Nature, vol. 13(1), pages 1-9, December.
    2. Changsong Gao & Di Liu & Chenhui Xu & Weidong Xie & Xianghong Zhang & Junhua Bai & Zhixian Lin & Cheng Zhang & Yuanyuan Hu & Tailiang Guo & Huipeng Chen, 2024. "Toward grouped-reservoir computing: organic neuromorphic vertical transistor with distributed reservoir states for efficient recognition and prediction," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    3. Rui Yuan & Qingxi Duan & Pek Jun Tiw & Ge Li & Zhuojian Xiao & Zhaokun Jing & Ke Yang & Chang Liu & Chen Ge & Ru Huang & Yuchao Yang, 2022. "A calibratable sensory neuron based on epitaxial VO2 for spike-based neuromorphic multisensory system," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    4. Igor S. Utochkin & Vladislav A. Khvostov & Yulia M. Stakina, 2017. "Ensemble-Based Segmentation in the Perception of Multiple Feature Conjunctions," HSE Working papers WP BRP 78/PSY/2017, National Research University Higher School of Economics.
    5. Jastrzębski, Jan & Ciechanowska, Iwona & Chuderski, Adam, 2018. "The strong link between fluid intelligence and working memory cannot be explained away by strategy use," Intelligence, Elsevier, vol. 66(C), pages 44-53.
    6. Zhiwei Chen & Wenjie Li & Zhen Fan & Shuai Dong & Yihong Chen & Minghui Qin & Min Zeng & Xubing Lu & Guofu Zhou & Xingsen Gao & Jun-Ming Liu, 2023. "All-ferroelectric implementation of reservoir computing," Nature Communications, Nature, vol. 14(1), pages 1-12, December.
    7. Aki Kondo & Jun Saiki, 2012. "Feature-Specific Encoding Flexibility in Visual Working Memory," PLOS ONE, Public Library of Science, vol. 7(12), pages 1-8, December.
    8. Rohit Abraham John & Yiğit Demirağ & Yevhen Shynkarenko & Yuliia Berezovska & Natacha Ohannessian & Melika Payvand & Peng Zeng & Maryna I. Bodnarchuk & Frank Krumeich & Gökhan Kara & Ivan Shorubalko &, 2022. "Reconfigurable halide perovskite nanocrystal memristors for neuromorphic computing," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    9. Xiangpeng Liang & Yanan Zhong & Jianshi Tang & Zhengwu Liu & Peng Yao & Keyang Sun & Qingtian Zhang & Bin Gao & Hadi Heidari & He Qian & Huaqiang Wu, 2022. "Rotating neurons for all-analog implementation of cyclic reservoir computing," Nature Communications, Nature, vol. 13(1), pages 1-11, December.
    10. Jongmin Lee & Bum Ho Jeong & Eswaran Kamaraj & Dohyung Kim & Hakjun Kim & Sanghyuk Park & Hui Joon Park, 2023. "Light-enhanced molecular polarity enabling multispectral color-cognitive memristor for neuromorphic visual system," Nature Communications, Nature, vol. 14(1), pages 1-19, December.
    11. Robert W. Faff & Sebastian Kernbach, 2021. "A visualisation approach for pitching research," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 61(4), pages 5177-5197, December.
    12. Yuri A. Markov & Natalia A. Tiurina & Igor S. Utochkin, 2018. "Different features are stored independently in visual working memory but mediated by object-based representations," HSE Working papers WP BRP 101/PSY/2018, National Research University Higher School of Economics.
    13. Tullo, Domenico & Faubert, Jocelyn & Bertone, Armando, 2018. "The characterization of attention resource capacity and its relationship with fluid reasoning intelligence: A multiple object tracking study," Intelligence, Elsevier, vol. 69(C), pages 158-168.
    14. Jifan Zhou & Jun Yin & Tong Chen & Xiaowei Ding & Zaifeng Gao & Mowei Shen, 2011. "Visual Working Memory Capacity Does Not Modulate the Feature-Based Information Filtering in Visual Working Memory," PLOS ONE, Public Library of Science, vol. 6(9), pages 1-10, September.
    15. Yang, J. & Primo, E. & Aleja, D. & Criado, R. & Boccaletti, S. & Alfaro-Bittner, K., 2022. "Implementing and morphing Boolean gates with adaptive synchronization: The case of spiking neurons," Chaos, Solitons & Fractals, Elsevier, vol. 162(C).
    16. Jasmin Lehmann & Lorenz Granrath & Ryan Browne & Toshimi Ogawa & Keisuke Kokubun & Yasuyuki Taki & Kristiina Jokinen & Sarah Janboecke & Christophe Lohr & Rainer Wieching & Roberta Bevilacqua & Sara C, 2024. "Digital Twins for Supporting Ageing Well: Approaches in Current Research and Innovation in Europe and Japan," Sustainability, MDPI, vol. 16(7), pages 1-16, April.
    17. Nathaniel J. S. Ashby & Stephan Dickert & Andreas Glockner, 2012. "Focusing on what you own: Biased information uptake due to ownership," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 7(3), pages 254-267, May.
    18. Giovanni Maria Matrone & Eveline R. W. Doremaele & Abhijith Surendran & Zachary Laswick & Sophie Griggs & Gang Ye & Iain McCulloch & Francesca Santoro & Jonathan Rivnay & Yoeri Burgt, 2024. "A modular organic neuromorphic spiking circuit for retina-inspired sensory coding and neurotransmitter-mediated neural pathways," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    19. Yan Wang & Yue Gong & Shenming Huang & Xuechao Xing & Ziyu Lv & Junjie Wang & Jia-Qin Yang & Guohua Zhang & Ye Zhou & Su-Ting Han, 2021. "Memristor-based biomimetic compound eye for real-time collision detection," Nature Communications, Nature, vol. 12(1), pages 1-12, December.
    20. Pengshan Xie & Yunchao Xu & Jingwen Wang & Dengji Li & Yuxuan Zhang & Zixin Zeng & Boxiang Gao & Quan Quan & Bowen Li & You Meng & Weijun Wang & Yezhan Li & Yan Yan & Yi Shen & Jia Sun & Johnny C. Ho, 2024. "Birdlike broadband neuromorphic visual sensor arrays for fusion imaging," Nature Communications, Nature, vol. 15(1), pages 1-12, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:14:y:2023:i:1:d:10.1038_s41467-023-37886-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.