IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v14y2023i1d10.1038_s41467-023-37886-y.html
   My bibliography  Save this article

Dynamic machine vision with retinomorphic photomemristor-reservoir computing

Author

Listed:
  • Hongwei Tan

    (Aalto University School of Science)

  • Sebastiaan van Dijken

    (Aalto University School of Science)

Abstract

Dynamic machine vision requires recognizing the past and predicting the future of a moving object based on present vision. Current machine vision systems accomplish this by processing numerous image frames or using complex algorithms. Here, we report motion recognition and prediction in recurrent photomemristor networks. In our system, a retinomorphic photomemristor array, working as dynamic vision reservoir, embeds past motion frames as hidden states into the present frame through inherent dynamic memory. The informative present frame facilitates accurate recognition of past and prediction of future motions with machine learning algorithms. This in-sensor motion processing capability eliminates redundant data flows and promotes real-time perception of moving objects for dynamic machine vision.

Suggested Citation

  • Hongwei Tan & Sebastiaan van Dijken, 2023. "Dynamic machine vision with retinomorphic photomemristor-reservoir computing," Nature Communications, Nature, vol. 14(1), pages 1-9, December.
    • Handle: RePEc:nat:natcom:v:14:y:2023:i:1:d:10.1038_s41467-023-37886-y
    DOI: 10.1038/s41467-023-37886-y
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-023-37886-y
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-023-37886-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Chao Du & Fuxi Cai & Mohammed A. Zidan & Wen Ma & Seung Hwan Lee & Wei D. Lu, 2017. "Reservoir computing using dynamic memristors for temporal information processing," Nature Communications, Nature, vol. 8(1), pages 1-10, December.
    2. Steven J. Luck & Edward K. Vogel, 1997. "The capacity of visual working memory for features and conjunctions," Nature, Nature, vol. 390(6657), pages 279-281, November.
    3. Hongwei Tan & Yifan Zhou & Quanzheng Tao & Johanna Rosen & Sebastiaan van Dijken, 2021. "Bioinspired multisensory neural network with crossmodal integration and recognition," Nature Communications, Nature, vol. 12(1), pages 1-9, December.
    4. Hongwei Tan & Quanzheng Tao & Ishan Pande & Sayani Majumdar & Fu Liu & Yifan Zhou & Per O. Å. Persson & Johanna Rosen & Sebastiaan van Dijken, 2020. "Tactile sensory coding and learning with bio-inspired optoelectronic spiking afferent nerves," Nature Communications, Nature, vol. 11(1), pages 1-9, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Pengzhan Li & Mingzhen Zhang & Qingli Zhou & Qinghua Zhang & Donggang Xie & Ge Li & Zhuohui Liu & Zheng Wang & Erjia Guo & Meng He & Can Wang & Lin Gu & Guozhen Yang & Kuijuan Jin & Chen Ge, 2024. "Reconfigurable optoelectronic transistors for multimodal recognition," Nature Communications, Nature, vol. 15(1), pages 1-11, December.
    2. Changsong Gao & Di Liu & Chenhui Xu & Weidong Xie & Xianghong Zhang & Junhua Bai & Zhixian Lin & Cheng Zhang & Yuanyuan Hu & Tailiang Guo & Huipeng Chen, 2024. "Toward grouped-reservoir computing: organic neuromorphic vertical transistor with distributed reservoir states for efficient recognition and prediction," Nature Communications, Nature, vol. 15(1), pages 1-13, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhongfang Zhang & Xiaolong Zhao & Xumeng Zhang & Xiaohu Hou & Xiaolan Ma & Shuangzhu Tang & Ying Zhang & Guangwei Xu & Qi Liu & Shibing Long, 2022. "In-sensor reservoir computing system for latent fingerprint recognition with deep ultraviolet photo-synapses and memristor array," Nature Communications, Nature, vol. 13(1), pages 1-9, December.
    2. Changsong Gao & Di Liu & Chenhui Xu & Weidong Xie & Xianghong Zhang & Junhua Bai & Zhixian Lin & Cheng Zhang & Yuanyuan Hu & Tailiang Guo & Huipeng Chen, 2024. "Toward grouped-reservoir computing: organic neuromorphic vertical transistor with distributed reservoir states for efficient recognition and prediction," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    3. Rui Yuan & Qingxi Duan & Pek Jun Tiw & Ge Li & Zhuojian Xiao & Zhaokun Jing & Ke Yang & Chang Liu & Chen Ge & Ru Huang & Yuchao Yang, 2022. "A calibratable sensory neuron based on epitaxial VO2 for spike-based neuromorphic multisensory system," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    4. Zhiwei Chen & Wenjie Li & Zhen Fan & Shuai Dong & Yihong Chen & Minghui Qin & Min Zeng & Xubing Lu & Guofu Zhou & Xingsen Gao & Jun-Ming Liu, 2023. "All-ferroelectric implementation of reservoir computing," Nature Communications, Nature, vol. 14(1), pages 1-12, December.
    5. Aki Kondo & Jun Saiki, 2012. "Feature-Specific Encoding Flexibility in Visual Working Memory," PLOS ONE, Public Library of Science, vol. 7(12), pages 1-8, December.
    6. Xiangpeng Liang & Yanan Zhong & Jianshi Tang & Zhengwu Liu & Peng Yao & Keyang Sun & Qingtian Zhang & Bin Gao & Hadi Heidari & He Qian & Huaqiang Wu, 2022. "Rotating neurons for all-analog implementation of cyclic reservoir computing," Nature Communications, Nature, vol. 13(1), pages 1-11, December.
    7. Jongmin Lee & Bum Ho Jeong & Eswaran Kamaraj & Dohyung Kim & Hakjun Kim & Sanghyuk Park & Hui Joon Park, 2023. "Light-enhanced molecular polarity enabling multispectral color-cognitive memristor for neuromorphic visual system," Nature Communications, Nature, vol. 14(1), pages 1-19, December.
    8. Tullo, Domenico & Faubert, Jocelyn & Bertone, Armando, 2018. "The characterization of attention resource capacity and its relationship with fluid reasoning intelligence: A multiple object tracking study," Intelligence, Elsevier, vol. 69(C), pages 158-168.
    9. Jifan Zhou & Jun Yin & Tong Chen & Xiaowei Ding & Zaifeng Gao & Mowei Shen, 2011. "Visual Working Memory Capacity Does Not Modulate the Feature-Based Information Filtering in Visual Working Memory," PLOS ONE, Public Library of Science, vol. 6(9), pages 1-10, September.
    10. Yang, J. & Primo, E. & Aleja, D. & Criado, R. & Boccaletti, S. & Alfaro-Bittner, K., 2022. "Implementing and morphing Boolean gates with adaptive synchronization: The case of spiking neurons," Chaos, Solitons & Fractals, Elsevier, vol. 162(C).
    11. Jasmin Lehmann & Lorenz Granrath & Ryan Browne & Toshimi Ogawa & Keisuke Kokubun & Yasuyuki Taki & Kristiina Jokinen & Sarah Janboecke & Christophe Lohr & Rainer Wieching & Roberta Bevilacqua & Sara C, 2024. "Digital Twins for Supporting Ageing Well: Approaches in Current Research and Innovation in Europe and Japan," Sustainability, MDPI, vol. 16(7), pages 1-16, April.
    12. Giovanni Maria Matrone & Eveline R. W. Doremaele & Abhijith Surendran & Zachary Laswick & Sophie Griggs & Gang Ye & Iain McCulloch & Francesca Santoro & Jonathan Rivnay & Yoeri Burgt, 2024. "A modular organic neuromorphic spiking circuit for retina-inspired sensory coding and neurotransmitter-mediated neural pathways," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    13. Li, Qian & Huang, Zhuowei (Joy) & Christianson, Kiel, 2016. "Visual attention toward tourism photographs with text: An eye-tracking study," Tourism Management, Elsevier, vol. 54(C), pages 243-258.
    14. Yuri A. Markov & Igor S. Utochkin, 2017. "The Effect of Object Distinctiveness on Object-Location Binding in Visual Working Memory," HSE Working papers WP BRP 79/PSY/2017, National Research University Higher School of Economics.
    15. S. Cerreia-Vioglio & F. Maccheroni & M. Marinacci & A. Rustichini, 2017. "Multinomial logit processes and preference discovery: inside and outside the black box," Working Papers 615, IGIER (Innocenzo Gasparini Institute for Economic Research), Bocconi University.
    16. Yaqian Liu & Di Liu & Changsong Gao & Xianghong Zhang & Rengjian Yu & Xiumei Wang & Enlong Li & Yuanyuan Hu & Tailiang Guo & Huipeng Chen, 2022. "Self-powered high-sensitivity all-in-one vertical tribo-transistor device for multi-sensing-memory-computing," Nature Communications, Nature, vol. 13(1), pages 1-11, December.
    17. Zhuohui Liu & Qinghua Zhang & Donggang Xie & Mingzhen Zhang & Xinyan Li & Hai Zhong & Ge Li & Meng He & Dashan Shang & Can Wang & Lin Gu & Guozhen Yang & Kuijuan Jin & Chen Ge, 2023. "Interface-type tunable oxygen ion dynamics for physical reservoir computing," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    18. Ociepka, Michał & Kałamała, Patrycja & Chuderski, Adam, 2022. "High individual alpha frequency brains run fast, but it does not make them smart," Intelligence, Elsevier, vol. 92(C).
    19. repec:cup:judgdm:v:7:y:2012:i:3:p:254-267 is not listed on IDEAS
    20. Shaiyan Keshvari & Ronald van den Berg & Wei Ji Ma, 2013. "No Evidence for an Item Limit in Change Detection," PLOS Computational Biology, Public Library of Science, vol. 9(2), pages 1-9, February.
    21. Bin Zhu & Stephanie A. Watts, 2010. "Visualization of Network Concepts: The Impact of Working Memory Capacity Differences," Information Systems Research, INFORMS, vol. 21(2), pages 327-344, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:14:y:2023:i:1:d:10.1038_s41467-023-37886-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.