IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v13y2022i1d10.1038_s41467-022-32790-3.html
   My bibliography  Save this article

In-sensor image memorization and encoding via optical neurons for bio-stimulus domain reduction toward visual cognitive processing

Author

Listed:
  • Doeon Lee

    (University of Virginia)

  • Minseong Park

    (University of Virginia)

  • Yongmin Baek

    (University of Virginia)

  • Byungjoon Bae

    (University of Virginia)

  • Junseok Heo

    (Ajou University)

  • Kyusang Lee

    (University of Virginia
    University of Virginia)

Abstract

As machine vision technology generates large amounts of data from sensors, it requires efficient computational systems for visual cognitive processing. Recently, in-sensor computing systems have emerged as a potential solution for reducing unnecessary data transfer and realizing fast and energy-efficient visual cognitive processing. However, they still lack the capability to process stored images directly within the sensor. Here, we demonstrate a heterogeneously integrated 1-photodiode and 1 memristor (1P-1R) crossbar for in-sensor visual cognitive processing, emulating a mammalian image encoding process to extract features from the input images. Unlike other neuromorphic vision processes, the trained weight values are applied as an input voltage to the image-saved crossbar array instead of storing the weight value in the memristors, realizing the in-sensor computing paradigm. We believe the heterogeneously integrated in-sensor computing platform provides an advanced architecture for real-time and data-intensive machine-vision applications via bio-stimulus domain reduction.

Suggested Citation

  • Doeon Lee & Minseong Park & Yongmin Baek & Byungjoon Bae & Junseok Heo & Kyusang Lee, 2022. "In-sensor image memorization and encoding via optical neurons for bio-stimulus domain reduction toward visual cognitive processing," Nature Communications, Nature, vol. 13(1), pages 1-9, December.
  • Handle: RePEc:nat:natcom:v:13:y:2022:i:1:d:10.1038_s41467-022-32790-3
    DOI: 10.1038/s41467-022-32790-3
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-022-32790-3
    File Function: Abstract
    Download Restriction: no

    File URL: https://libkey.io/10.1038/s41467-022-32790-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Changsoon Choi & Juyoung Leem & Minsung Kim & Amir Taqieddin & Chullhee Cho & Kyoung Won Cho & Gil Ju Lee & Hyojin Seung & Hyung Jong Bae & Young Min Song & Taeghwan Hyeon & Narayana R. Aluru & SungWo, 2020. "Curved neuromorphic image sensor array using a MoS2-organic heterostructure inspired by the human visual recognition system," Nature Communications, Nature, vol. 11(1), pages 1-9, December.
    2. Peng Yao & Huaqiang Wu & Bin Gao & Jianshi Tang & Qingtian Zhang & Wenqiang Zhang & J. Joshua Yang & He Qian, 2020. "Fully hardware-implemented memristor convolutional neural network," Nature, Nature, vol. 577(7792), pages 641-646, January.
    3. M. Prezioso & F. Merrikh-Bayat & B. D. Hoskins & G. C. Adam & K. K. Likharev & D. B. Strukov, 2015. "Training and operation of an integrated neuromorphic network based on metal-oxide memristors," Nature, Nature, vol. 521(7550), pages 61-64, May.
    4. Lukas Mennel & Joanna Symonowicz & Stefan Wachter & Dmitry K. Polyushkin & Aday J. Molina-Mendoza & Thomas Mueller, 2020. "Ultrafast machine vision with 2D material neural network image sensors," Nature, Nature, vol. 579(7797), pages 62-66, March.
    5. Yang Chai, 2020. "In-sensor computing for machine vision," Nature, Nature, vol. 579(7797), pages 32-33, March.
    6. Qian-Bing Zhu & Bo Li & Dan-Dan Yang & Chi Liu & Shun Feng & Mao-Lin Chen & Yun Sun & Ya-Nan Tian & Xin Su & Xiao-Mu Wang & Song Qiu & Qing-Wen Li & Xiao-Ming Li & Hai-Bo Zeng & Hui-Ming Cheng & Dong-, 2021. "A flexible ultrasensitive optoelectronic sensor array for neuromorphic vision systems," Nature Communications, Nature, vol. 12(1), pages 1-7, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Changsoon Choi & Henry Hinton & Hyojin Seung & Sehui Chang & Ji Su Kim & Woosang You & Min Sung Kim & Jung Pyo Hong & Jung Ah Lim & Do Kyung Hwang & Gil Ju Lee & Houk Jang & Young Min Song & Dae-Hyeon, 2024. "Anti-distortion bioinspired camera with an inhomogeneous photo-pixel array," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    2. Guangdong Zhou & Jie Li & Qunliang Song & Lidan Wang & Zhijun Ren & Bai Sun & Xiaofang Hu & Wenhua Wang & Gaobo Xu & Xiaodie Chen & Lan Cheng & Feichi Zhou & Shukai Duan, 2023. "Full hardware implementation of neuromorphic visual system based on multimodal optoelectronic resistive memory arrays for versatile image processing," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    3. Dmytro D. Yaremkevich & Alexey V. Scherbakov & Luke Clerk & Serhii M. Kukhtaruk & Achim Nadzeyka & Richard Campion & Andrew W. Rushforth & Sergey Savel’ev & Alexander G. Balanov & Manfred Bayer, 2023. "On-chip phonon-magnon reservoir for neuromorphic computing," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    4. Zhuiri Peng & Lei Tong & Wenhao Shi & Langlang Xu & Xinyu Huang & Zheng Li & Xiangxiang Yu & Xiaohan Meng & Xiao He & Shengjie Lv & Gaochen Yang & Hao Hao & Tian Jiang & Xiangshui Miao & Lei Ye, 2024. "Multifunctional human visual pathway-replicated hardware based on 2D materials," Nature Communications, Nature, vol. 15(1), pages 1-9, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tian Zhang & Xin Guo & Pan Wang & Xinyi Fan & Zichen Wang & Yan Tong & Decheng Wang & Limin Tong & Linjun Li, 2024. "High performance artificial visual perception and recognition with a plasmon-enhanced 2D material neural network," Nature Communications, Nature, vol. 15(1), pages 1-10, December.
    2. Guangdong Zhou & Jie Li & Qunliang Song & Lidan Wang & Zhijun Ren & Bai Sun & Xiaofang Hu & Wenhua Wang & Gaobo Xu & Xiaodie Chen & Lan Cheng & Feichi Zhou & Shukai Duan, 2023. "Full hardware implementation of neuromorphic visual system based on multimodal optoelectronic resistive memory arrays for versatile image processing," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    3. Peng Chen & Fenghao Liu & Peng Lin & Peihong Li & Yu Xiao & Bihua Zhang & Gang Pan, 2023. "Open-loop analog programmable electrochemical memory array," Nature Communications, Nature, vol. 14(1), pages 1-9, December.
    4. Tao Guo & Shasha Li & Y. Norman Zhou & Wei D. Lu & Yong Yan & Yimin A. Wu, 2024. "Interspecies-chimera machine vision with polarimetry for real-time navigation and anti-glare pattern recognition," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    5. Djohan Bonnet & Tifenn Hirtzlin & Atreya Majumdar & Thomas Dalgaty & Eduardo Esmanhotto & Valentina Meli & Niccolo Castellani & Simon Martin & Jean-François Nodin & Guillaume Bourgeois & Jean-Michel P, 2023. "Bringing uncertainty quantification to the extreme-edge with memristor-based Bayesian neural networks," Nature Communications, Nature, vol. 14(1), pages 1-13, December.
    6. Ruibin Mao & Bo Wen & Arman Kazemi & Yahui Zhao & Ann Franchesca Laguna & Rui Lin & Ngai Wong & Michael Niemier & X. Sharon Hu & Xia Sheng & Catherine E. Graves & John Paul Strachan & Can Li, 2022. "Experimentally validated memristive memory augmented neural network with efficient hashing and similarity search," Nature Communications, Nature, vol. 13(1), pages 1-13, December.
    7. Pei-Yu Huang & Bi-Yi Jiang & Hong-Ji Chen & Jia-Yi Xu & Kang Wang & Cheng-Yi Zhu & Xin-Yan Hu & Dong Li & Liang Zhen & Fei-Chi Zhou & Jing-Kai Qin & Cheng-Yan Xu, 2023. "Neuro-inspired optical sensor array for high-accuracy static image recognition and dynamic trace extraction," Nature Communications, Nature, vol. 14(1), pages 1-9, December.
    8. Bin Gao & Ying Zhou & Qingtian Zhang & Shuanglin Zhang & Peng Yao & Yue Xi & Qi Liu & Meiran Zhao & Wenqiang Zhang & Zhengwu Liu & Xinyi Li & Jianshi Tang & He Qian & Huaqiang Wu, 2022. "Memristor-based analogue computing for brain-inspired sound localization with in situ training," Nature Communications, Nature, vol. 13(1), pages 1-8, December.
    9. Ik-Jyae Kim & Min-Kyu Kim & Jang-Sik Lee, 2023. "Highly-scaled and fully-integrated 3-dimensional ferroelectric transistor array for hardware implementation of neural networks," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    10. Chenhao Wang & Xinyi Xu & Xiaodong Pi & Mark D. Butala & Wen Huang & Lei Yin & Wenbing Peng & Munir Ali & Srikrishna Chanakya Bodepudi & Xvsheng Qiao & Yang Xu & Wei Sun & Deren Yang, 2022. "Neuromorphic device based on silicon nanosheets," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    11. Ren, Lujie & Mou, Jun & Banerjee, Santo & Zhang, Yushu, 2023. "A hyperchaotic map with a new discrete memristor model: Design, dynamical analysis, implementation and application," Chaos, Solitons & Fractals, Elsevier, vol. 167(C).
    12. Jangsaeng Kim & Eun Chan Park & Wonjun Shin & Ryun-Han Koo & Chang-Hyeon Han & He Young Kang & Tae Gyu Yang & Youngin Goh & Kilho Lee & Daewon Ha & Suraj S. Cheema & Jae Kyeong Jeong & Daewoong Kwon, 2024. "Analog reservoir computing via ferroelectric mixed phase boundary transistors," Nature Communications, Nature, vol. 15(1), pages 1-14, December.
    13. Fadi Jebali & Atreya Majumdar & Clément Turck & Kamel-Eddine Harabi & Mathieu-Coumba Faye & Eloi Muhr & Jean-Pierre Walder & Oleksandr Bilousov & Amadéo Michaud & Elisa Vianello & Tifenn Hirtzlin & Fr, 2024. "Powering AI at the edge: A robust, memristor-based binarized neural network with near-memory computing and miniaturized solar cell," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    14. Han Xu & Dashan Shang & Qing Luo & Junjie An & Yue Li & Shuyu Wu & Zhihong Yao & Woyu Zhang & Xiaoxin Xu & Chunmeng Dou & Hao Jiang & Liyang Pan & Xumeng Zhang & Ming Wang & Zhongrui Wang & Jianshi Ta, 2023. "A low-power vertical dual-gate neurotransistor with short-term memory for high energy-efficient neuromorphic computing," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    15. Thomas Dalgaty & Filippo Moro & Yiğit Demirağ & Alessio Pra & Giacomo Indiveri & Elisa Vianello & Melika Payvand, 2024. "Mosaic: in-memory computing and routing for small-world spike-based neuromorphic systems," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    16. Yijun Li & Jianshi Tang & Bin Gao & Jian Yao & Anjunyi Fan & Bonan Yan & Yuchao Yang & Yue Xi & Yuankun Li & Jiaming Li & Wen Sun & Yiwei Du & Zhengwu Liu & Qingtian Zhang & Song Qiu & Qingwen Li & He, 2023. "Monolithic three-dimensional integration of RRAM-based hybrid memory architecture for one-shot learning," Nature Communications, Nature, vol. 14(1), pages 1-9, December.
    17. Boyuan Cui & Zhen Fan & Wenjie Li & Yihong Chen & Shuai Dong & Zhengwei Tan & Shengliang Cheng & Bobo Tian & Ruiqiang Tao & Guo Tian & Deyang Chen & Zhipeng Hou & Minghui Qin & Min Zeng & Xubing Lu & , 2022. "Ferroelectric photosensor network: an advanced hardware solution to real-time machine vision," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    18. Zhuiri Peng & Lei Tong & Wenhao Shi & Langlang Xu & Xinyu Huang & Zheng Li & Xiangxiang Yu & Xiaohan Meng & Xiao He & Shengjie Lv & Gaochen Yang & Hao Hao & Tian Jiang & Xiangshui Miao & Lei Ye, 2024. "Multifunctional human visual pathway-replicated hardware based on 2D materials," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    19. He-Shan Zhang & Xue-Mei Dong & Zi-Cheng Zhang & Ze-Pu Zhang & Chao-Yi Ban & Zhe Zhou & Cheng Song & Shi-Qi Yan & Qian Xin & Ju-Qing Liu & Yin-Xiang Li & Wei Huang, 2022. "Co-assembled perylene/graphene oxide photosensitive heterobilayer for efficient neuromorphics," Nature Communications, Nature, vol. 13(1), pages 1-9, December.
    20. Rui Wang & Tuo Shi & Xumeng Zhang & Jinsong Wei & Jian Lu & Jiaxue Zhu & Zuheng Wu & Qi Liu & Ming Liu, 2022. "Implementing in-situ self-organizing maps with memristor crossbar arrays for data mining and optimization," Nature Communications, Nature, vol. 13(1), pages 1-10, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:13:y:2022:i:1:d:10.1038_s41467-022-32790-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.