IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v14y2023i1d10.1038_s41467-023-39860-0.html
   My bibliography  Save this article

Large depth-of-field ultra-compact microscope by progressive optimization and deep learning

Author

Listed:
  • Yuanlong Zhang

    (Tsinghua University
    Tsinghua University
    Tsinghua University
    Beijing Municipal Education Commission)

  • Xiaofei Song

    (Tsinghua University)

  • Jiachen Xie

    (Tsinghua University
    Tsinghua University
    Tsinghua University
    Beijing Municipal Education Commission)

  • Jing Hu

    (Zhejiang University)

  • Jiawei Chen

    (OPPO Research Institute)

  • Xiang Li

    (OPPO Research Institute)

  • Haiyu Zhang

    (OPPO Research Institute)

  • Qiqun Zhou

    (OPPO Research Institute)

  • Lekang Yuan

    (Tsinghua University)

  • Chui Kong

    (Fudan University)

  • Yibing Shen

    (Zhejiang University)

  • Jiamin Wu

    (Tsinghua University
    Tsinghua University
    Tsinghua University
    Beijing Municipal Education Commission)

  • Lu Fang

    (Tsinghua University)

  • Qionghai Dai

    (Tsinghua University
    Tsinghua University
    Tsinghua University
    Beijing Municipal Education Commission)

Abstract

The optical microscope is customarily an instrument of substantial size and expense but limited performance. Here we report an integrated microscope that achieves optical performance beyond a commercial microscope with a 5×, NA 0.1 objective but only at 0.15 cm3 and 0.5 g, whose size is five orders of magnitude smaller than that of a conventional microscope. To achieve this, a progressive optimization pipeline is proposed which systematically optimizes both aspherical lenses and diffractive optical elements with over 30 times memory reduction compared to the end-to-end optimization. By designing a simulation-supervision deep neural network for spatially varying deconvolution during optical design, we accomplish over 10 times improvement in the depth-of-field compared to traditional microscopes with great generalization in a wide variety of samples. To show the unique advantages, the integrated microscope is equipped in a cell phone without any accessories for the application of portable diagnostics. We believe our method provides a new framework for the design of miniaturized high-performance imaging systems by integrating aspherical optics, computational optics, and deep learning.

Suggested Citation

  • Yuanlong Zhang & Xiaofei Song & Jiachen Xie & Jing Hu & Jiawei Chen & Xiang Li & Haiyu Zhang & Qiqun Zhou & Lekang Yuan & Chui Kong & Yibing Shen & Jiamin Wu & Lu Fang & Qionghai Dai, 2023. "Large depth-of-field ultra-compact microscope by progressive optimization and deep learning," Nature Communications, Nature, vol. 14(1), pages 1-15, December.
  • Handle: RePEc:nat:natcom:v:14:y:2023:i:1:d:10.1038_s41467-023-39860-0
    DOI: 10.1038/s41467-023-39860-0
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-023-39860-0
    File Function: Abstract
    Download Restriction: no

    File URL: https://libkey.io/10.1038/s41467-023-39860-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Ethan Tseng & Shane Colburn & James Whitehead & Luocheng Huang & Seung-Hwan Baek & Arka Majumdar & Felix Heide, 2021. "Neural nano-optics for high-quality thin lens imaging," Nature Communications, Nature, vol. 12(1), pages 1-7, December.
    2. Denise J. Cai & Daniel Aharoni & Tristan Shuman & Justin Shobe & Jeremy Biane & Weilin Song & Brandon Wei & Michael Veshkini & Mimi La-Vu & Jerry Lou & Sergio E. Flores & Isaac Kim & Yoshitake Sano & , 2016. "A shared neural ensemble links distinct contextual memories encoded close in time," Nature, Nature, vol. 534(7605), pages 115-118, June.
    3. Jiamin Wu & Yuduo Guo & Chao Deng & Anke Zhang & Hui Qiao & Zhi Lu & Jiachen Xie & Lu Fang & Qionghai Dai, 2022. "An integrated imaging sensor for aberration-corrected 3D photography," Nature, Nature, vol. 612(7938), pages 62-71, December.
    4. Tsai-Wen Chen & Trevor J. Wardill & Yi Sun & Stefan R. Pulver & Sabine L. Renninger & Amy Baohan & Eric R. Schreiter & Rex A. Kerr & Michael B. Orger & Vivek Jayaraman & Loren L. Looger & Karel Svobod, 2013. "Ultrasensitive fluorescent proteins for imaging neuronal activity," Nature, Nature, vol. 499(7458), pages 295-300, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Xinge Yang & Qiang Fu & Wolfgang Heidrich, 2024. "Curriculum learning for ab initio deep learned refractive optics," Nature Communications, Nature, vol. 15(1), pages 1-8, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Johannes Friedrich & Pengcheng Zhou & Liam Paninski, 2017. "Fast online deconvolution of calcium imaging data," PLOS Computational Biology, Public Library of Science, vol. 13(3), pages 1-26, March.
    2. Joana Mendes Duarte & Robin Nguyen & Marios Kyprou & Kaizhen Li & Anastasija Milentijevic & Carlo Cerquetella & Thomas Forro & Stéphane Ciocchi, 2024. "Hippocampal contextualization of social rewards in mice," Nature Communications, Nature, vol. 15(1), pages 1-15, December.
    3. Adrienne I. Kinman & Derek N. Merryweather & Sarah R. Erwin & Regan E. Campbell & Kaitlin E. Sullivan & Larissa Kraus & Margarita Kapustina & Brianna N. Bristow & Mingjia Y. Zhang & Madeline W. Elder , 2025. "Atypical hippocampal excitatory neurons express and govern object memory," Nature Communications, Nature, vol. 16(1), pages 1-19, December.
    4. Omer Mano & Damon A Clark, 2017. "Graphics Processing Unit-Accelerated Code for Computing Second-Order Wiener Kernels and Spike-Triggered Covariance," PLOS ONE, Public Library of Science, vol. 12(1), pages 1-11, January.
    5. Zhaoyi Li & Raphaël Pestourie & Joon-Suh Park & Yao-Wei Huang & Steven G. Johnson & Federico Capasso, 2022. "Inverse design enables large-scale high-performance meta-optics reshaping virtual reality," Nature Communications, Nature, vol. 13(1), pages 1-11, December.
    6. Caio Vaz Rimoli & Claudio Moretti & Fernando Soldevila & Enora Brémont & Cathie Ventalon & Sylvain Gigan, 2024. "Demixing fluorescence time traces transmitted by multimode fibers," Nature Communications, Nature, vol. 15(1), pages 1-14, December.
    7. Zengpeng Han & Nengsong Luo & Wenyu Ma & Xiaodong Liu & Yuxiang Cai & Jiaxin Kou & Jie Wang & Lei Li & Siqi Peng & Zihong Xu & Wen Zhang & Yuxiang Qiu & Yang Wu & Chaohui Ye & Kunzhang Lin & Fuqiang X, 2023. "AAV11 enables efficient retrograde targeting of projection neurons and enhances astrocyte-directed transduction," Nature Communications, Nature, vol. 14(1), pages 1-13, December.
    8. Qingbin Fan & Weizhu Xu & Xuemei Hu & Wenqi Zhu & Tao Yue & Cheng Zhang & Feng Yan & Lu Chen & Henri J. Lezec & Yanqing Lu & Amit Agrawal & Ting Xu, 2022. "Trilobite-inspired neural nanophotonic light-field camera with extreme depth-of-field," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    9. Philipp Berens & Jeremy Freeman & Thomas Deneux & Nikolay Chenkov & Thomas McColgan & Artur Speiser & Jakob H Macke & Srinivas C Turaga & Patrick Mineault & Peter Rupprecht & Stephan Gerhard & Rainer , 2018. "Community-based benchmarking improves spike rate inference from two-photon calcium imaging data," PLOS Computational Biology, Public Library of Science, vol. 14(5), pages 1-13, May.
    10. Kyuhyun Choi & Eugenio Piasini & Edgar Díaz-Hernández & Luigim Vargas Cifuentes & Nathan T. Henderson & Elizabeth N. Holly & Manivannan Subramaniyan & Charles R. Gerfen & Marc V. Fuccillo, 2023. "Distributed processing for value-based choice by prelimbic circuits targeting anterior-posterior dorsal striatal subregions in male mice," Nature Communications, Nature, vol. 14(1), pages 1-19, December.
    11. Jilt Sebastian & Mriganka Sur & Hema A Murthy & Mathew Magimai-Doss, 2021. "Signal-to-signal neural networks for improved spike estimation from calcium imaging data," PLOS Computational Biology, Public Library of Science, vol. 17(3), pages 1-19, March.
    12. Yue-Qing Zhou & Vyash Puliyadi & Xiaojing Chen & Joonhee Leo Lee & Lan-Yuan Zhang & James J. Knierim, 2024. "Vector coding and place coding in hippocampus share a common directional signal," Nature Communications, Nature, vol. 15(1), pages 1-15, December.
    13. Xinge Yang & Qiang Fu & Wolfgang Heidrich, 2024. "Curriculum learning for ab initio deep learned refractive optics," Nature Communications, Nature, vol. 15(1), pages 1-8, December.
    14. Jina Yun & Simon Hansen & Otto Morris & David T. Madden & Clare Peters Libeu & Arjun J. Kumar & Cameron Wehrfritz & Aaron H. Nile & Yingnan Zhang & Lijuan Zhou & Yuxin Liang & Zora Modrusan & Michelle, 2023. "Senescent cells perturb intestinal stem cell differentiation through Ptk7 induced noncanonical Wnt and YAP signaling," Nature Communications, Nature, vol. 14(1), pages 1-19, December.
    15. Teppei Matsui & Takayuki Hashimoto & Tomonari Murakami & Masato Uemura & Kohei Kikuta & Toshiki Kato & Kenichi Ohki, 2024. "Orthogonalization of spontaneous and stimulus-driven activity by hierarchical neocortical areal network in primates," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    16. Franziska Bierbuesse & Anaïs C. Bourges & Vincent Gielen & Viola Mönkemöller & Wim Vandenberg & Yi Shen & Johan Hofkens & Pieter Vanden Berghe & Robert E. Campbell & Benjamien Moeyaert & Peter Dedecke, 2022. "Absolute measurement of cellular activities using photochromic single-fluorophore biosensors and intermittent quantification," Nature Communications, Nature, vol. 13(1), pages 1-13, December.
    17. Gang Wu & Mohamed Abid & Mohamed Zerara & Jiung Cho & Miri Choi & Cormac Ó Coileáin & Kuan-Ming Hung & Ching-Ray Chang & Igor V. Shvets & Han-Chun Wu, 2024. "Miniaturized spectrometer with intrinsic long-term image memory," Nature Communications, Nature, vol. 15(1), pages 1-11, December.
    18. Jimin Wu & Yuzhi Chen & Ashok Veeraraghavan & Eyal Seidemann & Jacob T. Robinson, 2024. "Mesoscopic calcium imaging in a head-unrestrained male non-human primate using a lensless microscope," Nature Communications, Nature, vol. 15(1), pages 1-14, December.
    19. Ella Gabitov & Arnaud Boutin & Basile Pinsard & Nitzan Censor & Stuart M Fogel & Geneviève Albouy & Bradley R King & Julie Carrier & Leonardo G Cohen & Avi Karni & Julien Doyon, 2019. "Susceptibility of consolidated procedural memory to interference is independent of its active task-based retrieval," PLOS ONE, Public Library of Science, vol. 14(1), pages 1-19, January.
    20. Kaizhen Li & Konstantinos Koukoutselos & Masanori Sakaguchi & Stéphane Ciocchi, 2024. "Distinct ventral hippocampal inhibitory microcircuits regulating anxiety and fear behaviors," Nature Communications, Nature, vol. 15(1), pages 1-13, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:14:y:2023:i:1:d:10.1038_s41467-023-39860-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.