A deep learning algorithm for 3D cell detection in whole mouse brain image datasets

Understanding the function of the nervous system necessitates mapping the spatial distributions of its constituent cells defined by function, anatomy or gene expression. Recently, developments in tissue preparation and microscopy allow cellular populations to be imaged throughout the entire rodent brain. However, mapping these neurons manually is prone to bias and is often impractically time consuming. Here we present an open-source algorithm for fully automated 3D detection of neuronal somata in mouse whole-brain microscopy images using standard desktop computer hardware. We demonstrate the applicability and power of our approach by mapping the brain-wide locations of large populations of cells labeled with cytoplasmic fluorescent proteins expressed via retrograde trans-synaptic viral infection.Author summary: Mapping cells in the brain is a key method in neuroscience, and was traditionally carried out on manually prepared thin sections. Today, modern microscopy approaches allow the entire mouse brain to be imaged in 3D at high resolution. Due to their often complex somatic morphology, detecting cytoplasmically labelled neurons in these large image datasets is highly challenging compared, for example, to detecting spherical cell nuclei. Additionally, a neuron can often be mistakenly detected multiple times, or two cells can be interpreted as a single cell. Here we have developed a freely available algorithm for detecting cytoplasmically labelled neuronal somata in these images which can be run faster than the data can be acquired, and without the bias of manual analysis. The ability to quickly map cellular distributions throughout the mouse brain will lead to a greater understanding of both its structure and function. As with flies, nematodes and fish, detecting and mapping cells in 3D throughout the entire mammalian brain will allow for new experiments designed to understand the structural basis of its myriad complex functions.