NuSeT: A deep learning tool for reliably separating and analyzing crowded cells

Segmenting cell nuclei within microscopy images is a ubiquitous task in biological research and clinical applications. Unfortunately, segmenting low-contrast overlapping objects that may be tightly packed is a major bottleneck in standard deep learning-based models. We report a Nuclear Segmentation Tool (NuSeT) based on deep learning that accurately segments nuclei across multiple types of fluorescence imaging data. Using a hybrid network consisting of U-Net and Region Proposal Networks (RPN), followed by a watershed step, we have achieved superior performance in detecting and delineating nuclear boundaries in 2D and 3D images of varying complexities. By using foreground normalization and additional training on synthetic images containing non-cellular artifacts, NuSeT improves nuclear detection and reduces false positives. NuSeT addresses common challenges in nuclear segmentation such as variability in nuclear signal and shape, limited training sample size, and sample preparation artifacts. Compared to other segmentation models, NuSeT consistently fares better in generating accurate segmentation masks and assigning boundaries for touching nuclei.Author summary: Nuclear size and shape are essential indicators of cell cycle stage and cellular pathology. Efficient segmentation of nuclei in complex environments, especially for high-value yet low-quality samples is critical for detecting pathological states. In the majority of cases, biological features are still segmented using traditional segmentation methods requiring manual curation of segmentations, which is hugely time-consuming and does not achieve optimal performance. While a recent surge in deep learning tools has helped tremendously with the automation of segmentation tasks, existing platforms inefficiently segment nuclei in crowded cells with overlapping nuclear boundaries. NuSeT, assimilates the advantages of semantic segmentation (U-Net) and instance segmentation (Mask R-CNN), and consistently outperforms other start-of-the-art deep learning segmentation models in analyzing complex three-dimensional cell clusters and in tracking nuclei in crowded, dynamic environments. NuSeT can work with both fluorescent and histopathology image samples. We have also developed a graphic user interface for customized training and segmentation, that will aid considerably in the ease and accuracy of image segmentation in a wide range of image types.