IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v13y2022i1d10.1038_s41467-022-32097-3.html
   My bibliography  Save this article

Scarf enables a highly memory-efficient analysis of large-scale single-cell genomics data

Author

Listed:
  • Parashar Dhapola

    (Lund University)

  • Johan Rodhe

    (Lund University)

  • Rasmus Olofzon

    (Lund University)

  • Thomas Bonald

    (Institut Polytechnique de Paris)

  • Eva Erlandsson

    (Lund University)

  • Shamit Soneji

    (Lund University)

  • Göran Karlsson

    (Lund University)

Abstract

As the scale of single-cell genomics experiments grows into the millions, the computational requirements to process this data are beyond the reach of many. Herein we present Scarf, a modularly designed Python package that seamlessly interoperates with other single-cell toolkits and allows for memory-efficient single-cell analysis of millions of cells on a laptop or low-cost devices like single-board computers. We demonstrate Scarf’s memory and compute-time efficiency by applying it to the largest existing single-cell RNA-Seq and ATAC-Seq datasets. Scarf wraps memory-efficient implementations of a graph-based t-stochastic neighbour embedding and hierarchical clustering algorithm. Moreover, Scarf performs accurate reference-anchored mapping of datasets while maintaining memory efficiency. By implementing a subsampling algorithm, Scarf additionally has the capacity to generate representative sampling of cells from a given dataset wherein rare cell populations and lineage differentiation trajectories are conserved. Together, Scarf provides a framework wherein any researcher can perform advanced processing, subsampling, reanalysis, and integration of atlas-scale datasets on standard laptop computers. Scarf is available on Github: https://github.com/parashardhapola/scarf .

Suggested Citation

  • Parashar Dhapola & Johan Rodhe & Rasmus Olofzon & Thomas Bonald & Eva Erlandsson & Shamit Soneji & Göran Karlsson, 2022. "Scarf enables a highly memory-efficient analysis of large-scale single-cell genomics data," Nature Communications, Nature, vol. 13(1), pages 1-14, December.
  • Handle: RePEc:nat:natcom:v:13:y:2022:i:1:d:10.1038_s41467-022-32097-3
    DOI: 10.1038/s41467-022-32097-3
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-022-32097-3
    File Function: Abstract
    Download Restriction: no

    File URL: https://libkey.io/10.1038/s41467-022-32097-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jason D. Buenrostro & Beijing Wu & Ulrike M. Litzenburger & Dave Ruff & Michael L. Gonzales & Michael P. Snyder & Howard Y. Chang & William J. Greenleaf, 2015. "Single-cell chromatin accessibility reveals principles of regulatory variation," Nature, Nature, vol. 523(7561), pages 486-490, July.
    2. Dmitry Kobak & Philipp Berens, 2019. "The art of using t-SNE for single-cell transcriptomics," Nature Communications, Nature, vol. 10(1), pages 1-14, December.
    3. Betsabeh Khoramian Tusi & Samuel L. Wolock & Caleb Weinreb & Yung Hwang & Daniel Hidalgo & Rapolas Zilionis & Ari Waisman & Jun R. Huh & Allon M. Klein & Merav Socolovsky, 2018. "Population snapshots predict early haematopoietic and erythroid hierarchies," Nature, Nature, vol. 555(7694), pages 54-60, March.
    4. Junyue Cao & Malte Spielmann & Xiaojie Qiu & Xingfan Huang & Daniel M. Ibrahim & Andrew J. Hill & Fan Zhang & Stefan Mundlos & Lena Christiansen & Frank J. Steemers & Cole Trapnell & Jay Shendure, 2019. "The single-cell transcriptional landscape of mammalian organogenesis," Nature, Nature, vol. 566(7745), pages 496-502, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kensuke Miyake & Junya Ito & Jun Nakabayashi & Shigeyuki Shichino & Kenji Ishiwata & Hajime Karasuyama, 2023. "Single cell transcriptomics clarifies the basophil differentiation trajectory and identifies pre-basophils upstream of mature basophils," Nature Communications, Nature, vol. 14(1), pages 1-15, December.
    2. Takayoshi Matsumura & Haruhito Totani & Yoshitaka Gunji & Masahiro Fukuda & Rui Yokomori & Jianwen Deng & Malini Rethnam & Chong Yang & Tze King Tan & Tadayoshi Karasawa & Kazuomi Kario & Masafumi Tak, 2022. "A Myb enhancer-guided analysis of basophil and mast cell differentiation," Nature Communications, Nature, vol. 13(1), pages 1-17, December.
    3. Sivakamasundari Vijayakumar & Roberta Sala & Gugene Kang & Angela Chen & Michelle Ann Pablo & Abidemi Ismail Adebayo & Andrea Cipriano & Jonas L. Fowler & Danielle L. Gomes & Lay Teng Ang & Kyle M. Lo, 2023. "Monolayer platform to generate and purify primordial germ-like cells in vitro provides insights into human germline specification," Nature Communications, Nature, vol. 14(1), pages 1-19, December.
    4. Brian DeVeale & Leqian Liu & Ryan Boileau & Jennifer Swindlehurst-Chan & Bryan Marsh & Jacob W. Freimer & Adam Abate & Robert Blelloch, 2022. "G1/S restriction point coordinates phasic gene expression and cell differentiation," Nature Communications, Nature, vol. 13(1), pages 1-17, December.
    5. Yasuaki Uehara & Yusuke Tanaka & Shuyang Zhao & Nikolaos M. Nikolaidis & Lori B. Pitstick & Huixing Wu & Jane J. Yu & Erik Zhang & Yoshihiro Hasegawa & John G. Noel & Jason C. Gardner & Elizabeth J. K, 2023. "Insights into pulmonary phosphate homeostasis and osteoclastogenesis emerge from the study of pulmonary alveolar microlithiasis," Nature Communications, Nature, vol. 14(1), pages 1-17, December.
    6. J. McClatchy & R. Strogantsev & E. Wolfe & H. Y. Lin & M. Mohammadhosseini & B. A. Davis & C. Eden & D. Goldman & W. H. Fleming & P. Conley & G. Wu & L. Cimmino & H. Mohammed & A. Agarwal, 2023. "Clonal hematopoiesis related TET2 loss-of-function impedes IL1β-mediated epigenetic reprogramming in hematopoietic stem and progenitor cells," Nature Communications, Nature, vol. 14(1), pages 1-17, December.
    7. Ci Fu & Xiang Zhang & Amanda O. Veri & Kali R. Iyer & Emma Lash & Alice Xue & Huijuan Yan & Nicole M. Revie & Cassandra Wong & Zhen-Yuan Lin & Elizabeth J. Polvi & Sean D. Liston & Benjamin VanderSlui, 2021. "Leveraging machine learning essentiality predictions and chemogenomic interactions to identify antifungal targets," Nature Communications, Nature, vol. 12(1), pages 1-18, December.
    8. Junyi Chen & Xiaoying Wang & Anjun Ma & Qi-En Wang & Bingqiang Liu & Lang Li & Dong Xu & Qin Ma, 2022. "Deep transfer learning of cancer drug responses by integrating bulk and single-cell RNA-seq data," Nature Communications, Nature, vol. 13(1), pages 1-13, December.
    9. Kotaro Shimizu & Junichi Kikuta & Yumi Ohta & Yutaka Uchida & Yu Miyamoto & Akito Morimoto & Shinya Yari & Takashi Sato & Takefumi Kamakura & Kazuo Oshima & Ryusuke Imai & Yu-Chen Liu & Daisuke Okuzak, 2023. "Single-cell transcriptomics of human cholesteatoma identifies an activin A-producing osteoclastogenic fibroblast subset inducing bone destruction," Nature Communications, Nature, vol. 14(1), pages 1-12, December.
    10. Young Hee Lee & Yu-Been Kim & Kyu Sik Kim & Mirae Jang & Ha Young Song & Sang-Ho Jung & Dong-Soo Ha & Joon Seok Park & Jaegeon Lee & Kyung Min Kim & Deok-Hyeon Cheon & Inhyeok Baek & Min-Gi Shin & Eun, 2023. "Lateral hypothalamic leptin receptor neurons drive hunger-gated food-seeking and consummatory behaviours in male mice," Nature Communications, Nature, vol. 14(1), pages 1-16, December.
    11. Sandra Curras-Alonso & Juliette Soulier & Thomas Defard & Christian Weber & Sophie Heinrich & Hugo Laporte & Sophie Leboucher & Sonia Lameiras & Marie Dutreix & Vincent Favaudon & Florian Massip & Tho, 2023. "An interactive murine single-cell atlas of the lung responses to radiation injury," Nature Communications, Nature, vol. 14(1), pages 1-16, December.
    12. Seung-Hyun Jung & Byung-Hee Hwang & Sun Shin & Eun-Hye Park & Sin-Hee Park & Chan Woo Kim & Eunmin Kim & Eunho Choo & Ik Jun Choi & Filip K. Swirski & Kiyuk Chang & Yeun-Jun Chung, 2022. "Spatiotemporal dynamics of macrophage heterogeneity and a potential function of Trem2hi macrophages in infarcted hearts," Nature Communications, Nature, vol. 13(1), pages 1-15, December.
    13. Hailun Zhu & Sihai Dave Zhao & Alokananda Ray & Yu Zhang & Xin Li, 2022. "A comprehensive temporal patterning gene network in Drosophila medulla neuroblasts revealed by single-cell RNA sequencing," Nature Communications, Nature, vol. 13(1), pages 1-19, December.
    14. Luke Simpson & Andrew Strange & Doris Klisch & Sophie Kraunsoe & Takuya Azami & Daniel Goszczynski & Triet Minh & Benjamin Planells & Nadine Holmes & Fei Sang & Sonal Henson & Matthew Loose & Jennifer, 2024. "A single-cell atlas of pig gastrulation as a resource for comparative embryology," Nature Communications, Nature, vol. 15(1), pages 1-16, December.
    15. Hannah Drew Rickner & Lulu Jiang & Rui Hong & Nicholas K. O’Neill & Chromewell A. Mojica & Benjamin J. Snyder & Lushuang Zhang & Dipan Shaw & Maria Medalla & Benjamin Wolozin & Christine S. Cheng, 2022. "Single cell transcriptomic profiling of a neuron-astrocyte assembloid tauopathy model," Nature Communications, Nature, vol. 13(1), pages 1-22, December.
    16. Jingtao Wang & Gregory J. Fonseca & Jun Ding, 2024. "scSemiProfiler: Advancing large-scale single-cell studies through semi-profiling with deep generative models and active learning," Nature Communications, Nature, vol. 15(1), pages 1-27, December.
    17. Guocheng Fang & Zhen Qiao & Luqi Huang & Hui Zhu & Jun Xie & Tian Zhou & Zhongshu Xiong & I-Hsin Su & Dayong Jin & Yu-Cheng Chen, 2024. "Single-cell laser emitting cytometry for label-free nucleolus fingerprinting," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    18. Songming Tang & Xuejian Cui & Rongxiang Wang & Sijie Li & Siyu Li & Xin Huang & Shengquan Chen, 2024. "scCASE: accurate and interpretable enhancement for single-cell chromatin accessibility sequencing data," Nature Communications, Nature, vol. 15(1), pages 1-16, December.
    19. Rong Ma & Eric D. Sun & James Zou, 2023. "A spectral method for assessing and combining multiple data visualizations," Nature Communications, Nature, vol. 14(1), pages 1-14, December.
    20. Aaron T L Lun & Hervé Pagès & Mike L Smith, 2018. "beachmat: A Bioconductor C++ API for accessing high-throughput biological data from a variety of R matrix types," PLOS Computational Biology, Public Library of Science, vol. 14(5), pages 1-15, May.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:13:y:2022:i:1:d:10.1038_s41467-022-32097-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.