IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v10y2022i24p4788-d1005332.html
   My bibliography  Save this article

Adaptive Distributed Parallel Training Method for a Deep Learning Model Based on Dynamic Critical Paths of DAG

Author

Listed:
  • Yan Zeng

    (School of Computing Science, Hangzhou Danzi University, Hangzhou 310018, China
    Key Laboratory for Modeling and Simulation of Complex Systems, Ministry of Education, Hangzhou 310018, China
    Data Security Governance Zhejiang Engineering Research Center, Hangzhou 310018, China)

  • Wei Wang

    (School of Computing Science, Hangzhou Danzi University, Hangzhou 310018, China)

  • Yong Ding

    (School of Computing Science, Hangzhou Danzi University, Hangzhou 310018, China)

  • Jilin Zhang

    (School of Computing Science, Hangzhou Danzi University, Hangzhou 310018, China
    Key Laboratory for Modeling and Simulation of Complex Systems, Ministry of Education, Hangzhou 310018, China
    Data Security Governance Zhejiang Engineering Research Center, Hangzhou 310018, China)

  • Yongjian Ren

    (School of Computing Science, Hangzhou Danzi University, Hangzhou 310018, China
    Key Laboratory for Modeling and Simulation of Complex Systems, Ministry of Education, Hangzhou 310018, China
    Data Security Governance Zhejiang Engineering Research Center, Hangzhou 310018, China)

  • Guangzheng Yi

    (School of Computing Science, Hangzhou Danzi University, Hangzhou 310018, China)

Abstract

AI provides a new method for massive simulated data calculations in molecular dynamics, materials, and other scientific computing fields. However, the complex structures and large-scale parameters of neural network models make them difficult to develop and train. The automatic parallel technology based on graph algorithms is one of the most promising methods to solve this problem, despite the low efficiency in the design, implementation, and execution of distributed parallel policies for large-scale neural network models. In this paper, we propose an adaptive distributed parallel training method based on the dynamic generation of critical DAG (directed acyclic graph) paths, called FD-DPS, to solve this efficiency problem. Firstly, the proposed model splits operators with the dimension of the tensor, which can expand the space available for model parallelism. Secondly, a dynamic critical path generation method is employed to determine node priority changes in the DAG of the neural network models. Finally, the model implements the optimal scheduling of critical paths based on the priority of the nodes, thereby improving the performance of parallel strategies. Our experiments show that FD-DPS can achieve 12.76% and 11.78% faster training on PnasNet_mobile and ResNet_200 models, respectively, compared with the MP-DPS and Fast methods.

Suggested Citation

  • Yan Zeng & Wei Wang & Yong Ding & Jilin Zhang & Yongjian Ren & Guangzheng Yi, 2022. "Adaptive Distributed Parallel Training Method for a Deep Learning Model Based on Dynamic Critical Paths of DAG," Mathematics, MDPI, vol. 10(24), pages 1-21, December.
  • Handle: RePEc:gam:jmathe:v:10:y:2022:i:24:p:4788-:d:1005332
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/10/24/4788/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/2227-7390/10/24/4788/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Mohammed Abdullahi & Md Asri Ngadi, 2016. "Hybrid Symbiotic Organisms Search Optimization Algorithm for Scheduling of Tasks on Cloud Computing Environment," PLOS ONE, Public Library of Science, vol. 11(6), pages 1-29, June.
    2. John Jumper & Richard Evans & Alexander Pritzel & Tim Green & Michael Figurnov & Olaf Ronneberger & Kathryn Tunyasuvunakool & Russ Bates & Augustin Žídek & Anna Potapenko & Alex Bridgland & Clemens Me, 2021. "Highly accurate protein structure prediction with AlphaFold," Nature, Nature, vol. 596(7873), pages 583-589, August.
    3. Lukas Burger & Erik van Nimwegen, 2010. "Disentangling Direct from Indirect Co-Evolution of Residues in Protein Alignments," PLOS Computational Biology, Public Library of Science, vol. 6(1), pages 1-18, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ye Yuan & Lei Chen & Kexu Song & Miaomiao Cheng & Ling Fang & Lingfei Kong & Lanlan Yu & Ruonan Wang & Zhendong Fu & Minmin Sun & Qian Wang & Chengjun Cui & Haojue Wang & Jiuyang He & Xiaonan Wang & Y, 2024. "Stable peptide-assembled nanozyme mimicking dual antifungal actions," Nature Communications, Nature, vol. 15(1), pages 1-17, December.
    2. Ivica Odorčić & Mohamed Belal Hamed & Sam Lismont & Lucía Chávez-Gutiérrez & Rouslan G. Efremov, 2024. "Apo and Aβ46-bound γ-secretase structures provide insights into amyloid-β processing by the APH-1B isoform," Nature Communications, Nature, vol. 15(1), pages 1-14, December.
    3. Léon Faure & Bastien Mollet & Wolfram Liebermeister & Jean-Loup Faulon, 2023. "A neural-mechanistic hybrid approach improving the predictive power of genome-scale metabolic models," Nature Communications, Nature, vol. 14(1), pages 1-14, December.
    4. Tian Zhu & Merry H. Ma, 2022. "Deriving the Optimal Strategy for the Two Dice Pig Game via Reinforcement Learning," Stats, MDPI, vol. 5(3), pages 1-14, August.
    5. Stella Vitt & Simone Prinz & Martin Eisinger & Ulrich Ermler & Wolfgang Buckel, 2022. "Purification and structural characterization of the Na+-translocating ferredoxin: NAD+ reductase (Rnf) complex of Clostridium tetanomorphum," Nature Communications, Nature, vol. 13(1), pages 1-11, December.
    6. Pierre Azoulay & Joshua Krieger & Abhishek Nagaraj, 2024. "Old Moats for New Models: Openness, Control, and Competition in Generative AI," NBER Chapters, in: Entrepreneurship and Innovation Policy and the Economy, volume 4, National Bureau of Economic Research, Inc.
    7. Riya Shah & Thomas C. Panagiotou & Gregory B. Cole & Trevor F. Moraes & Brigitte D. Lavoie & Christopher A. McCulloch & Andrew Wilde, 2024. "The DIAPH3 linker specifies a β-actin network that maintains RhoA and Myosin-II at the cytokinetic furrow," Nature Communications, Nature, vol. 15(1), pages 1-17, December.
    8. Yashan Yang & Qianqian Shao & Mingcheng Guo & Lin Han & Xinyue Zhao & Aohan Wang & Xiangyun Li & Bo Wang & Ji-An Pan & Zhenguo Chen & Andrei Fokine & Lei Sun & Qianglin Fang, 2024. "Capsid structure of bacteriophage ΦKZ provides insights into assembly and stabilization of jumbo phages," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    9. Anthony C. Bishop & Glorisé Torres-Montalvo & Sravya Kotaru & Kyle Mimun & A. Joshua Wand, 2023. "Robust automated backbone triple resonance NMR assignments of proteins using Bayesian-based simulated annealing," Nature Communications, Nature, vol. 14(1), pages 1-15, December.
    10. Bret M. Boyd & Ian James & Kevin P. Johnson & Robert B. Weiss & Sarah E. Bush & Dale H. Clayton & Colin Dale, 2024. "Stochasticity, determinism, and contingency shape genome evolution of endosymbiotic bacteria," Nature Communications, Nature, vol. 15(1), pages 1-14, December.
    11. Jun-Yu Si & Yuan-Mei Chen & Ye-Hui Sun & Meng-Xue Gu & Mei-Ling Huang & Lu-Lu Shi & Xiao Yu & Xiao Yang & Qing Xiong & Cheng-Bao Ma & Peng Liu & Zheng-Li Shi & Huan Yan, 2024. "Sarbecovirus RBD indels and specific residues dictating multi-species ACE2 adaptiveness," Nature Communications, Nature, vol. 15(1), pages 1-15, December.
    12. Deyun Qiu & Jinxin V. Pei & James E. O. Rosling & Vandana Thathy & Dongdi Li & Yi Xue & John D. Tanner & Jocelyn Sietsma Penington & Yi Tong Vincent Aw & Jessica Yi Han Aw & Guoyue Xu & Abhai K. Tripa, 2022. "A G358S mutation in the Plasmodium falciparum Na+ pump PfATP4 confers clinically-relevant resistance to cipargamin," Nature Communications, Nature, vol. 13(1), pages 1-18, December.
    13. Shuo-Shuo Liu & Tian-Xia Jiang & Fan Bu & Ji-Lan Zhao & Guang-Fei Wang & Guo-Heng Yang & Jie-Yan Kong & Yun-Fan Qie & Pei Wen & Li-Bin Fan & Ning-Ning Li & Ning Gao & Xiao-Bo Qiu, 2024. "Molecular mechanisms underlying the BIRC6-mediated regulation of apoptosis and autophagy," Nature Communications, Nature, vol. 15(1), pages 1-16, December.
    14. Dick Schijven & Sourena Soheili-Nezhad & Simon E. Fisher & Clyde Francks, 2024. "Exome-wide analysis implicates rare protein-altering variants in human handedness," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    15. Ahrum Son & Hyunsoo Kim & Jolene K. Diedrich & Casimir Bamberger & Daniel B. McClatchy & Stuart A. Lipton & John R. Yates, 2024. "Using in vivo intact structure for system-wide quantitative analysis of changes in proteins," Nature Communications, Nature, vol. 15(1), pages 1-17, December.
    16. Justin N. Vaughn & Sandra E. Branham & Brian Abernathy & Amanda M. Hulse-Kemp & Adam R. Rivers & Amnon Levi & William P. Wechter, 2022. "Graph-based pangenomics maximizes genotyping density and reveals structural impacts on fungal resistance in melon," Nature Communications, Nature, vol. 13(1), pages 1-14, December.
    17. Eliza S. Nieweglowska & Axel F. Brilot & Melissa Méndez-Moran & Claire Kokontis & Minkyung Baek & Junrui Li & Yifan Cheng & David Baker & Joseph Bondy-Denomy & David A. Agard, 2023. "The ϕPA3 phage nucleus is enclosed by a self-assembling 2D crystalline lattice," Nature Communications, Nature, vol. 14(1), pages 1-12, December.
    18. Sash Lopaticki & Robyn McConville & Alan John & Niall Geoghegan & Shihab Deen Mohamed & Lisa Verzier & Ryan W. J. Steel & Cindy Evelyn & Matthew T. O’Neill & Niccolay Madiedo Soler & Nichollas E. Scot, 2022. "Tryptophan C-mannosylation is critical for Plasmodium falciparum transmission," Nature Communications, Nature, vol. 13(1), pages 1-18, December.
    19. Radoslaw Pluta & Eric Aragón & Nicholas A. Prescott & Lidia Ruiz & Rebeca A. Mees & Blazej Baginski & Julia R. Flood & Pau Martin-Malpartida & Joan Massagué & Yael David & Maria J. Macias, 2022. "Molecular basis for DNA recognition by the maternal pioneer transcription factor FoxH1," Nature Communications, Nature, vol. 13(1), pages 1-15, December.
    20. Xinheng He & Lifen Zhao & Yinping Tian & Rui Li & Qinyu Chu & Zhiyong Gu & Mingyue Zheng & Yusong Wang & Shaoning Li & Hualiang Jiang & Yi Jiang & Liuqing Wen & Dingyan Wang & Xi Cheng, 2024. "Highly accurate carbohydrate-binding site prediction with DeepGlycanSite," Nature Communications, Nature, vol. 15(1), pages 1-13, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:10:y:2022:i:24:p:4788-:d:1005332. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.