IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v10y2022i10p1736-d819052.html
   My bibliography  Save this article

Improving the Performance of MapReduce for Small-Scale Cloud Processes Using a Dynamic Task Adjustment Mechanism

Author

Listed:
  • Tzu-Chi Huang

    (Department of Electronic Engineering, Lunghwa University of Science and Technology, Taoyuan 333, Taiwan)

  • Guo-Hao Huang

    (Department of Electronic Engineering, Lunghwa University of Science and Technology, Taoyuan 333, Taiwan)

  • Ming-Fong Tsai

    (Department of Electronic Engineering, National United University, Miaoli 360, Taiwan)

Abstract

The MapReduce architecture can reliably distribute massive datasets to cloud worker nodes for processing. When each worker node processes the input data, the Map program generates intermediate data that are used by the Reduce program for integration. However, as the worker nodes process the MapReduce tasks, there are differences in the number of intermediate data created, due to variation in the operating-system environments and the input data, which results in the phenomenon of laggard nodes and affects the completion time for each small-scale cloud application task. In this paper, we propose a dynamic task adjustment mechanism for an intermediate-data processing cycle prediction algorithm, with the aim of improving the execution performance of small-scale cloud applications. Our mechanism dynamically adjusts the number of Map and Reduce program tasks based on the intermediate-data processing capabilities of each cloud worker node, in order to mitigate the problem of performance degradation caused by the limitations on the Google Cloud platform (Hadoop cluster) due to the phenomenon of laggards. The proposed dynamic task adjustment mechanism was compared with a simulated Hadoop system in a performance analysis, and an improvement of at least 5% in the processing efficiency was found for a small-scale cloud application.

Suggested Citation

  • Tzu-Chi Huang & Guo-Hao Huang & Ming-Fong Tsai, 2022. "Improving the Performance of MapReduce for Small-Scale Cloud Processes Using a Dynamic Task Adjustment Mechanism," Mathematics, MDPI, vol. 10(10), pages 1-17, May.
    • Handle: RePEc:gam:jmathe:v:10:y:2022:i:10:p:1736-:d:819052
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/10/10/1736/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/10/10/1736/
    Download Restriction: no
    ---><---

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:10:y:2022:i:10:p:1736-:d:819052. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.