IDEAS home Printed from https://ideas.repec.org/a/spr/joinma/v34y2023i3d10.1007_s10845-021-01851-7.html
   My bibliography  Save this article

Simulation and deep reinforcement learning for adaptive dispatching in semiconductor manufacturing systems

Author

Listed:
  • Ahmed H. Sakr

    (École Polytechnique de Montréal
    Cairo University)

  • Ayman Aboelhassan

    (École Polytechnique de Montréal
    Cairo University)

  • Soumaya Yacout

    (École Polytechnique de Montréal)

  • Samuel Bassetto

    (École Polytechnique de Montréal)

Abstract

Fabrication areas in semiconductor industry are considered one of the most complex production systems. This complexity is caused by the high-mix of products and end-user market-based demands in that industry. Its dynamic and challenging processing requirements affect the handling capabilities of traditional production management paradigms. In this paper, we propose an application for dispatching and resources allocation through reinforcement learning. The application is based on a discrete-event simulation model for a case study of a real semiconductor manufacturing system. The model is built using both data-driven and agent-based approaches. The model simulates the various processing aspects that are present normally in these complex systems. The model’s agents are responsible for dispatching tasks and allocation of the different system’s resources. They employ Deep-Q-Network reinforcement learning. They learn simultaneously through the model execution. An independent Deep-Q-Network is trained for each agent. The model provides the training environment for the agents in which their decisions are applied and assessed for their adequacy. Our formulation of the environment’s state and the reward function for the learning algorithms creates cooperative decision-making policies for the agents. This results in improving the global performance of the whole system, and the performance of each agent’s resources. Our approach is compared to heuristics-based strategies that are applied in our case study. It achieved better production performance than the currently applied strategy.

Suggested Citation

  • Ahmed H. Sakr & Ayman Aboelhassan & Soumaya Yacout & Samuel Bassetto, 2023. "Simulation and deep reinforcement learning for adaptive dispatching in semiconductor manufacturing systems," Journal of Intelligent Manufacturing, Springer, vol. 34(3), pages 1311-1324, March.
  • Handle: RePEc:spr:joinma:v:34:y:2023:i:3:d:10.1007_s10845-021-01851-7
    DOI: 10.1007/s10845-021-01851-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10845-021-01851-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s10845-021-01851-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Rocchetta, R. & Bellani, L. & Compare, M. & Zio, E. & Patelli, E., 2019. "A reinforcement learning framework for optimal operation and maintenance of power grids," Applied Energy, Elsevier, vol. 241(C), pages 291-301.
    2. Reményi, Christoph & Staudacher, Stephan, 2014. "Systematic simulation based approach for the identification and implementation of a scheduling rule in the aircraft engine maintenance," International Journal of Production Economics, Elsevier, vol. 147(PA), pages 94-107.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nyong-Bassey, Bassey Etim & Giaouris, Damian & Patsios, Charalampos & Papadopoulou, Simira & Papadopoulos, Athanasios I. & Walker, Sara & Voutetakis, Spyros & Seferlis, Panos & Gadoue, Shady, 2020. "Reinforcement learning based adaptive power pinch analysis for energy management of stand-alone hybrid energy storage systems considering uncertainty," Energy, Elsevier, vol. 193(C).
    2. Ning Wang & Weisheng Xu & Weihui Shao & Zhiyu Xu, 2019. "A Q-Cube Framework of Reinforcement Learning Algorithm for Continuous Double Auction among Microgrids," Energies, MDPI, vol. 12(15), pages 1-26, July.
    3. Correa-Jullian, Camila & López Droguett, Enrique & Cardemil, José Miguel, 2020. "Operation scheduling in a solar thermal system: A reinforcement learning-based framework," Applied Energy, Elsevier, vol. 268(C).
    4. Shang, Yuwei & Wu, Wenchuan & Guo, Jianbo & Ma, Zhao & Sheng, Wanxing & Lv, Zhe & Fu, Chenran, 2020. "Stochastic dispatch of energy storage in microgrids: An augmented reinforcement learning approach," Applied Energy, Elsevier, vol. 261(C).
    5. Keerthana Sivamayil & Elakkiya Rajasekar & Belqasem Aljafari & Srete Nikolovski & Subramaniyaswamy Vairavasundaram & Indragandhi Vairavasundaram, 2023. "A Systematic Study on Reinforcement Learning Based Applications," Energies, MDPI, vol. 16(3), pages 1-23, February.
    6. van Ooijen, Henny & Bertrand, J. Will M. & Buyukkaramikli, Nasuh C., 2019. "Coordinating failed goods collecting and repair capacity policies in the maintenance of commoditized capital goods," International Journal of Production Economics, Elsevier, vol. 208(C), pages 29-42.
    7. Mancuso, A. & Compare, M. & Salo, A. & Zio, E., 2021. "Optimal Prognostics and Health Management-driven inspection and maintenance strategies for industrial systems," Reliability Engineering and System Safety, Elsevier, vol. 210(C).
    8. Stover, Oliver & Karve, Pranav & Mahadevan, Sankaran, 2023. "Reliability and risk metrics to assess operational adequacy and flexibility of power grids," Reliability Engineering and System Safety, Elsevier, vol. 231(C).
    9. Daeil Lee & Seoryong Koo & Inseok Jang & Jonghyun Kim, 2022. "Comparison of Deep Reinforcement Learning and PID Controllers for Automatic Cold Shutdown Operation," Energies, MDPI, vol. 15(8), pages 1-25, April.
    10. Veronica Fornlöf & Diego Galar & Anna Syberfeldt & Torgny Almgren, 2016. "RUL estimation and maintenance optimization for aircraft engines: a system of system approach," International Journal of System Assurance Engineering and Management, Springer;The Society for Reliability, Engineering Quality and Operations Management (SREQOM),India, and Division of Operation and Maintenance, Lulea University of Technology, Sweden, vol. 7(4), pages 450-461, December.
    11. Pinciroli, Luca & Baraldi, Piero & Zio, Enrico, 2023. "Maintenance optimization in industry 4.0," Reliability Engineering and System Safety, Elsevier, vol. 234(C).
    12. Guan, Xiaoshu & Xiang, Zhengliang & Bao, Yuequan & Li, Hui, 2022. "Structural dominant failure modes searching method based on deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 219(C).
    13. Compare, Michele & Antonello, Federico & Pinciroli, Luca & Zio, Enrico, 2022. "A general model for life-cycle cost analysis of Condition-Based Maintenance enabled by PHM capabilities," Reliability Engineering and System Safety, Elsevier, vol. 224(C).
    14. Zheng, Lingwei & Wu, Hao & Guo, Siqi & Sun, Xinyu, 2023. "Real-time dispatch of an integrated energy system based on multi-stage reinforcement learning with an improved action-choosing strategy," Energy, Elsevier, vol. 277(C).
    15. Theissler, Andreas & Pérez-Velázquez, Judith & Kettelgerdes, Marcel & Elger, Gordon, 2021. "Predictive maintenance enabled by machine learning: Use cases and challenges in the automotive industry," Reliability Engineering and System Safety, Elsevier, vol. 215(C).
    16. Hao, Zhaojun & Di Maio, Francesco & Zio, Enrico, 2023. "A sequential decision problem formulation and deep reinforcement learning solution of the optimization of O&M of cyber-physical energy systems (CPESs) for reliable and safe power production and supply," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    17. Andrade, Antonio Ramos & Stow, Julian, 2017. "Assessing the potential cost savings of introducing the maintenance option of ‘Economic Tyre Turning’ in Great Britain railway wheelsets," Reliability Engineering and System Safety, Elsevier, vol. 168(C), pages 317-325.
    18. Yi Kuang & Xiuli Wang & Hongyang Zhao & Yijun Huang & Xianlong Chen & Xifan Wang, 2020. "Agent-Based Energy Sharing Mechanism Using Deep Deterministic Policy Gradient Algorithm," Energies, MDPI, vol. 13(19), pages 1-20, September.
    19. Shi, Yan & Lu, Zhenzhou & Huang, Hongzhong & Liu, Yu & Li, Yanfeng & Zio, Enrico & Zhou, Yicheng, 2022. "A new preventive maintenance strategy optimization model considering lifecycle safety," Reliability Engineering and System Safety, Elsevier, vol. 221(C).
    20. Luis Hernández-Callejo, 2019. "A Comprehensive Review of Operation and Control, Maintenance and Lifespan Management, Grid Planning and Design, and Metering in Smart Grids," Energies, MDPI, vol. 12(9), pages 1-50, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:joinma:v:34:y:2023:i:3:d:10.1007_s10845-021-01851-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.