IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v11y2023i14p3067-d1191727.html
   My bibliography  Save this article

Effects of Exploration Weight and Overtuned Kernel Parameters on Gaussian Process-Based Bayesian Optimization Search Performance

Author

Listed:
  • Yuto Omae

    (College of Industrial Technology, Nihon University, 1-2-1, Izumi, Narashino, Chiba 275-8575, Japan)

Abstract

Gaussian process-based Bayesian optimization (GPBO) is used to search parameters in machine learning, material design, etc. It is a method for finding optimal solutions in a search space through the following four procedures. (1) Develop a Gaussian process regression (GPR) model using observed data. (2) The GPR model is used to obtain the estimated mean and estimated variance for the search space. (3) The point where the sum of the estimated mean and the weighted estimated variance (upper confidence bound, UCB) is largest is the next search point (in the case of a maximum search). (4) Repeat the above procedures. Thus, the generalization performance of the GPR is directly related to the search performance of the GPBO. In procedure (1), the kernel parameters (KPs) of the GPR are tuned via gradient descent (GD) using the log-likelihood as the objective function. However, if the number of iterations of the GD is too high, there is a risk that the KPs will overfit the observed data. In this case, because the estimated mean and variance output by the GPR model are inappropriate, the next search point cannot be properly determined. Therefore, overtuned KPs degrade the GPBO search performance. However, this negative effect can be mitigated by changing the parameters of the GPBO. We focus on the weight of the estimated variances (exploration weight) of the UCB as one of these parameters. In a GPBO with a large exploration weight, the observed data appear in various regions in the search space. If the KP is tuned using such data, the GPR model can estimate the diverse regions somewhat correctly, even if the KP overfits the observed data, i.e., the negative effect of overtuned KPs on the GPR is mitigated by setting a larger exploration weight for the UCB. This suggests that the negative effect of overtuned KPs on the GPBO search performance may be related to the UCB exploration weight. In the present study, this hypothesis was tested using simple numerical simulations. Specifically, GPBO was applied to a simple black-box function with two optimal solutions. As parameters of GPBO, we set the number of KP iterations of GD in the range of 0–500 and the exploration weight as { 1 , 5 } . The number of KP iterations expresses the degree of overtuning, and the exploration weight expresses the strength of the GPBO search. The results indicate that, in the overtuned KP situation, GPBO with a larger exploration weight has better search performance. This suggests that, when searching for solutions with a small GPBO exploration weight, one must be careful about overtuning KPs. The findings of this study are useful for successful exploration with GPBO in all situations where it is used, e.g., machine learning hyperparameter tuning.

Suggested Citation

  • Yuto Omae, 2023. "Effects of Exploration Weight and Overtuned Kernel Parameters on Gaussian Process-Based Bayesian Optimization Search Performance," Mathematics, MDPI, vol. 11(14), pages 1-13, July.
  • Handle: RePEc:gam:jmathe:v:11:y:2023:i:14:p:3067-:d:1191727
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/11/14/3067/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/2227-7390/11/14/3067/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Renhui Zhang & Xutao Zhao, 2020. "Inverse Method of Centrifugal Pump Blade Based on Gaussian Process Regression," Mathematical Problems in Engineering, Hindawi, vol. 2020, pages 1-10, February.
    2. Wei Xie & Wen Nie & Pooya Saffari & Luis F. Robledo & Pierre-Yves Descote & Wenbin Jian, 2021. "Landslide hazard assessment based on Bayesian optimization–support vector machine in Nanping City, China," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 109(1), pages 931-948, October.
    3. Blonigen, Bruce A. & Knittel, Christopher R. & Soderbery, Anson, 2017. "Keeping it fresh: Strategic product redesigns and welfare," International Journal of Industrial Organization, Elsevier, vol. 53(C), pages 170-214.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tobias Salz & Emanuel Vespa, 2020. "Estimating dynamic games of oligopolistic competition: an experimental investigation," RAND Journal of Economics, RAND Corporation, vol. 51(2), pages 447-469, June.
    2. Ruijie Zhang & Jiaqiong Wang & Wenfei Qian & Linlin Geng, 2024. "Optimization of Magnetic Pump Impeller Based on Blade Load Curve and Internal Flow Study," Mathematics, MDPI, vol. 12(4), pages 1-17, February.
    3. Friberg, Richard & Romahn, André, 2015. "Divestiture requirements as a tool for competition policy: A case from the Swedish beer market," International Journal of Industrial Organization, Elsevier, vol. 42(C), pages 1-18.
    4. An-Hsiang Liu & Ralph Siebert, 2020. "The Competitive Effects of Declining Entry Costs over Time: Evidence from the Static Random Access Memory Market," CESifo Working Paper Series 8552, CESifo.
    5. Mathias Reynaert, 2021. "Abatement Strategies and the Cost of Environmental Regulation: Emission Standards on the European Car Market," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 88(1), pages 454-488.
    6. Linn, Joshua, "undated". "Explaining the Adoption of Diesel Fuel Passenger Cars in Europe," RFF Working Paper Series dp-14-08-rev, Resources for the Future.
    7. Alex Burnap & John R. Hauser & Artem Timoshenko, 2019. "Product Aesthetic Design: A Machine Learning Augmentation," Papers 1907.07786, arXiv.org, revised Nov 2022.
    8. Kahn, Matthew E. & Walsh, Randall, 2015. "Cities and the Environment," Handbook of Regional and Urban Economics, in: Gilles Duranton & J. V. Henderson & William C. Strange (ed.), Handbook of Regional and Urban Economics, edition 1, volume 5, chapter 0, pages 405-465, Elsevier.
    9. Noton, Carlos, 2016. "Structural estimation of price adjustment costs in the European car market," International Journal of Industrial Organization, Elsevier, vol. 49(C), pages 105-147.
    10. Liu, An-Hsiang & Siebert, Ralph B., 2022. "The competitive effects of declining entry costs over time: Evidence from the static random access memory market," International Journal of Industrial Organization, Elsevier, vol. 80(C).
    11. Thomas Klier & Joshua Linn & Yichen C. Zhou, 2020. "The effects of fuel prices and vehicle sales on fuel‐saving technology adoption in passenger vehicles," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 29(3), pages 543-578, July.
    12. Linn, Joshua, 2023. "Emissions Standards and Electric Vehicle Targets for Passenger Vehicles," RFF Working Paper Series 23-05, Resources for the Future.
    13. Linn, Joshua, 2014. "Explaining the Adoption of Diesel Fuel Passenger Cars in Europe," Discussion Papers dp-14-08, Resources For the Future.
    14. Leard, Benjamin & McConnell, Virginia, 2015. "New Markets for Pollution and Energy Efficiency: Credit Trading under Automobile Greenhouse Gas and Fuel Economy StandardsAbstract: Recent changes to the Corporate Average Fuel Economy (CAFE) standard," RFF Working Paper Series dp-15-16, Resources for the Future.
    15. Chuhan Wang & Qigen Lin & Leibin Wang & Tong Jiang & Buda Su & Yanjun Wang & Sanjit Kumar Mondal & Jinlong Huang & Ying Wang, 2022. "The influences of the spatial extent selection for non-landslide samples on statistical-based landslide susceptibility modelling: a case study of Anhui Province in China," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 112(3), pages 1967-1988, July.
    16. Deborah Simon Mwakapesa & Yimin Mao & Xiaoji Lan & Yaser Ahangari Nanehkaran, 2023. "Landslide Susceptibility Mapping Using DIvisive ANAlysis (DIANA) and RObust Clustering Using linKs (ROCK) Algorithms, and Comparison of Their Performance," Sustainability, MDPI, vol. 15(5), pages 1-20, February.
    17. Hazem Ghassan Abdo & Hussein Almohamad & Ahmed Abdullah Al Dughairi & Motirh Al-Mutiry, 2022. "GIS-Based Frequency Ratio and Analytic Hierarchy Process for Forest Fire Susceptibility Mapping in the Western Region of Syria," Sustainability, MDPI, vol. 14(8), pages 1-20, April.
    18. Xiaojie Geng & Shunchuan Wu & Yanjie Zhang & Junlong Sun & Haiyong Cheng & Zhongxin Zhang & Shijiang Pu, 2023. "Developing hybrid XGBoost model integrated with entropy weight and Bayesian optimization for predicting tunnel squeezing intensity," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 119(1), pages 751-771, October.
    19. Siqi Liu & Bhoomija Ranjan & Benjamin Reed Shiller, 2020. "Are Coarse Ratings Fine? Applications to Crashworthiness Ratings," Working Papers 132, Brandeis University, Department of Economics and International Business School.
    20. Tingyu Zhang & Quan Fu & Chao Li & Fangfang Liu & Huanyuan Wang & Ling Han & Renata Pacheco Quevedo & Tianqing Chen & Na Lei, 2022. "Modeling landslide susceptibility using data mining techniques of kernel logistic regression, fuzzy unordered rule induction algorithm, SysFor and random forest," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 114(3), pages 3327-3358, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:11:y:2023:i:14:p:3067-:d:1191727. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.