IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v13y2025i4p665-d1593909.html
   My bibliography  Save this article

Selective Reviews of Bandit Problems in AI via a Statistical View

Author

Listed:
  • Pengjie Zhou

    (Institute of Artificial Intelligence, Beihang University, Beijing 100191, China)

  • Haoyu Wei

    (Department of Economics, University of California San Diego, La Jolla, CA 92093, USA)

  • Huiming Zhang

    (Institute of Artificial Intelligence, Beihang University, Beijing 100191, China)

Abstract

Reinforcement Learning (RL) is a widely researched area in artificial intelligence that focuses on teaching agents decision-making through interactions with their environment. A key subset includes multi-armed bandit (MAB) and stochastic continuum-armed bandit (SCAB) problems, which model sequential decision-making under uncertainty. This review outlines the foundational models and assumptions of bandit problems, explores non-asymptotic theoretical tools like concentration inequalities and minimax regret bounds, and compares frequentist and Bayesian algorithms for managing exploration–exploitation trade-offs. Additionally, we explore K -armed contextual bandits and SCAB, focusing on their methodologies and regret analyses. We also examine the connections between SCAB problems and functional data analysis. Finally, we highlight recent advances and ongoing challenges in the field.

Suggested Citation

  • Pengjie Zhou & Haoyu Wei & Huiming Zhang, 2025. "Selective Reviews of Bandit Problems in AI via a Statistical View," Mathematics, MDPI, vol. 13(4), pages 1-54, February.
    • Handle: RePEc:gam:jmathe:v:13:y:2025:i:4:p:665-:d:1593909
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/13/4/665/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/13/4/665/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Xueru Zhang & Dennis K. J. Lin & Lin Wang, 2023. "Digital Triplet: A Sequential Methodology for Digital Twin Learning," Mathematics, MDPI, vol. 11(12), pages 1-16, June.
    2. Jianqing Fan & Yongyi Guo & Mengxin Yu, 2024. "Policy Optimization Using Semiparametric Models for Dynamic Pricing," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 119(545), pages 552-564, January.
    3. Lalit Jain & Zhaoqi Li & Erfan Loghmani & Blake Mason & Hema Yoganarasimhan, 2024. "Effective Adaptive Exploration of Prices and Promotions in Choice-Based Demand Models," Marketing Science, INFORMS, vol. 43(5), pages 1002-1030, September.
    4. Yi Cheng & Donald A. Berry, 2007. "Optimal adaptive randomized designs for clinical trials," Biometrika, Biometrika Trust, vol. 94(3), pages 673-689.
    5. Mengjia Yu & Xiaohui Chen, 2021. "Finite sample change point inference and identification for high‐dimensional mean vectors," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(2), pages 247-270, April.
    6. Yao Zheng & Guang Cheng, 2021. "Finite-time analysis of vector autoregressive models under linear restrictions [Nested reduced-rank autogressive models for multiple time series]," Biometrika, Biometrika Trust, vol. 108(2), pages 469-489.
    7. Ruodu Wang & Aaditya Ramdas, 2022. "False discovery rate control with e‐values," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(3), pages 822-852, July.
    8. Huiming Zhang & Haoyu Wei, 2022. "Sharper Sub-Weibull Concentrations," Mathematics, MDPI, vol. 10(13), pages 1-29, June.
    9. Yao, Fang & Muller, Hans-Georg & Wang, Jane-Ling, 2005. "Functional Data Analysis for Sparse Longitudinal Data," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 577-590, June.
    10. Hang Zhou & Fang Yao & Huiming Zhang, 2023. "Functional linear regression for discretely observed data: from ideal to reality," Biometrika, Biometrika Trust, vol. 110(2), pages 381-393.
    11. Hao Ji & Hans-Georg Müller, 2017. "Optimal designs for longitudinal and functional data," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 79(3), pages 859-876, June.
    12. Yi Liu & Veronika Ročková, 2023. "Variable Selection Via Thompson Sampling," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 118(541), pages 287-304, January.
    13. Sungwook Kim & Michael P. Fay & Michael A. Proschan, 2021. "Valid and approximately valid confidence intervals for current status data," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(3), pages 438-452, July.
    14. Wei Qian & Ching-Kang Ing & Ji Liu, 2024. "Adaptive Algorithm for Multi-Armed Bandit Problem with High-Dimensional Covariates," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 119(546), pages 970-982, April.
    15. Marco Battiston & Stefano Favaro & Yee Whye Teh, 2018. "Multi-Armed Bandit for Species Discovery: A Bayesian Nonparametric Approach," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(521), pages 455-466, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Pengjie Zhou & Haoyu Wei & Huiming Zhang, 2024. "Selective Reviews of Bandit Problems in AI via a Statistical View," Papers 2412.02251, arXiv.org, revised Feb 2025.
    2. Rha, Hyungmin & Kao, Ming-Hung & Pan, Rong, 2020. "Design optimal sampling plans for functional regression models," Computational Statistics & Data Analysis, Elsevier, vol. 146(C).
    3. Zhong, Rou & Liu, Shishi & Li, Haocheng & Zhang, Jingxiao, 2022. "Robust functional principal component analysis for non-Gaussian longitudinal data," Journal of Multivariate Analysis, Elsevier, vol. 189(C).
    4. Kao, Ming-Hung & Huang, Ping-Han, 2024. "Hybrid exact-approximate design approach for sparse functional data," Computational Statistics & Data Analysis, Elsevier, vol. 190(C).
    5. Park, So Young & Xiao, Luo & Willbur, Jayson D. & Staicu, Ana-Maria & Jumbe, N. L’ntshotsholé, 2018. "A joint design for functional data with application to scheduling ultrasound scans," Computational Statistics & Data Analysis, Elsevier, vol. 122(C), pages 101-114.
    6. Xu, Wenchao & Zhang, Xinyu & Liang, Hua, 2024. "Linearized maximum rank correlation estimation when covariates are functional," Journal of Multivariate Analysis, Elsevier, vol. 202(C).
    7. Wang, Jingxing & Chung, Seokhyun & AlShelahi, Abdullah & Kontar, Raed & Byon, Eunshin & Saigal, Romesh, 2021. "Look-ahead decision making for renewable energy: A dynamic “predict and store” approach," Applied Energy, Elsevier, vol. 296(C).
    8. Li, Pai-Ling & Chiou, Jeng-Min, 2011. "Identifying cluster number for subspace projected functional data clustering," Computational Statistics & Data Analysis, Elsevier, vol. 55(6), pages 2090-2103, June.
    9. Guangxing Wang & Sisheng Liu & Fang Han & Chong‐Zhi Di, 2023. "Robust functional principal component analysis via a functional pairwise spatial sign operator," Biometrics, The International Biometric Society, vol. 79(2), pages 1239-1253, June.
    10. Li, Pai-Ling & Chiou, Jeng-Min & Shyr, Yu, 2017. "Functional data classification using covariate-adjusted subspace projection," Computational Statistics & Data Analysis, Elsevier, vol. 115(C), pages 21-34.
    11. Poskitt, D.S. & Sengarapillai, Arivalzahan, 2013. "Description length and dimensionality reduction in functional data analysis," Computational Statistics & Data Analysis, Elsevier, vol. 58(C), pages 98-113.
    12. González-Rodríguez, Gil & Colubi, Ana, 2017. "On the consistency of bootstrap methods in separable Hilbert spaces," Econometrics and Statistics, Elsevier, vol. 1(C), pages 118-127.
    13. Shuxi Zeng & Elizabeth C. Lange & Elizabeth A. Archie & Fernando A. Campos & Susan C. Alberts & Fan Li, 2023. "A Causal Mediation Model for Longitudinal Mediators and Survival Outcomes with an Application to Animal Behavior," Journal of Agricultural, Biological and Environmental Statistics, Springer;The International Biometric Society;American Statistical Association, vol. 28(2), pages 197-218, June.
    14. J. Q. Shi & B. Wang & R. Murray-Smith & D. M. Titterington, 2007. "Gaussian Process Functional Regression Modeling for Batch Data," Biometrics, The International Biometric Society, vol. 63(3), pages 714-723, September.
    15. Weishampel, Anthony & Staicu, Ana-Maria & Rand, William, 2023. "Classification of social media users with generalized functional data analysis," Computational Statistics & Data Analysis, Elsevier, vol. 179(C).
    16. Nagy, Stanislav & Ferraty, Frédéric, 2019. "Data depth for measurable noisy random functions," Journal of Multivariate Analysis, Elsevier, vol. 170(C), pages 95-114.
    17. Chenlin Zhang & Huazhen Lin & Li Liu & Jin Liu & Yi Li, 2023. "Functional data analysis with covariate‐dependent mean and covariance structures," Biometrics, The International Biometric Society, vol. 79(3), pages 2232-2245, September.
    18. Tomáš Rubín & Victor M. Panaretos, 2020. "Functional lagged regression with sparse noisy observations," Journal of Time Series Analysis, Wiley Blackwell, vol. 41(6), pages 858-882, November.
    19. Beran, Jan & Liu, Haiyan, 2016. "Estimation of eigenvalues, eigenvectors and scores in FDA models with dependent errors," Journal of Multivariate Analysis, Elsevier, vol. 147(C), pages 218-233.
    20. Golovkine, Steven & Klutchnikoff, Nicolas & Patilea, Valentin, 2022. "Clustering multivariate functional data using unsupervised binary trees," Computational Statistics & Data Analysis, Elsevier, vol. 168(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:13:y:2025:i:4:p:665-:d:1593909. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.