IDEAS home Printed from https://ideas.repec.org/a/bla/jorssc/v71y2022i4p987-1013.html
   My bibliography  Save this article

Reliability analysis of artificial intelligence systems using recurrent events data from autonomous vehicles

Author

Listed:
  • Jie Min
  • Yili Hong
  • Caleb B. King
  • William Q. Meeker

Abstract

Artificial intelligence (AI) systems have become increasingly common and the trend will continue. Examples of AI systems include autonomous vehicles (AV), computer vision, natural language processing and AI medical experts. To allow for safe and effective deployment of AI systems, the reliability of such systems needs to be assessed. Traditionally, reliability assessment is based on reliability test data and the subsequent statistical modelling and analysis. The availability of reliability data for AI systems, however, is limited because such data are typically sensitive and proprietary. The California Department of Motor Vehicles (DMV) oversees and regulates an AV testing program, in which many AV manufacturers are conducting AV road tests. Manufacturers participating in the program are required to report recurrent disengagement events to California DMV. This information is being made available to the public. In this paper, we use recurrent disengagement events as a representation of the reliability of the AI system in AV, and propose a statistical framework for modelling and analysing the recurrent events data from AV driving tests. We use traditional parametric models in software reliability and propose a new non‐parametric model based on monotonic splines to describe the event process and to estimate the cumulative baseline intensity function of the event process. We develop inference procedures for selecting the best models, quantifying uncertainty and testing heterogeneity in the event process. We then analyse the recurrent events data from four AV manufacturers, and make inferences on the reliability of the AI systems in AV. We also describe how the proposed analysis can be applied to assess the reliability of other AI systems. This paper has online supplementary materials.

Suggested Citation

  • Jie Min & Yili Hong & Caleb B. King & William Q. Meeker, 2022. "Reliability analysis of artificial intelligence systems using recurrent events data from autonomous vehicles," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 71(4), pages 987-1013, August.
    • Handle: RePEc:bla:jorssc:v:71:y:2022:i:4:p:987-1013
    DOI: 10.1111/rssc.12564
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/rssc.12564
    Download Restriction: no
    File URL: https://libkey.io/10.1111/rssc.12564?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Kevin Burke & M. C. Jones & Angela Noufaily, 2020. "A flexible parametric modelling framework for survival analysis," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 69(2), pages 429-457, April.
    2. Vinayak V Dixit & Sai Chand & Divya J Nair, 2016. "Autonomous Vehicles: Disengagements, Accidents and Reaction Times," PLOS ONE, Public Library of Science, vol. 11(12), pages 1-14, December.
    3. Kalra, Nidhi & Paddock, Susan M., 2016. "Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?," Transportation Research Part A: Policy and Practice, Elsevier, vol. 94(C), pages 182-193.
    4. Simon N. Wood & Natalya Pya & Benjamin Säfken, 2016. "Smoothing Parameter and Model Selection for General Smooth Models," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 111(516), pages 1548-1563, October.
    5. Hong, Yili & Escobar, Luis A. & Meeker, William Q., 2010. "Coverage probabilities of simultaneous confidence bands and regions for log-location-scale distributions," Statistics & Probability Letters, Elsevier, vol. 80(7-8), pages 733-738, April.
    6. J. F. Lawless, 1995. "The Analysis of Recurrent Events for Multiple Subjects," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 44(4), pages 487-498, December.
    7. Li Xu & Chris Gotwalt & Yili Hong & Caleb B. King & William Q. Meeker, 2020. "Applications of the Fractional-Random-Weight Bootstrap," The American Statistician, Taylor & Francis Journals, vol. 74(4), pages 345-358, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kassens-Noor, Eva & Dake, Dana & Decaminada, Travis & Kotval-K, Zeenat & Qu, Teresa & Wilson, Mark & Pentland, Brian, 2020. "Sociomobility of the 21st century: Autonomous vehicles, planning, and the future city," Transport Policy, Elsevier, vol. 99(C), pages 329-335.
    2. Mohamed Abdel-Aty & Shengxuan Ding, 2024. "A matched case-control analysis of autonomous vs human-driven vehicle accidents," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    3. Cian Ryan & Finbarr Murphy & Martin Mullins, 2019. "Semiautonomous Vehicle Risk Analysis: A Telematics‐Based Anomaly Detection Approach," Risk Analysis, John Wiley & Sons, vol. 39(5), pages 1125-1140, May.
    4. Ryan, Cian & Murphy, Finbarr & Mullins, Martin, 2020. "Spatial risk modelling of behavioural hotspots: Risk-aware path planning for autonomous vehicles," Transportation Research Part A: Policy and Practice, Elsevier, vol. 134(C), pages 152-163.
    5. Georgios Gioldasis & Antonio Musolesi & Michel Simioni, 2020. "Model uncertainty, nonlinearities and out-of-sample comparison: evidence from international technology diffusion," Working Papers hal-02790523, HAL.
    6. Amara-Ouali, Yvenn & Fasiolo, Matteo & Goude, Yannig & Yan, Hui, 2023. "Daily peak electrical load forecasting with a multi-resolution approach," International Journal of Forecasting, Elsevier, vol. 39(3), pages 1272-1286.
    7. Hazel Si Min Lim & Araz Taeihagh, 2018. "Autonomous Vehicles for Smart and Sustainable Cities: An In-Depth Exploration of Privacy and Cybersecurity Implications," Energies, MDPI, vol. 11(5), pages 1-23, April.
    8. Andrea Bertolini & Massimo Riccaboni, 2021. "Grounding the case for a European approach to the regulation of automated driving: the technology-selection effect of liability rules," European Journal of Law and Economics, Springer, vol. 51(2), pages 243-284, April.
    9. Chenyin Gao & Katherine Jenny Thompson & Jae Kwang Kim & Shu Yang, 2022. "Nearest neighbour ratio imputation with incomplete multinomial outcome in survey sampling," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 185(4), pages 1903-1930, October.
    10. Areej Ahmad Alshaafee & Noorminshah A. Iahad & Mohammed A. Al-Sharafi, 2021. "Benefits or Risks: What Influences Novice Drivers Regarding Adopting Smart Cars?," Sustainability, MDPI, vol. 13(21), pages 1-20, October.
    11. Simon N. Wood, 2020. "Inference and computation with generalized additive models and their extensions," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 29(2), pages 307-339, June.
    12. Georgios Gioldasis & Antonio Musolesi & Michel Simioni, 2021. "Interactive R&D Spillovers: an estimation strategy based on forecasting-driven model selection," Working Papers hal-03224910, HAL.
    13. Adam Braima S. Mastor & Abdulaziz S. Alghamdi & Oscar Ngesa & Joseph Mung’atu & Christophe Chesneau & Ahmed Z. Afify, 2023. "The Extended Exponential-Weibull Accelerated Failure Time Model with Application to Sudan COVID-19 Data," Mathematics, MDPI, vol. 11(2), pages 1-26, January.
    14. Paul Ghelasi & Florian Ziel, 2024. "From day-ahead to mid and long-term horizons with econometric electricity price forecasting models," Papers 2406.00326, arXiv.org, revised Aug 2024.
    15. Valtiala, Juho & Niskanen, Olli & Torvinen, Mikael & Riekkinen, Kirsikka & Suokannas, Antti, 2023. "The relationship between agricultural land parcel size and cultivation costs," Land Use Policy, Elsevier, vol. 131(C).
    16. Khastgir, Siddartha & Brewerton, Simon & Thomas, John & Jennings, Paul, 2021. "Systems Approach to Creating Test Scenarios for Automated Driving Systems," Reliability Engineering and System Safety, Elsevier, vol. 215(C).
    17. Yueqi Mao & Qiang Mei & Peng Jing & Ye Zha & Ying Xue & Jiahui Huang & Danning Shao & Pan Luo, 2022. "Factors Affecting the Parental Intention of Using AVs to Escort Children: An Integrated SEM–Hybrid Choice Model Approach," Sustainability, MDPI, vol. 14(18), pages 1-21, September.
    18. Blume, Maximilian & Oberländer, Anna Maria & Röglinger, Maximilian & Rosemann, Michael & Wyrtki, Katrin, 2020. "Ex ante assessment of disruptive threats: Identifying relevant threats before one is disrupted," Technological Forecasting and Social Change, Elsevier, vol. 158(C).
    19. Luca Scrucca, 2022. "A COVINDEX based on a GAM beta regression model with an application to the COVID-19 pandemic in Italy," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 31(4), pages 881-900, October.
    20. E. Zanini & E. Eastoe & M. J. Jones & D. Randell & P. Jonathan, 2020. "Flexible covariate representations for extremes," Environmetrics, John Wiley & Sons, Ltd., vol. 31(5), August.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssc:v:71:y:2022:i:4:p:987-1013. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.