IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0232929.html
   My bibliography  Save this article

A content analysis-based approach to explore simulation verification and identify its current challenges

Author

Listed:
  • Christopher J Lynch
  • Saikou Y Diallo
  • Hamdi Kavak
  • Jose J Padilla

Abstract

Verification is a crucial process to facilitate the identification and removal of errors within simulations. This study explores semantic changes to the concept of simulation verification over the past six decades using a data-supported, automated content analysis approach. We collect and utilize a corpus of 4,047 peer-reviewed Modeling and Simulation (M&S) publications dealing with a wide range of studies of simulation verification from 1963 to 2015. We group the selected papers by decade of publication to provide insights and explore the corpus from four perspectives: (i) the positioning of prominent concepts across the corpus as a whole; (ii) a comparison of the prominence of verification, validation, and Verification and Validation (V&V) as separate concepts; (iii) the positioning of the concepts specifically associated with verification; and (iv) an evaluation of verification’s defining characteristics within each decade. Our analysis reveals unique characterizations of verification in each decade. The insights gathered helped to identify and discuss three categories of verification challenges as avenues of future research, awareness, and understanding for researchers, students, and practitioners. These categories include conveying confidence and maintaining ease of use; techniques’ coverage abilities for handling increasing simulation complexities; and new ways to provide error feedback to model users.

Suggested Citation

  • Christopher J Lynch & Saikou Y Diallo & Hamdi Kavak & Jose J Padilla, 2020. "A content analysis-based approach to explore simulation verification and identify its current challenges," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-33, May.
    • Handle: RePEc:plo:pone00:0232929
    DOI: 10.1371/journal.pone.0232929
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0232929
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0232929&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0232929?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Mingzhu Sun & Hui Xu & Xingjuan Zeng & Xin Zhao, 2017. "Automated numerical simulation of biological pattern formation based on visual feedback simulation framework," PLOS ONE, Public Library of Science, vol. 12(2), pages 1-16, February.
    2. Kleijnen, Jack P. C., 1995. "Verification and validation of simulation models," European Journal of Operational Research, Elsevier, vol. 82(1), pages 145-162, April.
    3. Ross Gore & Carlos Lemos & F. LeRon Shults & Wesley Wildman, 2018. "Forecasting Changes in Religiosity and Existential Security with an Agent-Based Model," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 21(1), pages 1-4.
    4. Joshua M. Epstein, 2014. "Agent_Zero:Toward Neurocognitive Foundations for Generative Social Science," Economics Books, Princeton University Press, edition 1, number 10169.
    5. R. W. Conway, 1963. "Some Tactical Problems in Digital Simulation," Management Science, INFORMS, vol. 10(1), pages 47-61, October.
    6. Harriet Dashnow & Andrew Lonsdale & Philip E Bourne, 2014. "Ten Simple Rules for Writing a PLOS Ten Simple Rules Article," PLOS Computational Biology, Public Library of Science, vol. 10(10), pages 1-5, October.
    7. Jose J Padilla & Hamdi Kavak & Christopher J Lynch & Ross J Gore & Saikou Y Diallo, 2018. "Temporal and spatiotemporal investigation of tourist attraction visit sentiment on Twitter," PLOS ONE, Public Library of Science, vol. 13(6), pages 1-20, June.
    8. Wilkinson, Leland & Friendly, Michael, 2009. "The History of the Cluster Heat Map," The American Statistician, American Statistical Association, vol. 63(2), pages 179-184.
    9. Mustafa Hekimoğlu & Yaman Barlas & Luis Luna-Reyes, 2016. "Sensitivity analysis for models with multiple behavior modes: a method based on behavior pattern measures," System Dynamics Review, System Dynamics Society, vol. 32(3-4), pages 332-362, July.
    10. Saltelli A. & Tarantola S., 2002. "On the Relative Importance of Input Factors in Mathematical Models: Safety Assessment for Nuclear Waste Disposal," Journal of the American Statistical Association, American Statistical Association, vol. 97, pages 702-709, September.
    11. Robert Axelrod, 1997. "Advancing the Art of Simulation in the Social Sciences," Working Papers 97-05-048, Santa Fe Institute.
    12. Saikou Y. Diallo & Ross J. Gore & Jose J. Padilla & Christopher J. Lynch, 2015. "An overview of modeling and simulation using content analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 977-1002, June.
    13. Jan C. Thiele & Winfried Kurth & Volker Grimm, 2014. "Facilitating Parameter Estimation and Sensitivity Analysis of Agent-Based Models: A Cookbook Using NetLogo and 'R'," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 17(3), pages 1-11.
    14. Mingxin Zhang & Alexander Verbraeck & Rongqing Meng & Bin Chen & Xiaogang Qiu, 2016. "Modeling Spatial Contacts for Epidemic Prediction in a Large-Scale Artificial City," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 19(4), pages 1-3.
    15. Hurrion, R. D., 1980. "An interactive visual simulation system for industrial management," European Journal of Operational Research, Elsevier, vol. 5(2), pages 86-93, August.
    16. George B. Kleindorfer & Liam O'Neill & Ram Ganeshan, 1998. "Validation in Simulation: Various Positions in the Philosophy of Science," Management Science, INFORMS, vol. 44(8), pages 1087-1099, August.
    17. Mathieu Jacomy & Tommaso Venturini & Sebastien Heymann & Mathieu Bastian, 2014. "ForceAtlas2, a Continuous Graph Layout Algorithm for Handy Network Visualization Designed for the Gephi Software," PLOS ONE, Public Library of Science, vol. 9(6), pages 1-12, June.
    18. Joshua M. Epstein & Robert L. Axtell, 1996. "Growing Artificial Societies: Social Science from the Bottom Up," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262550253, December.
    19. Paul Windrum & Giorgio Fagiolo & Alessio Moneta, 2007. "Empirical Validation of Agent-Based Models: Alternatives and Prospects," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 10(2), pages 1-8.
    20. Thomas H. Naylor & J. M. Finger, 1967. "Verification of Computer Simulation Models," Management Science, INFORMS, vol. 14(2), pages 92-101, October.
    21. F. LeRon Shults & Ross Gore & Wesley Wildman & Christopher Lynch & Justin E. Lane & Monica Toft, 2018. "A Generative Model of the Mutual Escalation of Anxiety Between Religious Groups," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 21(4), pages 1-7.
    22. Lee, Ju-Sung & Filatova, Tatiana & Ligmann-Zielinska, Arika & Hassani-Mahmooei, Behrooz & Stonedahl, Forrest & Lorscheid, Iris & Voinov, Alexey & Polhill, J. Gareth & Sun, Zhanli & Parker, Dawn C., 2015. "The complexities of agent-based modeling output analysis," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 18(4).
    23. Rémy Courdier & François Guerrin & Fenintsoa Andriamasinoro & Jean-Marie Paillat, 2002. "Agent-Based Simulation of Complex Systems: Application to Collective Management of Animal Wastes," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 5(3), pages 1-4.
    24. Misbah Razzaq & Jamil Ahmad, 2015. "Petri Net and Probabilistic Model Checking Based Approach for the Modelling, Simulation and Verification of Internet Worm Propagation," PLOS ONE, Public Library of Science, vol. 10(12), pages 1-22, December.
    25. B L Nelson, 2016. "‘Some tactical problems in digital simulation’ for the next 10 years," Journal of Simulation, Taylor & Francis Journals, vol. 10(1), pages 2-11, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    2. Juan Manuel Larrosa, 2016. "Agentes computacionales y análisis económico," Revista de Economía Institucional, Universidad Externado de Colombia - Facultad de Economía, vol. 18(34), pages 87-113, January-J.
    3. Günter Küppers & Johannes Lenhard, 2005. "Validation of Simulation: Patterns in the Social and Natural Sciences," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 8(4), pages 1-3.
    4. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    5. Flaminio Squazzoni, 2010. "The impact of agent-based models in the social sciences after 15 years of incursions," History of Economic Ideas, Fabrizio Serra Editore, Pisa - Roma, vol. 18(2), pages 197-234.
    6. F. LeRon Shults & Wesley J. Wildman, 2020. "Human Simulation and Sustainability: Ontological, Epistemological, and Ethical Reflections," Sustainability, MDPI, vol. 12(23), pages 1-16, December.
    7. Onggo, Bhakti Stephan & Karatas, Mumtaz, 2016. "Test-driven simulation modelling: A case study using agent-based maritime search-operation simulation," European Journal of Operational Research, Elsevier, vol. 254(2), pages 517-531.
    8. Vandin, Andrea & Giachini, Daniele & Lamperti, Francesco & Chiaromonte, Francesca, 2022. "Automated and distributed statistical analysis of economic agent-based models," Journal of Economic Dynamics and Control, Elsevier, vol. 143(C).
    9. José Manuel Galán & Luis R. Izquierdo & Segismundo S. Izquierdo & José Ignacio Santos & Ricardo del Olmo & Adolfo López-Paredes & Bruce Edmonds, 2009. "Errors and Artefacts in Agent-Based Modelling," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 12(1), pages 1-1.
    10. Kathrin Eismann, 2021. "Diffusion and persistence of false rumors in social media networks: implications of searchability on rumor self-correction on Twitter," Journal of Business Economics, Springer, vol. 91(9), pages 1299-1329, November.
    11. Ngo-Hoang, Dai-Long, 2019. "A research paper of Hossein Sabzian (2019), Theories and Practice of Agent based Modeling: Some practical Implications for Economic Planners, ArXiv, 54p," AgriXiv xutyz_v1, Center for Open Science.
    12. Andrew W. Bausch, 2014. "Evolving intergroup cooperation," Computational and Mathematical Organization Theory, Springer, vol. 20(4), pages 369-393, December.
    13. Corrado Monti & Marco Pangallo & Gianmarco De Francisci Morales & Francesco Bonchi, 2022. "On learning agent-based models from data," Papers 2205.05052, arXiv.org, revised Nov 2022.
    14. Xuefei Lu & Alessandro Rudi & Emanuele Borgonovo & Lorenzo Rosasco, 2020. "Faster Kriging: Facing High-Dimensional Simulators," Operations Research, INFORMS, vol. 68(1), pages 233-249, January.
    15. Patrick Afflerbach & Christopher Dun & Henner Gimpel & Dominik Parak & Johannes Seyfried, 2021. "A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(4), pages 329-348, August.
    16. Giannoccaro, Ilaria, 2015. "Adaptive supply chains in industrial districts: A complexity science approach focused on learning," International Journal of Production Economics, Elsevier, vol. 170(PB), pages 576-589.
    17. repec:hal:spmain:info:hdl:2441/13thfd12aa8rmplfudlgvgahff is not listed on IDEAS
    18. Coronese, Matteo & Occelli, Martina & Lamperti, Francesco & Roventini, Andrea, 2023. "AgriLOVE: Agriculture, land-use and technical change in an evolutionary, agent-based model," Ecological Economics, Elsevier, vol. 208(C).
    19. Hossein Sabzian & Mohammad Ali Shafia & Ali Maleki & Seyeed Mostapha Seyeed Hashemi & Ali Baghaei & Hossein Gharib, 2019. "Theories and Practice of Agent based Modeling: Some practical Implications for Economic Planners," Papers 1901.08932, arXiv.org.
    20. Marc Deissenroth & Martin Klein & Kristina Nienhaus & Matthias Reeg, 2017. "Assessing the Plurality of Actors and Policy Interactions: Agent-Based Modelling of Renewable Energy Market Integration," Complexity, Hindawi, vol. 2017, pages 1-24, December.
    21. Richard E. Nance & Robert G. Sargent, 2002. "Perspectives on the Evolution of Simulation," Operations Research, INFORMS, vol. 50(1), pages 161-172, February.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0232929. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.