IDEAS home Printed from https://ideas.repec.org/a/wly/syseng/v17y2014i2p237-248.html
   My bibliography  Save this article

Credibility Assessment of Models and Simulations Based on NASA’s Models and Simulation Standard Using the Delphi Method

Author

Listed:
  • Jaemyung Ahn
  • Olivier L. de Weck
  • Martin Steele

Abstract

This paper introduces a procedure to assess the credibility of models and simulations (M&S) as a group activity based on NASA’s new standard for M&S NASA‐STD‐7009. The Delphi method, which is characterized by iterative surveys with controlled feedback, was selected to implement the assessment. The proposed procedure is expected to address the issues in the M&S assessment related to a high level of required expertise and group decision making. An actual credibility assessment study using the proposed procedure on an M&S platform referred to as SpaceNet has been carried out by ten panel members through a two‐round Delphi. The study concluded that the overall credibility of SpaceNet version 1.3 was between the development level and production level. The variances of the assessments in the second‐round survey were significantly reduced compared with the first‐round results, which indicates the effectiveness of the proposed procedure.

Suggested Citation

  • Jaemyung Ahn & Olivier L. de Weck & Martin Steele, 2014. "Credibility Assessment of Models and Simulations Based on NASA’s Models and Simulation Standard Using the Delphi Method," Systems Engineering, John Wiley & Sons, vol. 17(2), pages 237-248, June.
    • Handle: RePEc:wly:syseng:v:17:y:2014:i:2:p:237-248
    DOI: 10.1002/sys.21266
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/sys.21266
    Download Restriction: no
    File URL: https://libkey.io/10.1002/sys.21266?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Graefe, Andreas & Armstrong, J. Scott, 2011. "Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task," International Journal of Forecasting, Elsevier, vol. 27(1), pages 183-195, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jukrin Moon & Dongoo Lee & Taesik Lee & Jaemyung Ahn & Jindong Shin & Kyungho Yoon & Dongsik Choi, 2015. "Group Decision Procedure to Model the Dependency Structure of Complex Systems: Framework and Case Study for Critical Infrastructures," Systems Engineering, John Wiley & Sons, vol. 18(4), pages 323-338, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Prommer, Lisa & Tiberius, Victor & Kraus, Sascha, 2020. "Exploring the future of startup leadership development," Journal of Business Venturing Insights, Elsevier, vol. 14(C).
    2. Armstrong, J. Scott & Green, Kesten C. & Graefe, Andreas, 2015. "Golden rule of forecasting: Be conservative," Journal of Business Research, Elsevier, vol. 68(8), pages 1717-1731.
    3. Litsiou, Konstantia & Polychronakis, Yiannis & Karami, Azhdar & Nikolopoulos, Konstantinos, 2022. "Relative performance of judgmental methods for forecasting the success of megaprojects," International Journal of Forecasting, Elsevier, vol. 38(3), pages 1185-1196.
    4. Keller, Jonas & von der Gracht, Heiko A., 2014. "The influence of information and communication technology (ICT) on future foresight processes — Results from a Delphi survey," Technological Forecasting and Social Change, Elsevier, vol. 85(C), pages 81-92.
    5. Pavel Atanasov & Phillip Rescober & Eric Stone & Samuel A. Swift & Emile Servan-Schreiber & Philip Tetlock & Lyle Ungar & Barbara Mellers, 2017. "Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls," Management Science, INFORMS, vol. 63(3), pages 691-706, March.
    6. Christoph Markmann & Alexander Spickermann & Heiko A. von der Gracht & Alexander Brem, 2021. "Improving the question formulation in Delphi‐like surveys: Analysis of the effects of abstract language and amount of information on response behavior," Futures & Foresight Science, John Wiley & Sons, vol. 3(1), March.
    7. Hanea, A.M. & McBride, M.F. & Burgman, M.A. & Wintle, B.C. & Fidler, F. & Flander, L. & Twardy, C.R. & Manning, B. & Mascaro, S., 2017. "I nvestigate D iscuss E stimate A ggregate for structured expert judgement," International Journal of Forecasting, Elsevier, vol. 33(1), pages 267-279.
    8. Julia A. Minson & Jennifer S. Mueller & Richard P. Larrick, 2018. "The Contingent Wisdom of Dyads: When Discussion Enhances vs. Undermines the Accuracy of Collaborative Judgments," Management Science, INFORMS, vol. 64(9), pages 4177-4192, September.
    9. Pennings, Clint L.P. & van Dalen, Jan & Rook, Laurens, 2019. "Coordinating judgmental forecasting: Coping with intentional biases," Omega, Elsevier, vol. 87(C), pages 46-56.
    10. Christoph Diermann & Arnd Huchzermeier, 2017. "Case Article—Canyon Bicycles: Judgmental Demand Forecasting in Direct Sales," INFORMS Transactions on Education, INFORMS, vol. 17(2), pages 58-62, January.
    11. Bo Cowgill & Eric Zitzewitz, 2015. "Corporate Prediction Markets: Evidence from Google, Ford, and Firm X," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 82(4), pages 1309-1341.
    12. Laura Studen & Victor Tiberius, 2020. "Social Media, Quo Vadis? Prospective Development and Implications," Future Internet, MDPI, vol. 12(9), pages 1-22, August.
    13. Wright, George & Rowe, Gene, 2011. "Group-based judgmental forecasting: An integration of extant knowledge and the development of priorities for a new research agenda," International Journal of Forecasting, Elsevier, vol. 27(1), pages 1-13, January.
    14. Winkler, Jens & Kuklinski, Christian Paul Jian-Wei & Moser, Roger, 2015. "Decision making in emerging markets: The Delphi approach's contribution to coping with uncertainty and equivocality," Journal of Business Research, Elsevier, vol. 68(5), pages 1118-1126.
    15. Jukrin Moon & Dongoo Lee & Taesik Lee & Jaemyung Ahn & Jindong Shin & Kyungho Yoon & Dongsik Choi, 2015. "Group Decision Procedure to Model the Dependency Structure of Complex Systems: Framework and Case Study for Critical Infrastructures," Systems Engineering, John Wiley & Sons, vol. 18(4), pages 323-338, July.
    16. Eksoz, Can & Mansouri, S. Afshin & Bourlakis, Michael, 2014. "Collaborative forecasting in the food supply chain: A conceptual framework," International Journal of Production Economics, Elsevier, vol. 158(C), pages 120-135.
    17. Spickermann, Alexander & Zimmermann, Martin & von der Gracht, Heiko A., 2014. "Surface- and deep-level diversity in panel selection — Exploring diversity effects on response behaviour in foresight," Technological Forecasting and Social Change, Elsevier, vol. 85(C), pages 105-120.
    18. Cary Deck & David Porter, 2013. "Prediction Markets In The Laboratory," Journal of Economic Surveys, Wiley Blackwell, vol. 27(3), pages 589-603, July.
    19. Katherine A Smith & Segolame Setlhare & Allan DeCaen & Aaron Donoghue & Janell L Mensinger & Bingqing Zhang & Brennan Snow & Dikai Zambo & Kagiso Ndlovu & Ryan Littman-Quinn & Farhan Bhanji & Peter A , 2019. "Feasibility and preliminary validity evidence for remote video-based assessment of clinicians in a global health setting," PLOS ONE, Public Library of Science, vol. 14(8), pages 1-13, August.
    20. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:syseng:v:17:y:2014:i:2:p:237-248. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1002/(ISSN)1520-6858 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.