IDEAS home Printed from https://ideas.repec.org/a/wly/syseng/v17y2014i4p375-391.html
   My bibliography  Save this article

The Development of Progress Plans Using a Performance‐Based Expert Judgment Model to Assess Technical Performance and Risk

Author

Listed:
  • Justin W. Eggstaff
  • Thomas A. Mazzuchi
  • Shahram Sarkani

Abstract

Systems engineers are routinely tasked with facilitating the delicate balance between cost, schedule, and technical performance in acquisition programs that are continuously subjected to various outside influences. While there are several quantitative methods to estimate acquisition program cost and schedule performance as well as identify their risks (e.g., Earned Value Management), the estimation of technical performance and technical risk is generally heuristic in nature. In order to monitor the progress of the technical aspects of an acquisition program, the systems engineering discipline utilizes the process of tracking Technical Measures to gain insight into the design and development, to assess risks and issues, and to evaluate the likelihood of realizing objectives. However, with the diversity of so many technical programs, the estimation and risk analysis of technical performance in technology acquisition programs rely on the opinions of experts because the identification and application of relevant quantitative data for constructive modeling is not practical. The Expert‐weighted Technical Risk Index methodology proposed in this article introduces a well‐established method for mathematically combining expert judgment into the realm of systems engineering to develop predictive progress plans for technical performance estimation and risk analysis. © 2013 Wiley Periodicals, Inc. Syst Eng 17

Suggested Citation

  • Justin W. Eggstaff & Thomas A. Mazzuchi & Shahram Sarkani, 2014. "The Development of Progress Plans Using a Performance‐Based Expert Judgment Model to Assess Technical Performance and Risk," Systems Engineering, John Wiley & Sons, vol. 17(4), pages 375-391, December.
    • Handle: RePEc:wly:syseng:v:17:y:2014:i:4:p:375-391
    DOI: 10.1002/sys.21273
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/sys.21273
    Download Restriction: no
    File URL: https://libkey.io/10.1002/sys.21273?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    2. Stephen C. Hora, 2004. "Probability Judgments for Continuous Quantities: Linear Combinations and Calibration," Management Science, INFORMS, vol. 50(5), pages 597-604, May.
    3. R. Winkler & Javier Muñoz & José Cervera & José Bernardo & Gail Blattenberger & Joseph Kadane & Dennis Lindley & Allan Murphy & Robert Oliver & David Ríos-Insua, 1996. "Scoring rules and the evaluation of probabilities," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 5(1), pages 1-60, June.
    4. Donna H. Rhodes & Ricardo Valerdi & Garry J. Roedler, 2009. "Systems engineering leading indicators for assessing program and technical effectiveness," Systems Engineering, John Wiley & Sons, vol. 12(1), pages 21-35, March.
    5. Ali Mosleh & George Apostolakis, 1986. "The Assessment of Probability Distributions from Expert Opinions with an Application to Seismic Fragility Curves," Risk Analysis, John Wiley & Sons, vol. 6(4), pages 447-461, December.
    6. Victor Richmond R. Jose & Robert L. Winkler, 2009. "Evaluating Quantile Assessments," Operations Research, INFORMS, vol. 57(5), pages 1287-1297, October.
    7. Cooke, Roger M. & Goossens, Louis L.H.J., 2008. "TU Delft expert judgment data base," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 657-674.
    8. Garthwaite, Paul H. & Kadane, Joseph B. & O'Hagan, Anthony, 2005. "Statistical Methods for Eliciting Probability Distributions," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 680-701, June.
    9. Norman Dalkey & Olaf Helmer, 1963. "An Experimental Application of the DELPHI Method to the Use of Experts," Management Science, INFORMS, vol. 9(3), pages 458-467, April.
    10. Robert L. Winkler, 1968. "The Consensus of Subjective Probability Distributions," Management Science, INFORMS, vol. 15(2), pages 61-75, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Claudio M. Rocco & Kash Barker & Elvis Hernández‐Perdomo, 2016. "Stochastic Ranking of Alternatives with Ordered Weighted Averaging: Comparing Network Recovery Strategies," Systems Engineering, John Wiley & Sons, vol. 19(5), pages 436-447, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eggstaff, Justin W. & Mazzuchi, Thomas A. & Sarkani, Shahram, 2014. "The effect of the number of seed variables on the performance of Cooke′s classical model," Reliability Engineering and System Safety, Elsevier, vol. 121(C), pages 72-82.
    2. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    3. Gilberto Montibeller & Detlof von Winterfeldt, 2015. "Cognitive and Motivational Biases in Decision and Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 35(7), pages 1230-1251, July.
    4. Makariou, Despoina & Barrieu, Pauline & Tzougas, George, 2021. "A finite mixture modelling perspective for combining experts’ opinions with an application to quantile-based risk measures," LSE Research Online Documents on Economics 110763, London School of Economics and Political Science, LSE Library.
    5. Robert L. Winkler & Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose, 2019. "Probability Forecasts and Their Combination: A Research Perspective," Decision Analysis, INFORMS, vol. 16(4), pages 239-260, December.
    6. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    7. Anil Gaba & Ilia Tsetlin & Robert L. Winkler, 2017. "Combining Interval Forecasts," Decision Analysis, INFORMS, vol. 14(1), pages 1-20, March.
    8. James K. Hammitt & Yifan Zhang, 2013. "Combining Experts’ Judgments: Comparison of Algorithmic Methods Using Synthetic Data," Risk Analysis, John Wiley & Sons, vol. 33(1), pages 109-120, January.
    9. Abhinav B. Agrawal & Kash Barker & Yacov Y. Haimes, 2011. "Adaptive multiplayer approach for risk‐based decision‐making: 2006 Virginia Gubernatorial Inauguration," Systems Engineering, John Wiley & Sons, vol. 14(4), pages 455-470, December.
    10. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    11. Neil A. Stiber & Mitchell J. Small & Marina Pantazidou, 2004. "Site‐Specific Updating and Aggregation of Bayesian Belief Network Models for Multiple Experts," Risk Analysis, John Wiley & Sons, vol. 24(6), pages 1529-1538, December.
    12. Despoina Makariou & Pauline Barrieu & George Tzougas, 2021. "A Finite Mixture Modelling Perspective for Combining Experts’ Opinions with an Application to Quantile-Based Risk Measures," Risks, MDPI, vol. 9(6), pages 1-25, June.
    13. Knut Are Aastveit & James Mitchell & Francesco Ravazzolo & Herman van Dijk, 2018. "The Evolution of Forecast Density Combinations in Economics," Tinbergen Institute Discussion Papers 18-069/III, Tinbergen Institute.
    14. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    15. Patrick Afflerbach & Christopher Dun & Henner Gimpel & Dominik Parak & Johannes Seyfried, 2021. "A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(4), pages 329-348, August.
    16. Robert L. Winkler & Robert T. Clemen, 2004. "Multiple Experts vs. Multiple Methods: Combining Correlation Assessments," Decision Analysis, INFORMS, vol. 1(3), pages 167-176, September.
    17. Yang, Dazhi & van der Meer, Dennis, 2021. "Post-processing in solar forecasting: Ten overarching thinking tools," Renewable and Sustainable Energy Reviews, Elsevier, vol. 140(C).
    18. Hanea, D.M. & Jagtman, H.M. & van Alphen, L.L.M.M. & Ale, B.J.M., 2010. "Quantitative and qualitative analysis of the expert and non-expert opinion in fire risk in buildings," Reliability Engineering and System Safety, Elsevier, vol. 95(7), pages 729-741.
    19. Makridakis, Spyros & Spiliotis, Evangelos & Assimakopoulos, Vassilios & Chen, Zhi & Gaba, Anil & Tsetlin, Ilia & Winkler, Robert L., 2022. "The M5 uncertainty competition: Results, findings and conclusions," International Journal of Forecasting, Elsevier, vol. 38(4), pages 1365-1385.
    20. Erin Baker & Olaitan Olaleye, 2013. "Combining Experts: Decomposition and Aggregation Order," Risk Analysis, John Wiley & Sons, vol. 33(6), pages 1116-1127, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:syseng:v:17:y:2014:i:4:p:375-391. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1002/(ISSN)1520-6858 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.