Improving Probability Judgment in Intelligence Analysis: From Structured Analysis to Statistical Aggregation
Author
Abstract
Suggested Citation
DOI: 10.1111/risa.13443
Download full text from publisher
References listed on IDEAS
- Roger M. Cooke, 2015. "The Aggregation of Expert Judgment: Do Good Things Come to Those Who Weight?," Risk Analysis, John Wiley & Sons, vol. 35(1), pages 12-15, January.
- James K. Hammitt & Yifan Zhang, 2013.
"Combining Experts’ Judgments: Comparison of Algorithmic Methods Using Synthetic Data,"
Risk Analysis, John Wiley & Sons, vol. 33(1), pages 109-120, January.
- Hammitt, James K. & Zhang, Yifan, 2012. "Combining Experts’ Judgments: Comparison of Algorithmic Methods using Synthetic Data," TSE Working Papers 12-293, Toulouse School of Economics (TSE).
- Joel B. Predd & Daniel N. Osherson & Sanjeev R. Kulkarni & H. Vincent Poor, 2008. "Aggregating Probabilistic Forecasts from Incoherent and Abstaining Experts," Decision Analysis, INFORMS, vol. 5(4), pages 177-189, December.
- Barry Charles Ezell & Steven P. Bennett & Detlof Von Winterfeldt & John Sokolowski & Andrew J. Collins, 2010. "Probabilistic Risk Analysis and Terrorism Risk," Risk Analysis, John Wiley & Sons, vol. 30(4), pages 575-589, April.
- Guanchun Wang & Sanjeev R. Kulkarni & H. Vincent Poor & Daniel N. Osherson, 2011. "Aggregating Large Sets of Probabilistic Forecasts by Weighted Coherent Adjustment," Decision Analysis, INFORMS, vol. 8(2), pages 128-144, June.
- Jacqueline Anne MacDonald & Mitchell J. Small & M. G. Morgan, 2008. "Explosion Probability of Unexploded Ordnance: Expert Beliefs," Risk Analysis, John Wiley & Sons, vol. 28(4), pages 825-841, August.
- Alison Hubbard Ashton & Robert H. Ashton, 1985. "Aggregating Subjective Forecasts: Some Empirical Results," Management Science, INFORMS, vol. 31(12), pages 1499-1508, December.
- repec:cup:judgdm:v:12:y:2017:i:4:p:369-381 is not listed on IDEAS
- Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
- Christopher W. Karvetski & Kenneth C. Olson & David R. Mandel & Charles R. Twardy, 2013. "Probabilistic Coherence Weighting for Optimizing Expert Forecasts," Decision Analysis, INFORMS, vol. 10(4), pages 305-326, December.
- Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
- Robert L. Winkler & Robert T. Clemen, 2004. "Multiple Experts vs. Multiple Methods: Combining Correlation Assessments," Decision Analysis, INFORMS, vol. 1(3), pages 167-176, September.
- Yuyu Fan & David V. Budescu & David Mandel & Mark Himmelstein, 2019. "Improving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments," Decision Analysis, INFORMS, vol. 16(3), pages 197-217, September.
- H. J. Van der Fels‐Klerx & Roger M. Cooke & Maarten N. Nauta & Louis H. Goossens & Arie H. Havelaar, 2005. "A Structured Expert Judgment Study for a Model of Campylobacter Transmission During Broiler‐Chicken Processing," Risk Analysis, John Wiley & Sons, vol. 25(1), pages 109-124, February.
- Cooke, Roger M. & Goossens, Louis L.H.J., 2008. "TU Delft expert judgment data base," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 657-674.
- repec:cup:judgdm:v:13:y:2018:i:6:p:607-621 is not listed on IDEAS
- Wright, George & Rowe, Gene & Bolger, Fergus & Gammack, John, 1994. "Coherence, Calibration, and Expertise in Judgmental Probability Forecasting," Organizational Behavior and Human Decision Processes, Elsevier, vol. 57(1), pages 1-25, January.
- Neil A. Stiber & Mitchell J. Small & Marina Pantazidou, 2004. "Site‐Specific Updating and Aggregation of Bayesian Belief Network Models for Multiple Experts," Risk Analysis, John Wiley & Sons, vol. 24(6), pages 1529-1538, December.
- Osherson, Daniel & Vardi, Moshe Y., 2006. "Aggregating disparate estimates of chance," Games and Economic Behavior, Elsevier, vol. 56(1), pages 148-173, July.
- Stan Kaplan, 1997. "The Words of Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 17(4), pages 407-417, August.
Citations
Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
Cited by:
- David R. Mandel & Daniel Irwin, 2021. "Tracking accuracy of strategic intelligence forecasts: Findings from a long‐term Canadian study," Futures & Foresight Science, John Wiley & Sons, vol. 3(3-4), September.
- Christopher W. Karvetski & David R. Mandel, 2020. "Coherence of probability judgments from uncertain evidence: Does ACH help?," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(6), pages 939-958, November.
- repec:cup:judgdm:v:15:y:2020:i:6:p:939-958 is not listed on IDEAS
Most related items
These are the items that most often cite the same works as this one and are cited by the same works as this one.- repec:cup:judgdm:v:13:y:2018:i:6:p:607-621 is not listed on IDEAS
- Christopher W. Karvetski & Kenneth C. Olson & David R. Mandel & Charles R. Twardy, 2013. "Probabilistic Coherence Weighting for Optimizing Expert Forecasts," Decision Analysis, INFORMS, vol. 10(4), pages 305-326, December.
- David R. Mandel & Christopher W. Karvetski & Mandeep K. Dhami, 2018. "Boosting intelligence analysts’ judgment accuracy: What works, what fails?," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(6), pages 607-621, November.
- Patrick Afflerbach & Christopher Dun & Henner Gimpel & Dominik Parak & Johannes Seyfried, 2021. "A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(4), pages 329-348, August.
- Yuyu Fan & David V. Budescu & David Mandel & Mark Himmelstein, 2019. "Improving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments," Decision Analysis, INFORMS, vol. 16(3), pages 197-217, September.
- repec:cup:judgdm:v:12:y:2017:i:4:p:369-381 is not listed on IDEAS
- repec:cup:judgdm:v:15:y:2020:i:5:p:783-797 is not listed on IDEAS
- David R. Mandel & Robert N. Collins & Evan F. Risko & Jonathan A. Fugelsang, 2020. "Effect of confidence interval construction on judgment accuracy," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(5), pages 783-797, September.
- Barbara A. Mellers & Joshua D. Baker & Eva Chen & David R. Mandel & Philip E. Tetlock, 2017. "How generalizable is good judgment? A multi-task, multi-benchmark study," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 12(4), pages 369-381, July.
- David R. Mandel, 2020. "Studies past and future of the past and future: Commentary on Schoemaker 2020," Futures & Foresight Science, John Wiley & Sons, vol. 2(3-4), September.
- Anca M. Hanea & Marissa F. McBride & Mark A. Burgman & Bonnie C. Wintle, 2018. "The Value of Performance Weights and Discussion in Aggregated Expert Judgments," Risk Analysis, John Wiley & Sons, vol. 38(9), pages 1781-1794, September.
- Christopher W. Karvetski & David R. Mandel, 2020. "Coherence of probability judgments from uncertain evidence: Does ACH help?," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(6), pages 939-958, November.
- Makariou, Despoina & Barrieu, Pauline & Tzougas, George, 2021. "A finite mixture modelling perspective for combining experts’ opinions with an application to quantile-based risk measures," LSE Research Online Documents on Economics 110763, London School of Economics and Political Science, LSE Library.
- Rakesh K. Sarin, 2013. "From the Editor ---Median Aggregation, Scoring Rules, Expert Forecasts, Choices with Binary Attributes, Portfolio with Dependent Projects, and Information Security," Decision Analysis, INFORMS, vol. 10(4), pages 277-278, December.
- repec:cup:judgdm:v:15:y:2020:i:6:p:939-958 is not listed on IDEAS
- Despoina Makariou & Pauline Barrieu & George Tzougas, 2021. "A Finite Mixture Modelling Perspective for Combining Experts’ Opinions with an Application to Quantile-Based Risk Measures," Risks, MDPI, vol. 9(6), pages 1-25, June.
- David V. Budescu & Eva Chen, 2015. "Identifying Expertise to Extract the Wisdom of Crowds," Management Science, INFORMS, vol. 61(2), pages 267-280, February.
- Guanchun Wang & Sanjeev R. Kulkarni & H. Vincent Poor & Daniel N. Osherson, 2011. "Aggregating Large Sets of Probabilistic Forecasts by Weighted Coherent Adjustment," Decision Analysis, INFORMS, vol. 8(2), pages 128-144, June.
- Robert L. Winkler & Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose, 2019. "Probability Forecasts and Their Combination: A Research Perspective," Decision Analysis, INFORMS, vol. 16(4), pages 239-260, December.
- L. Robin Keller & Ali Abbas & Manel Baucells & Vicki M. Bier & David Budescu & John C. Butler & Philippe Delquié & Jason R. W. Merrick & Ahti Salo & George Wu, 2010.
"From the Editors..,"
Decision Analysis, INFORMS, vol. 7(4), pages 327-330, December.
- L. Robin Keller & Manel Baucells & Kevin F. McCardle & Gregory S. Parnell & Ahti Salo, 2007. "From the Editors..," Decision Analysis, INFORMS, vol. 4(4), pages 173-175, December.
- L. Robin Keller & Manel Baucells & John C. Butler & Philippe Delquié & Jason R. W. Merrick & Gregory S. Parnell & Ahti Salo, 2008. "From the Editors..," Decision Analysis, INFORMS, vol. 5(4), pages 173-176, December.
- L. Robin Keller & Manel Baucells & John C. Butler & Philippe Delquié & Jason R. W. Merrick & Gregory S. Parnell & Ahti Salo, 2009. "From the Editors ..," Decision Analysis, INFORMS, vol. 6(4), pages 199-201, December.
- Franz Dietrich & Christian List, 2017.
"Probabilistic opinion pooling generalized. Part one: general agendas,"
Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 48(4), pages 747-786, April.
- Dietrich, Franz & List, Christian, 2013. "Probabilistic opinion pooling generalized Part one: General agendas," MPRA Paper 57253, University Library of Munich, Germany, revised Jul 2014.
- Franz Dietrich & Christian List, 2017. "Probabilistic opinion pooling generalized. Part one: General agendas," Post-Print halshs-01485792, HAL.
- Franz Dietrich & Christian List, 2017. "Probabilistic opinion pooling generalized. Part one: General agendas," PSE-Ecole d'économie de Paris (Postprint) halshs-01485792, HAL.
- Dietrich, Franz & List, Christian, 2017. "Probabilistic opinion pooling generalized. Part one: general agendas," LSE Research Online Documents on Economics 73508, London School of Economics and Political Science, LSE Library.
- Franz Dietrich & Christian List, 2017. "Probabilistic opinion pooling generalized. Part one: General agendas," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-01485792, HAL.
- Philip E. Tetlock & Christopher Karvetski & Ville A. Satopää & Kevin Chen, 2024. "Long‐range subjective‐probability forecasts of slow‐motion variables in world politics: Exploring limits on expert judgment," Futures & Foresight Science, John Wiley & Sons, vol. 6(1), March.
- Robert F. Bordley, 2009. "Combining the Opinions of Experts Who Partition Events Differently," Decision Analysis, INFORMS, vol. 6(1), pages 38-46, March.
- Donnacha Bolger & Brett Houlding, 2016. "Reliability updating in linear opinion pooling for multiple decision makers," Journal of Risk and Reliability, , vol. 230(3), pages 309-322, June.
Corrections
All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:riskan:v:40:y:2020:i:5:p:1040-1057. See general information about how to correct material in RePEc.
If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.
If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .
If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1111/(ISSN)1539-6924 .
Please note that corrections may take a couple of weeks to filter through the various RePEc services.