IDEAS home Printed from https://ideas.repec.org/a/spr/annopr/v342y2024i3d10.1007_s10479-022-04990-z.html
   My bibliography  Save this article

Robust weighted aggregation of expert opinions in futures studies

Author

Listed:
  • Marco Marozzi

    (Ca’ Foscari University of Venice)

  • Mario Bolzan

    (University of Padua)

  • Simone Di Zio

    (University “G. d’Annunzio” of Chieti-Pescara)

Abstract

Expert judgments are widespread in many fields, and the way in which they are collected and the procedure by which they are aggregated are considered crucial steps. From a statistical perspective, expert judgments are subjective data and must be gathered and treated as carefully and scientifically as possible. In the elicitation phase, a multitude of experts is preferable to a single expert, and techniques based on anonymity and iterations, such as Delphi, offer many advantages in terms of reducing distortions, which are mainly related to cognitive biases. There are two approaches to the aggregation of the judgments given by a panel of experts, referred to as behavioural (implying an interaction between the experts) and mathematical (involving non-interacting participants and the aggregation of the judgments using a mathematical formula). Both have advantages and disadvantages, and with the mathematical approach, the main problem concerns the subjective choice of an appropriate formula for both normalization and aggregation. We propose a new method for aggregating and processing subjective data collected using the Delphi method, with the aim of obtaining robust rankings of the outputs. This method makes it possible to normalize and aggregate the opinions of a panel of experts, while modelling different sources of uncertainty. We use an uncertainty analysis approach that allows the contemporaneous use of different aggregation and normalization functions, so that the result does not depend on the choice of a specific mathematical formula, thereby solving the problem of choice. Furthermore, we can also model the uncertainty related to the weighting system, which reflects the different expertise of the participants as well as expert opinion accuracy. By combining the Delphi method with the robust ranking procedure, we offer a new protocol covering the elicitation, the aggregation and the processing of subjective data used in the construction of Delphi-based future scenarios. The method is very flexible and can be applied to the aggregation and processing of any subjective judgments, i.e. also those outside the context of futures studies. Finally, we show the validity, reproducibility and potential of the method through its application with regard to the future of Italian families.

Suggested Citation

  • Marco Marozzi & Mario Bolzan & Simone Di Zio, 2024. "Robust weighted aggregation of expert opinions in futures studies," Annals of Operations Research, Springer, vol. 342(3), pages 1471-1493, November.
    • Handle: RePEc:spr:annopr:v:342:y:2024:i:3:d:10.1007_s10479-022-04990-z
    DOI: 10.1007/s10479-022-04990-z
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10479-022-04990-z
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10479-022-04990-z?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Marco Marozzi, 2021. "Perceived Justifiability Towards Morally Debatable Behaviors Across Europe," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 153(2), pages 759-778, January.
    2. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    3. von der Gracht, Heiko A. & Darkow, Inga-Lena, 2010. "Scenarios for the logistics services industry: A Delphi-based analysis for 2025," International Journal of Production Economics, Elsevier, vol. 127(1), pages 46-59, September.
    4. Norman Dalkey & Olaf Helmer, 1963. "An Experimental Application of the DELPHI Method to the Use of Experts," Management Science, INFORMS, vol. 9(3), pages 458-467, April.
    5. Joanna Sale & Lynne Lohfeld & Kevin Brazil, 2002. "Revisiting the Quantitative-Qualitative Debate: Implications for Mixed-Methods Research," Quality & Quantity: International Journal of Methodology, Springer, vol. 36(1), pages 43-53, February.
    6. M. Saisana & A. Saltelli & S. Tarantola, 2005. "Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 168(2), pages 307-323, March.
    7. Usher, Will & Strachan, Neil, 2013. "An expert elicitation of climate, energy and economic uncertainties," Energy Policy, Elsevier, vol. 61(C), pages 811-821.
    8. Abigail R Colson & Roger M Cooke, 2018. "Expert Elicitation: Using the Classical Model to Validate Experts’ Judgments," Review of Environmental Economics and Policy, Association of Environmental and Resource Economists, vol. 12(1), pages 113-132.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Di Zio, Simone & Bolzan, Mario & Marozzi, Marco, 2021. "Classification of Delphi outputs through robust ranking and fuzzy clustering for Delphi-based scenarios," Technological Forecasting and Social Change, Elsevier, vol. 173(C).
    2. Bokrantz, Jon & Skoogh, Anders & Berlin, Cecilia & Stahre, Johan, 2017. "Maintenance in digitalised manufacturing: Delphi-based scenarios for 2030," International Journal of Production Economics, Elsevier, vol. 191(C), pages 154-169.
    3. Elena Verdolini & Laura Díaz Anadón & Erin Baker & Valentina Bosetti & Lara Aleluia Reis, 2018. "Future Prospects for Energy Technologies: Insights from Expert Elicitations," Review of Environmental Economics and Policy, Association of Environmental and Resource Economists, vol. 12(1), pages 133-153.
    4. Joel B. Predd & Daniel N. Osherson & Sanjeev R. Kulkarni & H. Vincent Poor, 2008. "Aggregating Probabilistic Forecasts from Incoherent and Abstaining Experts," Decision Analysis, INFORMS, vol. 5(4), pages 177-189, December.
    5. Fritschy, Carolin & Spinler, Stefan, 2019. "The impact of autonomous trucks on business models in the automotive and logistics industry–a Delphi-based scenario study," Technological Forecasting and Social Change, Elsevier, vol. 148(C).
    6. Tiberius, Victor & Gojowy, Robin & Dabić, Marina, 2022. "Forecasting the future of robo advisory: A three-stage Delphi study on economic, technological, and societal implications," Technological Forecasting and Social Change, Elsevier, vol. 182(C).
    7. Vineet Kaushik & Shobha Tewari, 2023. "Modeling Opportunity Indicators Fostering Social Entrepreneurship: A Hybrid Delphi and Best-Worst Approach," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 168(1), pages 667-698, August.
    8. Förster, Bernadette & von der Gracht, Heiko, 2014. "Assessing Delphi panel composition for strategic foresight — A comparison of panels based on company-internal and external participants," Technological Forecasting and Social Change, Elsevier, vol. 84(C), pages 215-229.
    9. Matheus Pereira Libório & Lívia Maria Leite Silva & Petr Iakovlevitch Ekel & Letícia Ribeiro Figueiredo & Patrícia Bernardes, 2022. "Consensus-Based Sub-Indicator Weighting Approach: Constructing Composite Indicators Compatible with Expert Opinion," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 164(3), pages 1073-1099, December.
    10. Milford, James & Henrion, Max & Hunter, Chad & Newes, Emily & Hughes, Caroline & Baldwin, Samuel F., 2022. "Energy sector portfolio analysis with uncertainty," Applied Energy, Elsevier, vol. 306(PA).
    11. Peppel, Marcel & Ringbeck, Jürgen & Spinler, Stefan, 2022. "How will last-mile delivery be shaped in 2040? A Delphi-based scenario study," Technological Forecasting and Social Change, Elsevier, vol. 177(C).
    12. Jiayuan Dong & Jiankan Liao & Xun Huan & Daniel Cooper, 2023. "Expert elicitation and data noise learning for material flow analysis using Bayesian inference," Journal of Industrial Ecology, Yale University, vol. 27(4), pages 1105-1122, August.
    13. Engelke, Henning & Mauksch, Stefanie & Darkow, Inga-Lena & von der Gracht, Heiko A., 2015. "Opportunities for social enterprise in Germany — Evidence from an expert survey," Technological Forecasting and Social Change, Elsevier, vol. 90(PB), pages 635-646.
    14. Leypoldt, Leo & Dienhart, Christina & Caferoglu, Hüseyin & Salge, Torsten-Oliver & Antons, David, 2024. "The hydrogen field in 2035: A Delphi study forecasting dominant technology bundles," Technological Forecasting and Social Change, Elsevier, vol. 207(C).
    15. Merfeld, Katrin & Wilhelms, Mark-Philipp & Henkel, Sven & Kreutzer, Karin, 2019. "Carsharing with shared autonomous vehicles: Uncovering drivers, barriers and future developments – A four-stage Delphi study," Technological Forecasting and Social Change, Elsevier, vol. 144(C), pages 66-81.
    16. Justin W. Eggstaff & Thomas A. Mazzuchi & Shahram Sarkani, 2014. "The Development of Progress Plans Using a Performance‐Based Expert Judgment Model to Assess Technical Performance and Risk," Systems Engineering, John Wiley & Sons, vol. 17(4), pages 375-391, December.
    17. Büchel, Hendrik & Spinler, Stefan, 2024. "The impact of the metaverse on e-commerce business models – A delphi-based scenario study," Technology in Society, Elsevier, vol. 76(C).
    18. Reefke, Hendrik & Sundaram, David, 2017. "Key themes and research opportunities in sustainable supply chain management – identification and evaluation," Omega, Elsevier, vol. 66(PB), pages 195-211.
    19. Zhou, Fan & Page, Lionel & Perrons, Robert K. & Zheng, Zuduo & Washington, Simon, 2019. "Long-term forecasts for energy commodities price: What the experts think," Energy Economics, Elsevier, vol. 84(C).
    20. Kopyto, Matthias & Lechler, Sabrina & von der Gracht, Heiko A. & Hartmann, Evi, 2020. "Potentials of blockchain technology in supply chain management: Long-term judgments of an international expert panel," Technological Forecasting and Social Change, Elsevier, vol. 161(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:annopr:v:342:y:2024:i:3:d:10.1007_s10479-022-04990-z. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.