IDEAS home Printed from https://ideas.repec.org/a/eee/tefoso/v163y2021ics004016252031310x.html
   My bibliography  Save this article

Consensus in the delphi method: What makes a decision change?

Author

Listed:
  • Barrios, Maite
  • Guilera, Georgina
  • Nuño, Laura
  • Gómez-Benito, Juana

Abstract

We examined whether giving feedback to participants in a Delphi study about the level of agreement across the expert panel had an effect on opinion change between rounds. We also considered the potential influence of participants’ sociodemographic and professional characteristics. Five three-round Delphi studies were conducted independently, in which a total of 628 mental health experts responded to all three rounds. In each study, participants had to decide, based on their experience, whether a series of categories were relevant. The percentage of group agreement (i.e., percentage of participants who considered each category as relevant) in round 2 was shown as feedback in round 3, and responses in rounds 2 and 3 were considered to analyze opinion change. Results showed that when the feedback given in round 3 indicated that ≥75% of experts considered a category to be relevant, there was a further shift in opinion towards the group opinion (i.e., the category then yielded even greater consensus), whereas if the feedback indicated <75% group agreement, individual opinions tended to shift against the group opinion (i.e., consensus over the category decreased). Neither sociodemographic nor professional variables had a significant effect in explaining opinion shift. These results show that in Delphi studies, feedback has an influence on individual responses and the achievement of consensus.

Suggested Citation

  • Barrios, Maite & Guilera, Georgina & Nuño, Laura & Gómez-Benito, Juana, 2021. "Consensus in the delphi method: What makes a decision change?," Technological Forecasting and Social Change, Elsevier, vol. 163(C).
    • Handle: RePEc:eee:tefoso:v:163:y:2021:i:c:s004016252031310x
    DOI: 10.1016/j.techfore.2020.120484
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S004016252031310X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.techfore.2020.120484?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Meijering, Jurian Vincent & Tobi, Hilde, 2018. "The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial," International Journal of Forecasting, Elsevier, vol. 34(2), pages 216-224.
    2. Meijering, J.V. & Kampen, J.K. & Tobi, H., 2013. "Quantifying the development of agreement among experts in Delphi studies," Technological Forecasting and Social Change, Elsevier, vol. 80(8), pages 1607-1614.
    3. Jiang, Ruth & Kleer, Robin & Piller, Frank T., 2017. "Predicting the future of additive manufacturing: A Delphi study on economic and societal implications of 3D printing for 2030," Technological Forecasting and Social Change, Elsevier, vol. 117(C), pages 84-97.
    4. Kauko, Karlo & Palmroos, Peter, 2014. "The Delphi method in forecasting financial markets— An experimental study," International Journal of Forecasting, Elsevier, vol. 30(2), pages 313-327.
    5. Meijering, Jurian V. & Tobi, Hilde, 2016. "The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 166-173.
    6. Rowe, Gene & Wright, George, 1999. "The Delphi technique as a forecasting tool: issues and analysis," International Journal of Forecasting, Elsevier, vol. 15(4), pages 353-375, October.
    7. Rowe, Gene & Wright, George, 1996. "The impact of task characteristics on the performance of structured group forecasting techniques," International Journal of Forecasting, Elsevier, vol. 12(1), pages 73-89, March.
    8. Zimmermann, Martin & Darkow, Inga-Lena & von der Gracht, Heiko A., 2012. "Integrating Delphi and participatory backcasting in pursuit of trustworthiness — The case of electric mobility in Germany," Technological Forecasting and Social Change, Elsevier, vol. 79(9), pages 1605-1621.
    9. Landeta, Jon & Barrutia, Jon, 2011. "People consultation to construct the future: A Delphi application," International Journal of Forecasting, Elsevier, vol. 27(1), pages 134-151, January.
    10. Förster, Bernadette & von der Gracht, Heiko, 2014. "Assessing Delphi panel composition for strategic foresight — A comparison of panels based on company-internal and external participants," Technological Forecasting and Social Change, Elsevier, vol. 84(C), pages 215-229.
    11. Landeta, Jon & Barrutia, Jon, 2011. "People consultation to construct the future: A Delphi application," International Journal of Forecasting, Elsevier, vol. 27(1), pages 134-151.
    12. Laura Nuño & Georgina Guilera & Michaela Coenen & Emilio Rojo & Juana Gómez-Benito & Maite Barrios, 2019. "Functioning in schizophrenia from the perspective of psychologists: A worldwide study," PLOS ONE, Public Library of Science, vol. 14(6), pages 1-16, June.
    13. von der Gracht, Heiko A., 2012. "Consensus measurement in Delphi studies," Technological Forecasting and Social Change, Elsevier, vol. 79(8), pages 1525-1536.
    14. Makkonen, Marika & Hujala, Teppo & Uusivuori, Jussi, 2016. "Policy experts' propensity to change their opinion along Delphi rounds," Technological Forecasting and Social Change, Elsevier, vol. 109(C), pages 61-68.
    15. Yaniv, Ilan & Milyavsky, Maxim, 2007. "Using advice from multiple sources to revise and improve judgments," Organizational Behavior and Human Decision Processes, Elsevier, vol. 103(1), pages 104-120, May.
    16. Stanislav Birko & Edward S Dove & Vural Özdemir, 2015. "Evaluation of Nine Consensus Indices in Delphi Foresight Research and Their Dependency on Delphi Survey Characteristics: A Simulation Study and Debate on Delphi Design and Interpretation," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-14, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Keren A. Vivas & Ramon E. Vera & Sudipta Dasmohapatra & Ronald Marquez & Sophie Van Schoubroeck & Naycari Forfora & Antonio José Azuaje & Richard B. Phillips & Hasan Jameel & Jason A. Delborne & Danie, 2024. "A Multi-Criteria Approach for Quantifying the Impact of Global Megatrends on the Pulp and Paper Industry: Insights into Digitalization, Social Behavior Change, and Sustainability," Logistics, MDPI, vol. 8(2), pages 1-32, April.
    2. Francisco Torres-Romero & Julio César Acosta-Prado, 2022. "Knowledge Management Practices and Ecological Restoration of the Tropical Dry Forest in Colombia," Land, MDPI, vol. 11(3), pages 1-19, February.
    3. Gricelda Herrera-Franco & F. Javier Montalván & Andrés Velastegui-Montoya & Jhon Caicedo-Potosí, 2022. "Vulnerability in a Populated Coastal Zone and Its Influence by Oil Wells in Santa Elena, Ecuador," Resources, MDPI, vol. 11(8), pages 1-18, July.
    4. Ali Saleh Alammary, 2024. "Blended Learning Delivery Methods for a Sustainable Learning Environment: A Delphi Study," Sustainability, MDPI, vol. 16(8), pages 1-17, April.
    5. Biljana Kulisic & Bruno Gagnon & Jörg Schweinle & Sam Van Holsbeeck & Mark Brown & Jurica Simurina & Ioannis Dimitriou & Heather McDonald, 2021. "The Contributions of Biomass Supply for Bioenergy in the Post-COVID-19 Recovery," Energies, MDPI, vol. 14(24), pages 1-31, December.
    6. Jon Landeta & Aitziber Lertxundi, 2024. "Quality indicators for Delphi studies," Futures & Foresight Science, John Wiley & Sons, vol. 6(1), March.
    7. Carol Nash, 2024. "Naturalistic Decision-Making in Intentional Communities: Insights from Youth, Disabled Persons, and Children on Achieving United Nations Sustainable Development Goals for Equality, Peace, and Justice," Challenges, MDPI, vol. 15(3), pages 1-26, September.
    8. Xu, Jinou & Pero, Margherita & Fabbri, Margherita, 2023. "Unfolding the link between big data analytics and supply chain planning," Technological Forecasting and Social Change, Elsevier, vol. 196(C).
    9. Beiderbeck, Daniel & Evans, Nicolas & Frevel, Nicolas & Schmidt, Sascha L., 2023. "The impact of technology on the future of football – A global Delphi study," Technological Forecasting and Social Change, Elsevier, vol. 187(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Winkler, Jens & Moser, Roger, 2016. "Biases in future-oriented Delphi studies: A cognitive perspective," Technological Forecasting and Social Change, Elsevier, vol. 105(C), pages 63-76.
    2. Kawamoto, Carlos Tadao & Wright, James Terence Coulter & Spers, Renata Giovinazzo & de Carvalho, Daniel Estima, 2019. "Can we make use of perception of questions' easiness in Delphi-like studies? Some results from an experiment with an alternative feedback," Technological Forecasting and Social Change, Elsevier, vol. 140(C), pages 296-305.
    3. Meijering, Jurian Vincent & Tobi, Hilde, 2018. "The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial," International Journal of Forecasting, Elsevier, vol. 34(2), pages 216-224.
    4. Belton, Ian & MacDonald, Alice & Wright, George & Hamlin, Iain, 2019. "Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process," Technological Forecasting and Social Change, Elsevier, vol. 147(C), pages 72-82.
    5. Förster, Bernadette & von der Gracht, Heiko, 2014. "Assessing Delphi panel composition for strategic foresight — A comparison of panels based on company-internal and external participants," Technological Forecasting and Social Change, Elsevier, vol. 84(C), pages 215-229.
    6. Spickermann, Alexander & Zimmermann, Martin & von der Gracht, Heiko A., 2014. "Surface- and deep-level diversity in panel selection — Exploring diversity effects on response behaviour in foresight," Technological Forecasting and Social Change, Elsevier, vol. 85(C), pages 105-120.
    7. Meijering, Jurian V. & Tobi, Hilde, 2016. "The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 166-173.
    8. Kopyto, Matthias & Lechler, Sabrina & von der Gracht, Heiko A. & Hartmann, Evi, 2020. "Potentials of blockchain technology in supply chain management: Long-term judgments of an international expert panel," Technological Forecasting and Social Change, Elsevier, vol. 161(C).
    9. Tiberius, Victor & Gojowy, Robin & Dabić, Marina, 2022. "Forecasting the future of robo advisory: A three-stage Delphi study on economic, technological, and societal implications," Technological Forecasting and Social Change, Elsevier, vol. 182(C).
    10. Alyami, Saleh. H. & Rezgui, Yacine & Kwan, Alan, 2013. "Developing sustainable building assessment scheme for Saudi Arabia: Delphi consultation approach," Renewable and Sustainable Energy Reviews, Elsevier, vol. 27(C), pages 43-54.
    11. Tobias Meyer & Heiko A. von der Gracht & Evi Hartmann, 2022. "Technology foresight for sustainable road freight transportation: Insights from a global real‐time Delphi study," Futures & Foresight Science, John Wiley & Sons, vol. 4(1), March.
    12. von Briel, Frederik, 2018. "The future of omnichannel retail: A four-stage Delphi study," Technological Forecasting and Social Change, Elsevier, vol. 132(C), pages 217-229.
    13. Sebastian Hinderer & Leif Brändle & Andreas Kuckertz, 2021. "Transition to a Sustainable Bioeconomy," Sustainability, MDPI, vol. 13(15), pages 1-16, July.
    14. Moutinho, José da Assunção & Fernandes, Gabriela & Rabechini, Roque, 2024. "Evaluation in design science: A framework to support project studies in the context of University Research Centres," Evaluation and Program Planning, Elsevier, vol. 102(C).
    15. Gebhardt, Maximilian & Spieske, Alexander & Birkel, Hendrik, 2022. "The future of the circular economy and its effect on supply chain dependencies: Empirical evidence from a Delphi study," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 157(C).
    16. Peppel, Marcel & Ringbeck, Jürgen & Spinler, Stefan, 2022. "How will last-mile delivery be shaped in 2040? A Delphi-based scenario study," Technological Forecasting and Social Change, Elsevier, vol. 177(C).
    17. Engelke, Henning & Mauksch, Stefanie & Darkow, Inga-Lena & von der Gracht, Heiko A., 2015. "Opportunities for social enterprise in Germany — Evidence from an expert survey," Technological Forecasting and Social Change, Elsevier, vol. 90(PB), pages 635-646.
    18. Haidar, Ali & Guimón, José & Alon, Ido, 2024. "Can graphene fuel a transformative change in energy storage technologies? A scenario analysis for the next two decades," Technological Forecasting and Social Change, Elsevier, vol. 202(C).
    19. Schlecht, Laura & Schneider, Sabrina & Buchwald, Arne, 2021. "The prospective value creation potential of Blockchain in business models: A delphi study," Technological Forecasting and Social Change, Elsevier, vol. 166(C).
    20. Bolger, Fergus & Rowe, Gene & Belton, Ian & Crawford, Megan M & Hamlin, Iain & Sissons, Aileen & Taylor Browne Lūka, Courtney & Vasilichi, Alexandrina & Wright, George, 2020. "The Simulated Group Response Paradigm: A new approach to the study of opinion change in Delphi and other structured-group techniques," OSF Preprints 4ufzg, Center for Open Science.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:tefoso:v:163:y:2021:i:c:s004016252031310x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.sciencedirect.com/science/journal/00401625 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.