IDEAS home Printed from https://ideas.repec.org/a/eee/tefoso/v103y2016icp166-173.html
   My bibliography  Save this article

The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment

Author

Listed:
  • Meijering, Jurian V.
  • Tobi, Hilde

Abstract

A real-world Delphi experiment was conducted to investigate the effect of two controlled opinion feedback conditions on the drop-out rate, experts' degree of opinion change, and the increase in the level of agreement among experts. Additionally, experts' perceived usefulness of feedback was explored. In the first and second Delphi round experts received a questionnaire which consisted of two sections. Within each section experts were asked to rate several items. In round 2, experts in one condition received feedback consisting of summary statistics and rationales (S&R condition), whereas experts in the other condition received rationales only (R condition). Results showed that drop-out of experts was greater in the S&R condition than in the R condition. No difference between conditions was found concerning experts' degree of opinion change. The increase in the level of agreement across the items in the second section of the questionnaire differed significantly between conditions. This difference was mainly due to a decrease in agreement in the R condition, suggesting that feedback of rationales may increase disagreement among experts. In round 3 experts preferred to receive both summary statistics and rationales, although they tended to perceive rationales as somewhat more useful than summary statistics.

Suggested Citation

  • Meijering, Jurian V. & Tobi, Hilde, 2016. "The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 166-173.
    • Handle: RePEc:eee:tefoso:v:103:y:2016:i:c:p:166-173
    DOI: 10.1016/j.techfore.2015.11.008
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0040162515003418
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.techfore.2015.11.008?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Hung, Hsin-Ling & Altschuld, James W. & Lee, Yi-Fang, 2008. "Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation," Evaluation and Program Planning, Elsevier, vol. 31(2), pages 191-198, May.
    2. Meijering, J.V. & Kampen, J.K. & Tobi, H., 2013. "Quantifying the development of agreement among experts in Delphi studies," Technological Forecasting and Social Change, Elsevier, vol. 80(8), pages 1607-1614.
    3. David M. Boje & J. Keith Murnighan, 1982. "Group Confidence Pressures in Iterative Decisions," Management Science, INFORMS, vol. 28(10), pages 1187-1196, October.
    4. Rowe, Gene & Wright, George, 1999. "The Delphi technique as a forecasting tool: issues and analysis," International Journal of Forecasting, Elsevier, vol. 15(4), pages 353-375, October.
    5. Rowe, Gene & Wright, George, 1996. "The impact of task characteristics on the performance of structured group forecasting techniques," International Journal of Forecasting, Elsevier, vol. 12(1), pages 73-89, March.
    6. Norman Dalkey & Olaf Helmer, 1963. "An Experimental Application of the DELPHI Method to the Use of Experts," Management Science, INFORMS, vol. 9(3), pages 458-467, April.
    7. Landeta, Jon & Barrutia, Jon, 2011. "People consultation to construct the future: A Delphi application," International Journal of Forecasting, Elsevier, vol. 27(1), pages 134-151, January.
    8. Landeta, Jon & Barrutia, Jon, 2011. "People consultation to construct the future: A Delphi application," International Journal of Forecasting, Elsevier, vol. 27(1), pages 134-151.
    9. Yaniv, Ilan, 2004. "Receiving other people's advice: Influence and benefit," Organizational Behavior and Human Decision Processes, Elsevier, vol. 93(1), pages 1-13, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Belton, Ian & Wright, George & Sissons, Aileen & Bolger, Fergus & Crawford, Megan M. & Hamlin, Iain & Taylor Browne Lūka, Courtney & Vasilichi, Alexandrina, 2021. "Delphi with feedback of rationales: How large can a Delphi group be such that participants are not overloaded, de-motivated, or disengaged?," Technological Forecasting and Social Change, Elsevier, vol. 170(C).
    2. Bonaccorsi, Andrea & Apreda, Riccardo & Fantoni, Gualtiero, 2020. "Expert biases in technology foresight. Why they are a problem and how to mitigate them," Technological Forecasting and Social Change, Elsevier, vol. 151(C).
    3. Schlecht, Laura & Schneider, Sabrina & Buchwald, Arne, 2021. "The prospective value creation potential of Blockchain in business models: A delphi study," Technological Forecasting and Social Change, Elsevier, vol. 166(C).
    4. Barrios, Maite & Guilera, Georgina & Nuño, Laura & Gómez-Benito, Juana, 2021. "Consensus in the delphi method: What makes a decision change?," Technological Forecasting and Social Change, Elsevier, vol. 163(C).
    5. Simon Haworth & Paul Montgomery & Jason Schaub, 2023. "A Delphi Study to Develop Items for a New Tool for Measuring Child Neglect for Use by Multi-Agency Practitioners in the UK," Social Sciences, MDPI, vol. 12(4), pages 1-20, April.
    6. Vieira, Ana C.L. & Oliveira, Mónica D. & Bana e Costa, Carlos A., 2020. "Enhancing knowledge construction processes within multicriteria decision analysis: The Collaborative Value Modelling framework," Omega, Elsevier, vol. 94(C).
    7. Kawamoto, Carlos Tadao & Wright, James Terence Coulter & Spers, Renata Giovinazzo & de Carvalho, Daniel Estima, 2019. "Can we make use of perception of questions' easiness in Delphi-like studies? Some results from an experiment with an alternative feedback," Technological Forecasting and Social Change, Elsevier, vol. 140(C), pages 296-305.
    8. Meijering, Jurian Vincent & Tobi, Hilde, 2018. "The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial," International Journal of Forecasting, Elsevier, vol. 34(2), pages 216-224.
    9. Badghish, Saeed & Shaik, Aqueeb Sohail & Sahore, Nidhi & Srivastava, Shalini & Masood, Ayesha, 2024. "Can transactional use of AI-controlled voice assistants for service delivery pickup pace in the near future? A social learning theory (SLT) perspective," Technological Forecasting and Social Change, Elsevier, vol. 198(C).
    10. Belton, Ian & MacDonald, Alice & Wright, George & Hamlin, Iain, 2019. "Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process," Technological Forecasting and Social Change, Elsevier, vol. 147(C), pages 72-82.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Förster, Bernadette & von der Gracht, Heiko, 2014. "Assessing Delphi panel composition for strategic foresight — A comparison of panels based on company-internal and external participants," Technological Forecasting and Social Change, Elsevier, vol. 84(C), pages 215-229.
    2. Belton, Ian & MacDonald, Alice & Wright, George & Hamlin, Iain, 2019. "Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process," Technological Forecasting and Social Change, Elsevier, vol. 147(C), pages 72-82.
    3. Barrios, Maite & Guilera, Georgina & Nuño, Laura & Gómez-Benito, Juana, 2021. "Consensus in the delphi method: What makes a decision change?," Technological Forecasting and Social Change, Elsevier, vol. 163(C).
    4. Winkler, Jens & Moser, Roger, 2016. "Biases in future-oriented Delphi studies: A cognitive perspective," Technological Forecasting and Social Change, Elsevier, vol. 105(C), pages 63-76.
    5. Bolger, Fergus & Rowe, Gene & Belton, Ian & Crawford, Megan M & Hamlin, Iain & Sissons, Aileen & Taylor Browne Lūka, Courtney & Vasilichi, Alexandrina & Wright, George, 2020. "The Simulated Group Response Paradigm: A new approach to the study of opinion change in Delphi and other structured-group techniques," OSF Preprints 4ufzg, Center for Open Science.
    6. Alyami, Saleh. H. & Rezgui, Yacine & Kwan, Alan, 2013. "Developing sustainable building assessment scheme for Saudi Arabia: Delphi consultation approach," Renewable and Sustainable Energy Reviews, Elsevier, vol. 27(C), pages 43-54.
    7. Meijering, Jurian Vincent & Tobi, Hilde, 2018. "The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial," International Journal of Forecasting, Elsevier, vol. 34(2), pages 216-224.
    8. Engelke, Henning & Mauksch, Stefanie & Darkow, Inga-Lena & von der Gracht, Heiko A., 2015. "Opportunities for social enterprise in Germany — Evidence from an expert survey," Technological Forecasting and Social Change, Elsevier, vol. 90(PB), pages 635-646.
    9. Nur, Suardi & Burton, Bruce & Bergmann, Ariel, 2023. "Evidence on optimal risk allocation models for Indonesian geothermal projects under PPP contracts," Utilities Policy, Elsevier, vol. 81(C).
    10. Kawamoto, Carlos Tadao & Wright, James Terence Coulter & Spers, Renata Giovinazzo & de Carvalho, Daniel Estima, 2019. "Can we make use of perception of questions' easiness in Delphi-like studies? Some results from an experiment with an alternative feedback," Technological Forecasting and Social Change, Elsevier, vol. 140(C), pages 296-305.
    11. Aldossary, Naief A. & Rezgui, Yacine & Kwan, Alan, 2015. "Consensus-based low carbon domestic design framework for sustainable homes," Renewable and Sustainable Energy Reviews, Elsevier, vol. 51(C), pages 417-432.
    12. Sebastian Hinderer & Leif Brändle & Andreas Kuckertz, 2021. "Transition to a Sustainable Bioeconomy," Sustainability, MDPI, vol. 13(15), pages 1-16, July.
    13. Rowe, Gene & Wright, George, 1999. "The Delphi technique as a forecasting tool: issues and analysis," International Journal of Forecasting, Elsevier, vol. 15(4), pages 353-375, October.
    14. Wright, George & Rowe, Gene, 2011. "Group-based judgmental forecasting: An integration of extant knowledge and the development of priorities for a new research agenda," International Journal of Forecasting, Elsevier, vol. 27(1), pages 1-13, January.
    15. Winkler, Jens & Kuklinski, Christian Paul Jian-Wei & Moser, Roger, 2015. "Decision making in emerging markets: The Delphi approach's contribution to coping with uncertainty and equivocality," Journal of Business Research, Elsevier, vol. 68(5), pages 1118-1126.
    16. Spickermann, Alexander & Zimmermann, Martin & von der Gracht, Heiko A., 2014. "Surface- and deep-level diversity in panel selection — Exploring diversity effects on response behaviour in foresight," Technological Forecasting and Social Change, Elsevier, vol. 85(C), pages 105-120.
    17. Worrell, James L. & Di Gangi, Paul M. & Bush, Ashley A., 2013. "Exploring the use of the Delphi method in accounting information systems research," International Journal of Accounting Information Systems, Elsevier, vol. 14(3), pages 193-208.
    18. repec:eee:intfor:v:27:y:2011:i:1:p:1-13 is not listed on IDEAS
    19. Delke, Vincent & Schiele, Holger & Buchholz, Wolfgang & Kelly, Stephen, 2023. "Implementing Industry 4.0 technologies: Future roles in purchasing and supply management," Technological Forecasting and Social Change, Elsevier, vol. 196(C).
    20. Marco van Gelderen & Johan Wiklund & Jeffery S. McMullen, 2021. "Entrepreneurship in the Future: A Delphi Study of ETP and JBV Editorial Board Members," Entrepreneurship Theory and Practice, , vol. 45(5), pages 1239-1275, September.
    21. Chang, Victor & Liu, Ben S.C. & Sudharshan, D. & Xu, Qianwen Ariel, 2021. "Towards an effective negotiation modeling: Investigating transboundary disputes with cases of lower possibilities," Technological Forecasting and Social Change, Elsevier, vol. 165(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:tefoso:v:103:y:2016:i:c:p:166-173. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.sciencedirect.com/science/journal/00401625 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.