IDEAS home Printed from https://ideas.repec.org/a/eee/tefoso/v104y2016icp78-88.html
   My bibliography  Save this article

Advancing the State of Policy Delphi Practice: A Systematic Review Evaluating Methodological Evolution, Innovation, and Opportunities

Author

Listed:
  • de Loë, Rob C.
  • Melnychuk, Natalya
  • Murray, Dan
  • Plummer, Ryan

Abstract

The policy Delphi is a method that uses iterative stages of data collection to reveal positions on an issue within a panel of people with relevant knowledge. Policy Delphi surveys have become popular in a variety of disciplines since the method was first proposed in this journal in 1970. In this paper, we benchmark the state-of-the-art in policy Delphi methods, focusing on strengths and limitations, and on innovative ways of addressing key shortcomings. We report findings from a systematic review of 63 empirical studies conducted between 1971 and the end of 2014 that used the policy Delphi method. We found little consistency in how studies have been designed and executed. The inherent flexibility of the method is a strength, but a lack of consistency in how it is used undermines the ability of analysts to generate accessible insights. Specifically, our analysis reveals limited use of validity and reliability tests, a blurring of conventional and policy Delphi rationales, diverse data collection and analysis techniques, and mixed quality when reporting the approach, format, and results for individual studies. Indeed, potential new users of the method will struggle to understand what a policy Delphi survey actually is. We conclude with advice for addressing key shortcomings in current policy Delphi practice.

Suggested Citation

  • de Loë, Rob C. & Melnychuk, Natalya & Murray, Dan & Plummer, Ryan, 2016. "Advancing the State of Policy Delphi Practice: A Systematic Review Evaluating Methodological Evolution, Innovation, and Opportunities," Technological Forecasting and Social Change, Elsevier, vol. 104(C), pages 78-88.
    • Handle: RePEc:eee:tefoso:v:104:y:2016:i:c:p:78-88
    DOI: 10.1016/j.techfore.2015.12.009
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0040162515004138
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.techfore.2015.12.009?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Nicole Lisa Klenk & Gordon M. Hickey, 2012. "Improving the social robustness of research networks for sustainable natural resource management: Results of a Delphi study in Canada," Science and Public Policy, Oxford University Press, vol. 39(3), pages 357-372, April.
    2. Hung, Hsin-Ling & Altschuld, James W. & Lee, Yi-Fang, 2008. "Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation," Evaluation and Program Planning, Elsevier, vol. 31(2), pages 191-198, May.
    3. Klenk, Nicole L. & Hickey, Gordon M., 2011. "A virtual and anonymous, deliberative and analytic participation process for planning and evaluation: The Concept Mapping Policy Delphi," International Journal of Forecasting, Elsevier, vol. 27(1), pages 152-165, January.
    4. Nick Novakowski & Barry Wellar, 2008. "Using the Delphi Technique in Normative Planning Research: Methodological Design Considerations," Environment and Planning A, , vol. 40(6), pages 1485-1500, June.
    5. Klenk, Nicole L. & Hickey, Gordon M., 2011. "A virtual and anonymous, deliberative and analytic participation process for planning and evaluation: The Concept Mapping Policy Delphi," International Journal of Forecasting, Elsevier, vol. 27(1), pages 152-165.
    6. Arovuori, Kyösti & Kola, Jukka, 2005. "Policies and Measures for Multifunctional Agriculture: Experts' Insight," International Food and Agribusiness Management Review, International Food and Agribusiness Management Association, vol. 8(3), pages 1-31.
    7. O'Loughlin, Rosalyn & Kelly, Alan, 2004. "Equity in resource allocation in the Irish health service: A policy Delphi study," Health Policy, Elsevier, vol. 67(3), pages 271-280, March.
    8. Gartner, William B., 1990. "What are we talking about when we talk about entrepreneurship?," Journal of Business Venturing, Elsevier, vol. 5(1), pages 15-28, January.
    9. Spickermann, Alexander & Zimmermann, Martin & von der Gracht, Heiko A., 2014. "Surface- and deep-level diversity in panel selection — Exploring diversity effects on response behaviour in foresight," Technological Forecasting and Social Change, Elsevier, vol. 85(C), pages 105-120.
    10. Rowe, Gene & Wright, George, 1999. "The Delphi technique as a forecasting tool: issues and analysis," International Journal of Forecasting, Elsevier, vol. 15(4), pages 353-375, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Belton, Ian & MacDonald, Alice & Wright, George & Hamlin, Iain, 2019. "Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process," Technological Forecasting and Social Change, Elsevier, vol. 147(C), pages 72-82.
    2. Makkonen, Mari & Pätäri, Satu & Jantunen, Ari & Viljainen, Satu, 2012. "Competition in the European electricity markets – outcomes of a Delphi study," Energy Policy, Elsevier, vol. 44(C), pages 431-440.
    3. Hop, G.E. & Mourits, M.C.M. & Oude Lansink, A.G.J.M. & Saatkamp, H.W., 2014. "Future structural developments in Dutch and German livestock production and implications for contagious livestock disease control," Technological Forecasting and Social Change, Elsevier, vol. 82(C), pages 95-114.
    4. Winkler, Jens & Kuklinski, Christian Paul Jian-Wei & Moser, Roger, 2015. "Decision making in emerging markets: The Delphi approach's contribution to coping with uncertainty and equivocality," Journal of Business Research, Elsevier, vol. 68(5), pages 1118-1126.
    5. Marzieh Fallah & Lanndon Ocampo, 2021. "The use of the Delphi method with non-parametric analysis for identifying sustainability criteria and indicators in evaluating ecotourism management: the case of Penang National Park (Malaysia)," Environment Systems and Decisions, Springer, vol. 41(1), pages 45-62, March.
    6. Klenk, Nicole L. & Hickey, Gordon M., 2013. "How can formal research networks produce more socially robust forest science?," Forest Policy and Economics, Elsevier, vol. 37(C), pages 44-56.
    7. Winkler, Jens & Moser, Roger, 2016. "Biases in future-oriented Delphi studies: A cognitive perspective," Technological Forecasting and Social Change, Elsevier, vol. 105(C), pages 63-76.
    8. Makkonen, Marika & Hujala, Teppo & Uusivuori, Jussi, 2016. "Policy experts' propensity to change their opinion along Delphi rounds," Technological Forecasting and Social Change, Elsevier, vol. 109(C), pages 61-68.
    9. Roßmann, Bernhard & Canzaniello, Angelo & von der Gracht, Heiko & Hartmann, Evi, 2018. "The future and social impact of Big Data Analytics in Supply Chain Management: Results from a Delphi study," Technological Forecasting and Social Change, Elsevier, vol. 130(C), pages 135-149.
    10. Aldossary, Naief A. & Rezgui, Yacine & Kwan, Alan, 2015. "Consensus-based low carbon domestic design framework for sustainable homes," Renewable and Sustainable Energy Reviews, Elsevier, vol. 51(C), pages 417-432.
    11. Bohan, Sarah & Tippmann, Esther & Levie, Jonathan & Igoe, Josephine & Bowers, Blake, 2024. "What is scaling?," Journal of Business Venturing, Elsevier, vol. 39(1).
    12. Gebhardt, Maximilian & Spieske, Alexander & Birkel, Hendrik, 2022. "The future of the circular economy and its effect on supply chain dependencies: Empirical evidence from a Delphi study," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 157(C).
    13. Fritschy, Carolin & Spinler, Stefan, 2019. "The impact of autonomous trucks on business models in the automotive and logistics industry–a Delphi-based scenario study," Technological Forecasting and Social Change, Elsevier, vol. 148(C).
    14. Wright, George & Rowe, Gene, 2011. "Group-based judgmental forecasting: An integration of extant knowledge and the development of priorities for a new research agenda," International Journal of Forecasting, Elsevier, vol. 27(1), pages 1-13, January.
    15. Klenk, Nicole L. & Hickey, Gordon M., 2011. "A virtual and anonymous, deliberative and analytic participation process for planning and evaluation: The Concept Mapping Policy Delphi," International Journal of Forecasting, Elsevier, vol. 27(1), pages 152-165.
    16. Spickermann, Alexander & Zimmermann, Martin & von der Gracht, Heiko A., 2014. "Surface- and deep-level diversity in panel selection — Exploring diversity effects on response behaviour in foresight," Technological Forecasting and Social Change, Elsevier, vol. 85(C), pages 105-120.
    17. Peppel, Marcel & Ringbeck, Jürgen & Spinler, Stefan, 2022. "How will last-mile delivery be shaped in 2040? A Delphi-based scenario study," Technological Forecasting and Social Change, Elsevier, vol. 177(C).
    18. Meijering, Jurian V. & Tobi, Hilde, 2016. "The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 166-173.
    19. repec:eee:intfor:v:27:y:2011:i:1:p:1-13 is not listed on IDEAS
    20. Alvarez Etxeberria, Igor & Garayar, Ainhoa & Calvo Sánchez, José Antonio, 2015. "Development of sustainability reports for farming operations in the Basque Country using the Delphi method," Revista de Contabilidad - Spanish Accounting Review, Elsevier, vol. 18(1), pages 44-54.
    21. Peeraer, Jef & Van Petegem, Peter, 2015. "Integration or transformation? Looking in the future of Information and Communication Technology in education in Vietnam," Evaluation and Program Planning, Elsevier, vol. 48(C), pages 47-56.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:tefoso:v:104:y:2016:i:c:p:78-88. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.sciencedirect.com/science/journal/00401625 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.