IDEAS home Printed from https://ideas.repec.org/a/eee/epplan/v31y2008i2p191-198.html
   My bibliography  Save this article

Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation

Author

Listed:
  • Hung, Hsin-Ling
  • Altschuld, James W.
  • Lee, Yi-Fang

Abstract

Although the Delphi is widely used, research on certain methodological issues is somewhat limited. After a brief introduction to the strengths, limitations, and methodological challenges of the technique, we share our experiences (as well as problems encountered) with an electronic Delphi of educational program evaluation (EPE) in the Asia-Pacific region. The study is described followed by a discussion of the difficulties in participant recruitment and selection, sample size, instrumentation, data collection and analysis and attempts to resolve them. Some of these problems are generic to the Delphi whereas others related to the specifics of the investigation. What we learned should be useful for future Delphis with a similar focus.

Suggested Citation

  • Hung, Hsin-Ling & Altschuld, James W. & Lee, Yi-Fang, 2008. "Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation," Evaluation and Program Planning, Elsevier, vol. 31(2), pages 191-198, May.
    • Handle: RePEc:eee:epplan:v:31:y:2008:i:2:p:191-198
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0149-7189(08)00013-X
    Download Restriction: Full text for ScienceDirect subscribers only
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Rowe, Gene & Wright, George, 1999. "The Delphi technique as a forecasting tool: issues and analysis," International Journal of Forecasting, Elsevier, vol. 15(4), pages 353-375, October.
    2. George T. Milkovich & Anthony J. Annoni & Thomas A. Mahoney, 1972. "The Use of the Delphi Procedures in Manpower Forecasting," Management Science, INFORMS, vol. 19(4-Part-1), pages 381-388, December.
    3. Norman Dalkey & Olaf Helmer, 1963. "An Experimental Application of the DELPHI Method to the Use of Experts," Management Science, INFORMS, vol. 9(3), pages 458-467, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. de Loë, Rob C. & Melnychuk, Natalya & Murray, Dan & Plummer, Ryan, 2016. "Advancing the State of Policy Delphi Practice: A Systematic Review Evaluating Methodological Evolution, Innovation, and Opportunities," Technological Forecasting and Social Change, Elsevier, vol. 104(C), pages 78-88.
    2. Lisheng Weng & Bao-Jie He & Lina Liu & Chenxi Li & Xin Zhang, 2019. "Sustainability Assessment of Cultural Heritage Tourism: Case Study of Pingyao Ancient City in China," Sustainability, MDPI, vol. 11(5), pages 1-16, March.
    3. Meijering, Jurian V. & Tobi, Hilde, 2016. "The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment," Technological Forecasting and Social Change, Elsevier, vol. 103(C), pages 166-173.
    4. Yildirim, Ercan & AR, Ilker Murat & Dabić, Marina & Baki, Birdogan & Peker, Iskender, 2022. "A multi-stage decision making model for determining a suitable innovation structure using an open innovation approach," Journal of Business Research, Elsevier, vol. 147(C), pages 379-391.
    5. Ali Bastas & Kapila Liyanage, 2018. "ISO 9001 and Supply Chain Integration Principles Based Sustainable Development: A Delphi Study," Sustainability, MDPI, vol. 10(12), pages 1-35, December.
    6. Feil, Alex Sandro & Antunes, Carlos Henggeler & da Silva, Patrícia Pereira & de Castro, Nivalde, 2024. "The critical drivers of the Brazilian electricity sector's transition through 2050: A Delphi study," Utilities Policy, Elsevier, vol. 87(C).
    7. Rabia Bashir & Aymen Sajjad & Shahid Bashir & Khawaja Fawad Latif & Saman Attiq, 2021. "Project Managers’ Competencies in International Development Projects: A Delphi Study," SAGE Open, , vol. 11(4), pages 21582440211, November.
    8. Toppinen, Anne & Röhr, Axel & Pätäri, Satu & Lähtinen, Katja & Toivonen, Ritva, 2018. "The future of wooden multistory construction in the forest bioeconomy – A Delphi study from Finland and Sweden," Journal of Forest Economics, Elsevier, vol. 31(C), pages 3-10.
    9. Klenk, Nicole L. & Hickey, Gordon M., 2011. "A virtual and anonymous, deliberative and analytic participation process for planning and evaluation: The Concept Mapping Policy Delphi," International Journal of Forecasting, Elsevier, vol. 27(1), pages 152-165.
    10. V. M. Jayasooriya & A. W. M. Ng & S. Muthukumaran & B. J. C. Perera, 2019. "Multi Criteria Decision Making in Selecting Stormwater Management Green Infrastructure for Industrial Areas Part 1: Stakeholder Preference Elicitation," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 33(2), pages 627-639, January.
    11. Valentina Tudisca & Adriana Valente, 2016. "(English) Design and implementation of an online Delphi study to develop indicators for evidenceinformed policy making (Italiano) Ideazione e implementazione di uno studio Delphi online per lo svilupp," IRPPS Working Papers 88:2016, National Research Council, Institute for Research on Population and Social Policies.
    12. Peeraer, Jef & Van Petegem, Peter, 2015. "Integration or transformation? Looking in the future of Information and Communication Technology in education in Vietnam," Evaluation and Program Planning, Elsevier, vol. 48(C), pages 47-56.
    13. Klenk, Nicole L. & Hickey, Gordon M., 2011. "A virtual and anonymous, deliberative and analytic participation process for planning and evaluation: The Concept Mapping Policy Delphi," International Journal of Forecasting, Elsevier, vol. 27(1), pages 152-165, January.
    14. Belton, Ian & MacDonald, Alice & Wright, George & Hamlin, Iain, 2019. "Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process," Technological Forecasting and Social Change, Elsevier, vol. 147(C), pages 72-82.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Prommer, Lisa & Tiberius, Victor & Kraus, Sascha, 2020. "Exploring the future of startup leadership development," Journal of Business Venturing Insights, Elsevier, vol. 14(C).
    2. Prianto Budi Saptono & Gustofan Mahmud & Intan Pratiwi & Dwi Purwanto & Ismail Khozen & Muhamad Akbar Aditama & Siti Khodijah & Maria Eurelia Wayan & Rina Yuliastuty Asmara & Ferry Jie, 2023. "Development of Climate-Related Disclosure Indicators for Application in Indonesia: A Delphi Method Study," Sustainability, MDPI, vol. 15(14), pages 1-25, July.
    3. Di Zio, Simone & Bolzan, Mario & Marozzi, Marco, 2021. "Classification of Delphi outputs through robust ranking and fuzzy clustering for Delphi-based scenarios," Technological Forecasting and Social Change, Elsevier, vol. 173(C).
    4. Alyami, Saleh. H. & Rezgui, Yacine & Kwan, Alan, 2013. "Developing sustainable building assessment scheme for Saudi Arabia: Delphi consultation approach," Renewable and Sustainable Energy Reviews, Elsevier, vol. 27(C), pages 43-54.
    5. Georg Aichholzer, 2002. "Das ExpertInnen-Delphi: methodische Grundlagen und Anwendungsfeld ‘Technology Foresight‘ (The Expert Delphi: Methodology and Application in 'Technology Foresight')," ITA manu:scripts 02_01, Institute of Technology Assessment (ITA).
    6. Hayes, Tom, 2007. "Delphi study of the future of marketing of higher education," Journal of Business Research, Elsevier, vol. 60(9), pages 927-931, September.
    7. Haarhaus, Tim & Liening, Andreas, 2020. "Building dynamic capabilities to cope with environmental uncertainty: The role of strategic foresight," Technological Forecasting and Social Change, Elsevier, vol. 155(C).
    8. Ribeiro, Barbara E. & Quintanilla, Miguel A., 2015. "Transitions in biofuel technologies: An appraisal of the social impacts of cellulosic ethanol using the Delphi method," Technological Forecasting and Social Change, Elsevier, vol. 92(C), pages 53-68.
    9. Yeh, Duen-Yian & Cheng, Ching-Hsue, 2015. "Recommendation system for popular tourist attractions in Taiwan using Delphi panel and repertory grid techniques," Tourism Management, Elsevier, vol. 46(C), pages 164-176.
    10. Hsin-Ke Lu & Sung-Chun Tsai & Peng-Chun Lin & Kuo-Chung Chu & Alexander N. Chen, 2020. "Toward a New Real-Time Approach for Group Consensus: A Usability Analysis of Synchronous Delphi System," Group Decision and Negotiation, Springer, vol. 29(2), pages 345-370, April.
    11. Vishal Gaur & Saravanan Kesavan & Ananth Raman & Marshall L. Fisher, 2007. "Estimating Demand Uncertainty Using Judgmental Forecasts," Manufacturing & Service Operations Management, INFORMS, vol. 9(4), pages 480-491, April.
    12. Torres-Sibille, Ana del Carmen & Cloquell-Ballester, Vicente-Agustín & Cloquell-Ballester, Víctor-Andrés & Artacho Ramírez, Miguel Ángel, 2009. "Aesthetic impact assessment of solar power plants: An objective and a subjective approach," Renewable and Sustainable Energy Reviews, Elsevier, vol. 13(5), pages 986-999, June.
    13. Laura Studen & Victor Tiberius, 2020. "Social Media, Quo Vadis? Prospective Development and Implications," Future Internet, MDPI, vol. 12(9), pages 1-22, August.
    14. Volkmar, Gioia & Fischer, Peter M. & Reinecke, Sven, 2022. "Artificial Intelligence and Machine Learning: Exploring drivers, barriers, and future developments in marketing management," Journal of Business Research, Elsevier, vol. 149(C), pages 599-614.
    15. Kawamoto, Carlos Tadao & Wright, James Terence Coulter & Spers, Renata Giovinazzo & de Carvalho, Daniel Estima, 2019. "Can we make use of perception of questions' easiness in Delphi-like studies? Some results from an experiment with an alternative feedback," Technological Forecasting and Social Change, Elsevier, vol. 140(C), pages 296-305.
    16. Meijering, Jurian Vincent & Tobi, Hilde, 2018. "The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial," International Journal of Forecasting, Elsevier, vol. 34(2), pages 216-224.
    17. Paméla Baillette & Bernard Fallery, 2016. "La méthode du Delphi argumentaire, une innovation managériale dans le cadre d'un projet complexe," Post-Print hal-02160359, HAL.
    18. Förster, Bernadette & von der Gracht, Heiko, 2014. "Assessing Delphi panel composition for strategic foresight — A comparison of panels based on company-internal and external participants," Technological Forecasting and Social Change, Elsevier, vol. 84(C), pages 215-229.
    19. Torres Sibille, Ana del Carmen & Cloquell-Ballester, Víctor-Andrés & Cloquell-Ballester, Vicente-Agustín & Darton, Richard, 2009. "Development and validation of a multicriteria indicator for the assessment of objective aesthetic impact of wind farms," Renewable and Sustainable Energy Reviews, Elsevier, vol. 13(1), pages 40-66, January.
    20. Ebrahim Aghazadeh & Hasan Yildirim & Murat Kuruoglu, 2022. "A Hybrid Fuzzy MCDM Methodology for Optimal Structural System Selection Compatible with Sustainable Materials in Mass-Housing Projects," Sustainability, MDPI, vol. 14(20), pages 1-22, October.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:epplan:v:31:y:2008:i:2:p:191-198. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/evalprogplan .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.