IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v105y2015i3d10.1007_s11192-015-1770-8.html
   My bibliography  Save this article

Research project evaluation and selection: an evidential reasoning rule-based method for aggregating peer review information with reliabilities

Author

Listed:
  • Wei-dong Zhu

    (Hefei University of Technology)

  • Fang Liu

    (Hefei University of Technology
    The University of Manchester)

  • Yu-wang Chen

    (The University of Manchester)

  • Jian-bo Yang

    (Hefei University of Technology
    The University of Manchester)

  • Dong-ling Xu

    (Hefei University of Technology
    The University of Manchester)

  • Dong-peng Wang

    (Hefei University of Technology)

Abstract

Research project evaluation and selection is mainly concerned with evaluating a number of research projects and then choosing some of them for implementation. It involves a complex multiple-experts multiple-criteria decision making process. Thus this paper presents an effective method for evaluating and selecting research projects by using the recently-developed evidential reasoning (ER) rule. The proposed ER rule based evaluation and selection method mainly includes (1) using belief structures to represent peer review information provided by multiple experts, (2) employing a confusion matrix for generating experts’ reliabilities, (3) implementing utility based information transformation to handle qualitative evaluation criteria with different evaluation grades, and (4) aggregating multiple experts’ evaluation information on multiple criteria using the ER rule. An experimental study on the evaluation and selection of research proposals submitted to the National Science Foundation of China demonstrates the applicability and effectiveness of the proposed method. The results show that (1) the ER rule based method can provide consistent and informative support to make informed decisions, and (2) the reliabilities of the review information provided by different experts should be taken into account in a rational research project evaluation and selection process, as they have a significant influence to the selection of eligible projects for panel review.

Suggested Citation

  • Wei-dong Zhu & Fang Liu & Yu-wang Chen & Jian-bo Yang & Dong-ling Xu & Dong-peng Wang, 2015. "Research project evaluation and selection: an evidential reasoning rule-based method for aggregating peer review information with reliabilities," Scientometrics, Springer;Akadémiai Kiadó, vol. 105(3), pages 1469-1490, December.
    • Handle: RePEc:spr:scient:v:105:y:2015:i:3:d:10.1007_s11192-015-1770-8
    DOI: 10.1007/s11192-015-1770-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-015-1770-8
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-015-1770-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Yang, Jian-Bo, 2001. "Rule and utility based evidential reasoning approach for multiattribute decision analysis under uncertainties," European Journal of Operational Research, Elsevier, vol. 131(1), pages 31-61, May.
    2. Olsson, Nils O.E. & Krane, Hans Petter & Rolstadås, Asbjørn & Veiseth, Mads, 2010. "Influence of reference points in ex post evaluations of rail infrastructure projects," Transport Policy, Elsevier, vol. 17(4), pages 251-258, August.
    3. Oral, Muhittin & Kettani, Ossama & Cinar, Unver, 2001. "Project evaluation and selection in a network of collaboration: A consensual disaggregation multi-criterion approach," European Journal of Operational Research, Elsevier, vol. 130(2), pages 332-346, April.
    4. Wang, Juite & Hwang, W.-L., 2007. "A fuzzy set approach for R&D portfolio selection using a real options valuation model," Omega, Elsevier, vol. 35(3), pages 247-257, June.
    5. Primož Južnič & Stojan Pečlin & Matjaž Žaucer & Tilen Mandelj & Miro Pušnik & Franci Demšar, 2010. "Scientometric indicators: peer-review, bibliometric methods and conflict of interests," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 429-441, November.
    6. Dong-Ling Xu, 2012. "An introduction and survey of the evidential reasoning approach for multiple criteria decision analysis," Annals of Operations Research, Springer, vol. 195(1), pages 163-187, May.
    7. Upali W. Jayasinghe & Herbert W. Marsh & Nigel Bond, 2006. "A new reader trial approach to peer review in funding research grants: An Australian experiment," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(3), pages 591-606, December.
    8. Huang, Chi-Cheng & Chu, Pin-Yu & Chiang, Yu-Hsiu, 2008. "A fuzzy AHP application in government-sponsored R&D project selection," Omega, Elsevier, vol. 36(6), pages 1038-1052, December.
    9. Solak, Senay & Clarke, John-Paul B. & Johnson, Ellis L. & Barnes, Earl R., 2010. "Optimization of R&D project portfolios under endogenous uncertainty," European Journal of Operational Research, Elsevier, vol. 207(1), pages 420-433, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Fernandez Martinez, Roberto & Lostado Lorza, Ruben & Santos Delgado, Ana Alexandra & Piedra, Nelson, 2021. "Use of classification trees and rule-based models to optimize the funding assignment to research projects: A case study of UTPL," Journal of Informetrics, Elsevier, vol. 15(1).
    2. Weidong Zhu & Shaorong Li & Hongtao Zhang & Tianjiao Zhang & Zhimin Li, 2022. "Evaluation of scientific research projects on the basis of evidential reasoning approach under the perspective of expert reliability," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 275-298, January.
    3. Li, Heyang & Wu, Meijun & Wang, Yougui & Zeng, An, 2022. "Bibliographic coupling networks reveal the advantage of diversification in scientific projects," Journal of Informetrics, Elsevier, vol. 16(3).
    4. Simon Hirzel & Tim Hettesheimer & Peter Viebahn & Manfred Fischedick, 2018. "A Decision Support System for Public Funding of Experimental Development in Energy Research," Energies, MDPI, vol. 11(6), pages 1-18, May.
    5. Zhu, Weidong & Zhang, Tianjiao & Wu, Yong & Li, Shaorong & Li, Zhimin, 2022. "Research on optimization of an enterprise financial risk early warning method based on the DS-RF model," International Review of Financial Analysis, Elsevier, vol. 81(C).
    6. Fang Liu & Wei-dong Zhu & Yu-wang Chen & Dong-ling Xu & Jian-bo Yang, 2017. "Evaluation, ranking and selection of R&D projects by multiple experts: an evidential reasoning rule based approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1501-1519, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Fang Liu & Wei-dong Zhu & Yu-wang Chen & Dong-ling Xu & Jian-bo Yang, 2017. "Evaluation, ranking and selection of R&D projects by multiple experts: an evidential reasoning rule based approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1501-1519, June.
    2. Milford, James & Henrion, Max & Hunter, Chad & Newes, Emily & Hughes, Caroline & Baldwin, Samuel F., 2022. "Energy sector portfolio analysis with uncertainty," Applied Energy, Elsevier, vol. 306(PA).
    3. Mavrotas, George & Makryvelios, Evangelos, 2021. "Combining multiple criteria analysis, mathematical programming and Monte Carlo simulation to tackle uncertainty in Research and Development project portfolio selection: A case study from Greece," European Journal of Operational Research, Elsevier, vol. 291(2), pages 794-806.
    4. Xiaojiao Qiao & Dan Shi, 2019. "Risk Analysis of Emergency Based on Fuzzy Evidential Reasoning," Complexity, Hindawi, vol. 2019, pages 1-10, November.
    5. Pérez, Fátima & Gómez, Trinidad & Caballero, Rafael & Liern, Vicente, 2018. "Project portfolio selection and planning with fuzzy constraints," Technological Forecasting and Social Change, Elsevier, vol. 131(C), pages 117-129.
    6. Kajal Chatterjee & Sheikh Ahmed Hossain & Samarjit Kar, 2018. "Prioritization of project proposals in portfolio management using fuzzy AHP," OPSEARCH, Springer;Operational Research Society of India, vol. 55(2), pages 478-501, June.
    7. Martzoukos, Spiros H. & Zacharias, Eleftherios, 2013. "Real option games with R&D and learning spillovers," Omega, Elsevier, vol. 41(2), pages 236-249.
    8. Wen-Tao Guo & Van-Nam Huynh & Songsak Sriboonchitta, 2017. "A proportional linguistic distribution based model for multiple attribute decision making under linguistic uncertainty," Annals of Operations Research, Springer, vol. 256(2), pages 305-328, September.
    9. Guilan Kong & Lili Jiang & Xiaofeng Yin & Tianbing Wang & Dong-Ling Xu & Jian-Bo Yang & Yonghua Hu, 2018. "Combining principal component analysis and the evidential reasoning approach for healthcare quality assessment," Annals of Operations Research, Springer, vol. 271(2), pages 679-699, December.
    10. Liu, Jiapeng & Liao, Xiuwu & Yang, Jian-bo, 2015. "A group decision-making approach based on evidential reasoning for multiple criteria sorting problem with uncertainty," European Journal of Operational Research, Elsevier, vol. 246(3), pages 858-873.
    11. Durbach, Ian N. & Stewart, Theodor J., 2012. "Modeling uncertainty in multi-criteria decision analysis," European Journal of Operational Research, Elsevier, vol. 223(1), pages 1-14.
    12. Faraz Salehi & S. Mohammad J. Mirzapour Al-E-Hashem & S. Mohammad Moattar Husseini & S. Hassan Ghodsypour, 2023. "A bi-level multi-follower optimization model for R&D project portfolio: an application to a pharmaceutical holding company," Annals of Operations Research, Springer, vol. 323(1), pages 331-360, April.
    13. Fu, Chao & Yang, Jian-Bo & Yang, Shan-Lin, 2015. "A group evidential reasoning approach based on expert reliability," European Journal of Operational Research, Elsevier, vol. 246(3), pages 886-893.
    14. Chao Fu & Dong-Ling Xu, 2016. "Determining attribute weights to improve solution reliability and its application to selecting leading industries," Annals of Operations Research, Springer, vol. 245(1), pages 401-426, October.
    15. Sevastjanov, Pavel & Dymova, Ludmila, 2009. "Stock screening with use of multiple criteria decision making and optimization," Omega, Elsevier, vol. 37(3), pages 659-671, June.
    16. Shahryar Monghasemi & Mohammad Reza Nikoo & Mohammad Ali Khaksar Fasaee & Jan Adamowski, 2017. "A Hybrid of Genetic Algorithm and Evidential Reasoning for Optimal Design of Project Scheduling: A Systematic Negotiation Framework for Multiple Decision-Makers," International Journal of Information Technology & Decision Making (IJITDM), World Scientific Publishing Co. Pte. Ltd., vol. 16(02), pages 389-420, March.
    17. Kao, Chiang & Pao, Hwei-Lan, 2012. "Predicting project approvals: A case of grants from the National Science Council of Taiwan," Omega, Elsevier, vol. 40(1), pages 89-95, January.
    18. Oral, Muhittin & Oukil, Amar & Malouin, Jean-Louis & Kettani, Ossama, 2014. "The appreciative democratic voice of DEA: A case of faculty academic performance evaluation," Socio-Economic Planning Sciences, Elsevier, vol. 48(1), pages 20-28.
    19. Salimi, Negin & Rezaei, Jafar, 2018. "Evaluating firms’ R&D performance using best worst method," Evaluation and Program Planning, Elsevier, vol. 66(C), pages 147-155.
    20. Wu, Xingli & Liao, Huchang, 2021. "Modeling personalized cognition of customers in online shopping," Omega, Elsevier, vol. 104(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:105:y:2015:i:3:d:10.1007_s11192-015-1770-8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.