IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v51y2005i4p655-661.html
   My bibliography  Save this article

Optimal Allocation of Proposals to Reviewers to Facilitate Effective Ranking

Author

Listed:
  • Wade D. Cook

    (Schulich School of Business, York University, Toronto, Ontario M3J 1P3, Canada)

  • Boaz Golany

    (Faculty of Industrial Engineering and Management, Technion---Israel Institute of Technology, Haifa 32000, Israel)

  • Moshe Kress

    (Center for Military Analyses, POB 2250, Haifa 31021, Israel, and Operations Research Department, Naval Postgraduate School, Monterey, California 93943)

  • Michal Penn

    (Faculty of Industrial Engineering and Management, Technion---Israel Institute of Technology, Haifa 32000, Israel)

  • Tal Raviv

    (Faculty of Industrial Engineering and Management, Technion---Israel Institute of Technology, Haifa 32000, Israel)

Abstract

Peer review of research proposals and articles is an essential element in research and development processes worldwide. Here we consider a problem that, to the best of our knowledge, has not been addressed until now: how to assign subsets of proposals to reviewers in scenarios where the reviewers supply their evaluations through ordinal ranking. The solution approach we propose for this assignment problem maximizes the number of proposal pairs that will be evaluated by one or more reviewers. This new approach should facilitate meaningful aggregation of partial rankings of subsets of proposals by multiple reviewers into a consensus ranking. We offer two ways to implement the approach: an integer-programming set-covering model and a heuristic procedure. The effectiveness and efficiency of the two models are tested through an extensive simulation experiment.

Suggested Citation

  • Wade D. Cook & Boaz Golany & Moshe Kress & Michal Penn & Tal Raviv, 2005. "Optimal Allocation of Proposals to Reviewers to Facilitate Effective Ranking," Management Science, INFORMS, vol. 51(4), pages 655-661, April.
    • Handle: RePEc:inm:ormnsc:v:51:y:2005:i:4:p:655-661
    DOI: 10.1287/mnsc.1040.0290
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mnsc.1040.0290
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mnsc.1040.0290?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Wade D. Cook & Lawrence M. Seiford, 1978. "Priority Ranking and Consensus Formation," Management Science, INFORMS, vol. 24(16), pages 1721-1732, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jian Jin & Baozhuang Niu & Ping Ji & Qian Geng, 2020. "An integer linear programming model of reviewer assignment with research interest considerations," Annals of Operations Research, Springer, vol. 291(1), pages 409-433, August.
    2. Xiaoyu Liu & Xuefeng Wang & Donghua Zhu, 2022. "Reviewer recommendation method for scientific research proposals: a case for NSFC," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3343-3366, June.
    3. Yeşilçimen, Ali & Yıldırım, E. Alper, 2019. "An alternative polynomial-sized formulation and an optimization based heuristic for the reviewer assignment problem," European Journal of Operational Research, Elsevier, vol. 276(2), pages 436-450.
    4. Thomas Feliciani & Junwen Luo & Lai Ma & Pablo Lucas & Flaminio Squazzoni & Ana Marušić & Kalpana Shankar, 2019. "A scoping review of simulation models of peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 555-594, October.
    5. Linton, Jonathan D., 2016. "Improving the Peer review process: Capturing more information and enabling high-risk/high-return research," Research Policy, Elsevier, vol. 45(9), pages 1936-1938.
    6. Fernandez Martinez, Roberto & Lostado Lorza, Ruben & Santos Delgado, Ana Alexandra & Piedra, Nelson, 2021. "Use of classification trees and rule-based models to optimize the funding assignment to research projects: A case study of UTPL," Journal of Informetrics, Elsevier, vol. 15(1).
    7. Song, Lianlian & Fu, Yelin & Zhou, Peng & Lai, Kin Keung, 2017. "Measuring national energy performance via Energy Trilemma Index: A Stochastic Multicriteria Acceptability Analysis," Energy Economics, Elsevier, vol. 66(C), pages 313-319.
    8. Katarína Cechlárová & Tamás Fleiner & Eva Potpinková, 2014. "Assigning evaluators to research grant applications: the case of Slovak Research and Development Agency," Scientometrics, Springer;Akadémiai Kiadó, vol. 99(2), pages 495-506, May.
    9. Dorit S. Hochbaum & Asaf Levin, 2006. "Methodologies and Algorithms for Group-Rankings Decision," Management Science, INFORMS, vol. 52(9), pages 1394-1408, September.
    10. Vladimír Bureš & Jiří Cabal & Pavel Čech & Karel Mls & Daniela Ponce, 2020. "The Influence of Criteria Selection Method on Consistency of Pairwise Comparison," Mathematics, MDPI, vol. 8(12), pages 1-13, December.
    11. Ma, Li-Ching, 2010. "Visualizing preferences on spheres for group decisions based on multiplicative preference relations," European Journal of Operational Research, Elsevier, vol. 203(1), pages 176-184, May.
    12. Cheng, Li-Chen & Chen, Yen-Liang & Chiang, Yu-Chia, 2016. "Identifying conflict patterns to reach a consensus – A novel group decision approach," European Journal of Operational Research, Elsevier, vol. 254(2), pages 622-631.
    13. Huang, Tony Cheng-Kui, 2013. "A novel group ranking model for revealing sequence and quantity knowledge," European Journal of Operational Research, Elsevier, vol. 231(3), pages 654-666.
    14. Chen, Yen-Liang & Cheng, Li-Chen, 2009. "Mining maximum consensus sequences from group ranking data," European Journal of Operational Research, Elsevier, vol. 198(1), pages 241-251, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jabeur, Khaled & Martel, Jean-Marc, 2007. "A collective choice method based on individual preferences relational systems (p.r.s.)," European Journal of Operational Research, Elsevier, vol. 177(3), pages 1549-1565, March.
    2. Yeawon Yoo & Adolfo R. Escobedo, 2021. "A New Binary Programming Formulation and Social Choice Property for Kemeny Rank Aggregation," Decision Analysis, INFORMS, vol. 18(4), pages 296-320, December.
    3. Andrea Aveni & Ludovico Crippa & Giulio Principi, 2024. "On the Weighted Top-Difference Distance: Axioms, Aggregation, and Approximation," Papers 2403.15198, arXiv.org, revised Mar 2024.
    4. Spyridakos, A. & Siskos, Y. & Yannacopoulos, D. & Skouris, A., 2001. "Multicriteria job evaluation for large organizations," European Journal of Operational Research, Elsevier, vol. 130(2), pages 375-387, April.
    5. Fujun Hou, 2015. "A Consensus Gap Indicator and Its Application to Group Decision Making," Group Decision and Negotiation, Springer, vol. 24(3), pages 415-428, May.
    6. J González-Pachón & C Romero, 2006. "An analytical framework for aggregating multiattribute utility functions," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 57(10), pages 1241-1247, October.
    7. Wade D. Cook & Tal Raviv & Alan J. Richardson, 2010. "Aggregating Incomplete Lists of Journal Rankings: An Application to Academic Accounting Journals," Accounting Perspectives, John Wiley & Sons, vol. 9(3), pages 217-235, September.
    8. Jabeur, Khaled & Martel, Jean-Marc, 2007. "An ordinal sorting method for group decision-making," European Journal of Operational Research, Elsevier, vol. 180(3), pages 1272-1289, August.
    9. Zachary F. Lansdowne, 1996. "Ordinal ranking methods for multicriterion decision making," Naval Research Logistics (NRL), John Wiley & Sons, vol. 43(5), pages 613-627, August.
    10. Jorge Alcalde-Unzu & Marc Vorsatz, 2008. "The Measurement of Consensus: An Axiomatic Analysis," Working Papers 2008-28, FEDEA.
    11. Espin, Rafael & Fernandez, Eduardo & Mazcorro, Gustavo & Lecich, Maria Ines, 2007. "A fuzzy approach to cooperative n-person games," European Journal of Operational Research, Elsevier, vol. 176(3), pages 1735-1751, February.
    12. Kelin Luo & Yinfeng Xu & Bowen Zhang & Huili Zhang, 2018. "Creating an acceptable consensus ranking for group decision making," Journal of Combinatorial Optimization, Springer, vol. 36(1), pages 307-328, July.
    13. Athanasios Spyridakos & Denis Yannacopoulos, 2015. "Incorporating collective functions to multicriteria disaggregation–aggregation approaches for small group decision making," Annals of Operations Research, Springer, vol. 227(1), pages 119-136, April.
    14. B. Ahn & S. Choi, 2012. "Aggregation of ordinal data using ordered weighted averaging operator weights," Annals of Operations Research, Springer, vol. 201(1), pages 1-16, December.
    15. Aleksandar Hatzivelkos, 2018. "Borda and Plurality Comparison with Regard to Compromise as a Sorites Paradox," Interdisciplinary Description of Complex Systems - scientific journal, Croatian Interdisciplinary Society Provider Homepage: http://indecs.eu, vol. 16(3-B), pages 465-484.
    16. Fujun Hou, 2018. "Mutual Conversion Between Preference Maps And Cook-Seiford Vectors," Papers 1812.03566, arXiv.org.
    17. Hanna Bury & Dariusz Wagner, 2009. "Group judgement with ties. A position-based approach," Operations Research and Decisions, Wroclaw University of Technology, Institute of Organization and Management, vol. 4, pages 9-26.
    18. Cook, Wade D. & Kress, Moshe & Seiford, Lawrence M., 1997. "A general framework for distance-based consensus in ordinal ranking models," European Journal of Operational Research, Elsevier, vol. 96(2), pages 392-397, January.
    19. G. Laffond & J. Lainé, 2013. "Unanimity and the Anscombe’s paradox," TOP: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 21(3), pages 590-611, October.
    20. Hanna Bury & Dariusz Wagner, 2009. "Group judgment with ties. A position-based approach," Operations Research and Decisions, Wroclaw University of Science and Technology, Faculty of Management, vol. 19(4), pages 7-26.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:51:y:2005:i:4:p:655-661. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.