IDEAS home Printed from https://ideas.repec.org/a/eee/ejores/v245y2015i3p789-796.html
   My bibliography  Save this article

Measurement issues in the evaluation of projects in a project portfolio

Author

Listed:
  • Morton, Alec

Abstract

A common problem arising in many domains is how value the benefits of projects in a project portfolio. Recently, there has been some attention given to a decision analysis practice whereby analysts define a value function on some criterion by setting 0 as the value of the worst project. In particular, Clemen and Smith have argued this practice is not sound as it gives different results from the case where projects are “priced out”, and makes a strong implicit assumption about the value of not doing a project. In this paper we underscore the criticism of this way of using value functions by showing it can lead to a rank reversal. We provide a measurement theoretic account of the phenomenon, showing that the problem arises from using evaluating projects on an interval scale (such as a value scale) whereas to guard against such rank reversals, benefits must be measured on at least a ratio scale. Seen from this perspective, we discuss how the solution proposed by Clemen and Smith, of reformulating the underlying optimisation problem to allow for explicitly non-zero values, addresses the issue, and explore in what sense it may be open to similar problems. In closing we discuss what lessons from practice may be drawn from this analysis, focussing on settings where the Clemen and Smith proposal may not be the most natural way of modelling.

Suggested Citation

  • Morton, Alec, 2015. "Measurement issues in the evaluation of projects in a project portfolio," European Journal of Operational Research, Elsevier, vol. 245(3), pages 789-796.
  • Handle: RePEc:eee:ejores:v:245:y:2015:i:3:p:789-796
    DOI: 10.1016/j.ejor.2015.03.037
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0377221715002593
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.ejor.2015.03.037?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. James S. Dyer, 1990. "Remarks on the Analytic Hierarchy Process," Management Science, INFORMS, vol. 36(3), pages 249-258, March.
    2. Bana e Costa, Carlos A. & Ensslin, Leonardo & Correa, Emerson C. & Vansnick, Jean-Claude, 1999. "Decision Support Systems in action: Integrated application in a multicriteria decision aid process," European Journal of Operational Research, Elsevier, vol. 113(2), pages 315-335, March.
    3. de Almeida, Adiel Teixeira & Vetschera, Rudolf, 2012. "A note on scale transformations in the PROMETHEE V method," European Journal of Operational Research, Elsevier, vol. 219(1), pages 198-200.
    4. Gregory S. Parnell & G. Edgar Bennett & Joseph A. Engelbrecht & Richard Szafranski, 2002. "Improving Resource Allocation Within the National Reconnaissance Office," Interfaces, INFORMS, vol. 32(3), pages 77-90, June.
    5. Sen, Amartya K, 1977. "On Weights and Measures: Informational Constraints in Social Welfare Analysis," Econometrica, Econometric Society, vol. 45(7), pages 1539-1572, October.
    6. James S. Dyer, 1990. "A Clarification of "Remarks on the Analytic Hierarchy Process"," Management Science, INFORMS, vol. 36(3), pages 274-275, March.
    7. Robert T. Clemen & James E. Smith, 2009. "On the Choice of Baselines in Multiattribute Portfolio Analysis: A Cautionary Note," Decision Analysis, INFORMS, vol. 6(4), pages 256-262, December.
    8. Liesiö, Juuso & Punkka, Antti, 2014. "Baseline value specification and sensitivity analysis in multiattribute project portfolio selection," European Journal of Operational Research, Elsevier, vol. 237(3), pages 946-956.
    9. Belton, Valerie & Gear, Tony, 1983. "On a short-coming of Saaty's method of analytic hierarchies," Omega, Elsevier, vol. 11(3), pages 228-230.
    10. Pekka Mild & Ahti Salo, 2009. "Combining a Multiattribute Value Function with an Optimization Model: An Application to Dynamic Resource Allocation for Infrastructure Maintenance," Decision Analysis, INFORMS, vol. 6(3), pages 139-152, September.
    11. Lawrence Phillips & Carlos Bana e Costa, 2007. "Transparent prioritisation, budgeting and resource allocation with multi-criteria decision analysis and decision conferencing," Annals of Operations Research, Springer, vol. 154(1), pages 51-68, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Andrea C. Hupman & Jay Simon, 2023. "The Legacy of Peter Fishburn: Foundational Work and Lasting Impact," Decision Analysis, INFORMS, vol. 20(1), pages 1-15, March.
    2. Liesiö, Juuso & Kee, Taeyoung & Malo, Pekka, 2024. "Modeling project interactions in multiattribute portfolio decision analysis: Axiomatic foundations and practical implications," European Journal of Operational Research, Elsevier, vol. 316(3), pages 988-1000.
    3. Liesiö, Juuso & Salo, Ahti & Keisler, Jeffrey M. & Morton, Alec, 2021. "Portfolio decision analysis: Recent developments and future prospects," European Journal of Operational Research, Elsevier, vol. 293(3), pages 811-825.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rodrigues, Teresa C. & Montibeller, Gilberto & Oliveira, Mónica D. & Bana e Costa, Carlos A., 2017. "Modelling multicriteria value interactions with Reasoning Maps," European Journal of Operational Research, Elsevier, vol. 258(3), pages 1054-1071.
    2. Liesiö, Juuso & Kee, Taeyoung & Malo, Pekka, 2024. "Modeling project interactions in multiattribute portfolio decision analysis: Axiomatic foundations and practical implications," European Journal of Operational Research, Elsevier, vol. 316(3), pages 988-1000.
    3. Liesiö, Juuso & Salo, Ahti & Keisler, Jeffrey M. & Morton, Alec, 2021. "Portfolio decision analysis: Recent developments and future prospects," European Journal of Operational Research, Elsevier, vol. 293(3), pages 811-825.
    4. Mikhailov, L., 2004. "A fuzzy approach to deriving priorities from interval pairwise comparison judgements," European Journal of Operational Research, Elsevier, vol. 159(3), pages 687-704, December.
    5. Lai, S-K., 1995. "A preference-based interpretation of AHP," Omega, Elsevier, vol. 23(4), pages 453-462, August.
    6. Zahir, Sajjad, 1999. "Geometry of decision making and the vector space formulation of the analytic hierarchy process," European Journal of Operational Research, Elsevier, vol. 112(2), pages 373-396, January.
    7. Leung, Lawrence C. & Cao, Dong, 2001. "On the efficacy of modeling multi-attribute decision problems using AHP and Sinarchy," European Journal of Operational Research, Elsevier, vol. 132(1), pages 39-49, July.
    8. Van den Honert, R. C., 1998. "Stochastic group preference modelling in the multiplicative AHP: A model of group consensus," European Journal of Operational Research, Elsevier, vol. 110(1), pages 99-111, October.
    9. Majumdar, Abhijit & Tiwari, Manoj Kumar & Agarwal, Aastha & Prajapat, Kanika, 2021. "A new case of rank reversal in analytic hierarchy process due to aggregation of cost and benefit criteria," Operations Research Perspectives, Elsevier, vol. 8(C).
    10. Liu, Qizhi, 2022. "Identifying and correcting the defects of the Saaty analytic hierarchy/network process: A comparative study of the Saaty analytic hierarchy/network process and the Markov chain-based analytic network ," Operations Research Perspectives, Elsevier, vol. 9(C).
    11. Kumar, N. Vinod & Ganesh, L. S., 1996. "A simulation-based evaluation of the approximate and the exact eigenvector methods employed in AHP," European Journal of Operational Research, Elsevier, vol. 95(3), pages 656-662, December.
    12. Wedley, William C. & Choo, Eng Ung & Schoner, Bertram, 2001. "Magnitude adjustment for AHP benefit/cost ratios," European Journal of Operational Research, Elsevier, vol. 133(2), pages 342-351, January.
    13. Juuso Liesiö, 2014. "Measurable Multiattribute Value Functions for Portfolio Decision Analysis," Decision Analysis, INFORMS, vol. 11(1), pages 1-20, March.
    14. Belton, Valerie & Goodwin, Paul, 1996. "Remarks on the application of the analytic hierarchy process to judgmental forecasting," International Journal of Forecasting, Elsevier, vol. 12(1), pages 155-161, March.
    15. Joaquín Pérez, José L. Jimeno, Ethel Mokotoff, 2001. "Another potential strong shortcoming of AHP," Doctorado en Economía- documentos de trabajo 8/02, Programa de doctorado en Economía. Universidad de Alcalá., revised 01 Jun 2002.
    16. Jain, Bharat A. & Nag, Barin N., 1996. "A decision-support model for investment decisions in new ventures," European Journal of Operational Research, Elsevier, vol. 90(3), pages 473-486, May.
    17. Siraj, Sajid & Mikhailov, Ludmil & Keane, John A., 2015. "Contribution of individual judgments toward inconsistency in pairwise comparisons," European Journal of Operational Research, Elsevier, vol. 242(2), pages 557-567.
    18. Mónica D. Oliveira & Inês Mataloto & Panos Kanavos, 2019. "Multi-criteria decision analysis for health technology assessment: addressing methodological challenges to improve the state of the art," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 20(6), pages 891-918, August.
    19. Millet, Ido & Saaty, Thomas L., 2000. "On the relativity of relative measures - accommodating both rank preservation and rank reversals in the AHP," European Journal of Operational Research, Elsevier, vol. 121(1), pages 205-212, February.
    20. Dong, Yucheng & Xu, Yinfeng & Li, Hongyi & Dai, Min, 2008. "A comparative study of the numerical scales and the prioritization methods in AHP," European Journal of Operational Research, Elsevier, vol. 186(1), pages 229-242, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ejores:v:245:y:2015:i:3:p:789-796. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/eor .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.