IDEAS home Printed from https://ideas.repec.org/a/spr/soinre/v164y2022i3d10.1007_s11205-022-02989-4.html
   My bibliography  Save this article

Consensus-Based Sub-Indicator Weighting Approach: Constructing Composite Indicators Compatible with Expert Opinion

Author

Listed:
  • Matheus Pereira Libório

    (Pontifical Catholic University of Minas Gerais)

  • Lívia Maria Leite Silva

    (Federal University of Minas Gerais)

  • Petr Iakovlevitch Ekel

    (Pontifical Catholic University of Minas Gerais
    Federal University of Minas Gerais)

  • Letícia Ribeiro Figueiredo

    (Pontifical Catholic University of Minas Gerais)

  • Patrícia Bernardes

    (Pontifical Catholic University of Minas Gerais)

Abstract

The weighting of sub-indicators is a relevant problem in the composite indicators literature and impacts several fields of science. None of the existing weighting approaches, Equal-Weights, Data-Driven, and Participatory, is exempt from criticism. Specifically, weights obtained by the Participatory approach are associated with two frequent problems: assessments errors and international comparisons. Mainly, the assessments errors occur when the number of sub-indicators to be assessed is high, as it requires more cognitive effort from decision-makers. The problem of international comparison occurs because the weights of the sub-indicators reflect the specific characteristics of the countries and are not necessarily the same. Selecting experts who know the countries involved increases the impact of expert assessments on the results as the number of experts qualified to carry out the assessments decreases. These are common problems in composite indicators such as the Global Innovation Index, Multidimensional Poverty Index, Sustainable Development Goals Index, and Ease of Doing Business Index. This research presents solutions to these two problems. First, experts ordered seventeen sub-indicators by importance, decreasing the cognitive effort of the experts and the assessment errors that occur when the sub-indicators are assessed directly or compared in pairs. Second, the order of importance is converted into weights through six assessment format transformation functions. The deviant assessments are identified by the Concordance Correlation Coefficient and Intraclass Correlation Coefficient and excluded. Sub-indicators are weighted with a twenty-nine percent higher consensus degree, allowing the construction of composite indicators compatible with collective opinion.

Suggested Citation

  • Matheus Pereira Libório & Lívia Maria Leite Silva & Petr Iakovlevitch Ekel & Letícia Ribeiro Figueiredo & Patrícia Bernardes, 2022. "Consensus-Based Sub-Indicator Weighting Approach: Constructing Composite Indicators Compatible with Expert Opinion," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 164(3), pages 1073-1099, December.
    • Handle: RePEc:spr:soinre:v:164:y:2022:i:3:d:10.1007_s11205-022-02989-4
    DOI: 10.1007/s11205-022-02989-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11205-022-02989-4
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11205-022-02989-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Paolo Paruolo & Michaela Saisana & Andrea Saltelli, 2013. "Ratings and rankings: voodoo or science?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 176(3), pages 609-634, June.
    2. Ana Garcia-Bernabeu & José Manuel Cabello & Francisco Ruiz, 2020. "A Multi-Criteria Reference Point Based Approach for Assessing Regional Innovation Performance in Spain," Mathematics, MDPI, vol. 8(5), pages 1-21, May.
    3. Marta Kuc-Czarnecka & Samuele Lo Piano & Andrea Saltelli, 2020. "Quantitative Storytelling in the Making of a Composite Indicator," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 149(3), pages 775-802, June.
    4. Giuseppe Munda, 2005. "“Measuring Sustainability”: A Multi-Criterion Framework," Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development, Springer, vol. 7(1), pages 117-134, January.
    5. Marco Marozzi, 2016. "Construction, Robustness Assessment and Application of an Index of Perceived Level of Socio-Economic Threat from Immigrants: A Study of 47 European Countries and Regions," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 128(1), pages 413-437, August.
    6. Simeon Djankov & Rafael La Porta & Florencio Lopez-de-Silane & Andrei Shleifer, 2002. "Courts: the Lex Mundi Project," NBER Working Papers 8890, National Bureau of Economic Research, Inc.
    7. Pinheiro-Alves, Ricardo & Zambujal-Oliveira, João, 2012. "The Ease of Doing Business Index as a tool for investment location decisions," Economics Letters, Elsevier, vol. 117(1), pages 66-70.
    8. Simeon Djankov & Tim Ganser & Caralee McLiesh & Rita Ramalho & Andrei Shleifer, 2010. "The Effect of Corporate Taxes on Investment and Entrepreneurship," American Economic Journal: Macroeconomics, American Economic Association, vol. 2(3), pages 31-64, July.
    9. Chen, Chia-Cheng & Barnhart, Huiman X., 2008. "Comparison of ICC and CCC for assessing agreement for data without and with replications," Computational Statistics & Data Analysis, Elsevier, vol. 53(2), pages 554-564, December.
    10. Rogge, Nicky & Archer, Geoffrey, 2021. "Measuring and analyzing country change in establishing ease of doing business using a revised version of World Bank's ease of doing business index," European Journal of Operational Research, Elsevier, vol. 290(1), pages 373-385.
    11. D’Inverno, Giovanna & De Witte, Kristof, 2020. "Service level provision in municipalities: A flexible directional distance composite indicator," European Journal of Operational Research, Elsevier, vol. 286(3), pages 1129-1141.
    12. Marco Marozzi, 2021. "Perceived Justifiability Towards Morally Debatable Behaviors Across Europe," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 153(2), pages 759-778, January.
    13. Charnes, A. & Cooper, W. W. & Rhodes, E., 1978. "Measuring the efficiency of decision making units," European Journal of Operational Research, Elsevier, vol. 2(6), pages 429-444, November.
    14. Mehmet Pinar, 2019. "Multidimensional Well-Being and Inequality Across the European Regions with Alternative Interactions Between the Well-Being Dimensions," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 144(1), pages 31-72, July.
    15. Michael Breen & Robert Gillanders, 2012. "Corruption, institutions and regulation," Economics of Governance, Springer, vol. 13(3), pages 263-285, September.
    16. Issaka Dialga & Le Thi Hang Giang, 2017. "Highlighting Methodological Limitations in the Steps of Composite Indicators Construction," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 131(2), pages 441-465, March.
    17. Amélie Charles & Olivier Darné & Fabien Tripier, 2018. "Uncertainty and the Macroeconomy: Evidence from an uncertainty composite indicator," Post-Print hal-01757042, HAL.
    18. Simeon Djankov & Oliver Hart & Caralee McLiesh & Andrei Shleifer, 2008. "Debt Enforcement around the World," Journal of Political Economy, University of Chicago Press, vol. 116(6), pages 1105-1149, December.
    19. Marco Marozzi, 2015. "Measuring Trust in European Public Institutions," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 123(3), pages 879-895, September.
    20. Pasquale De Muro & Matteo Mazziotta & Adriano Pareto, 2011. "Composite Indices of Development and Poverty: An Application to MDGs," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 104(1), pages 1-18, October.
    21. Alkire, Sabina & Santos, Maria Emma, 2014. "Measuring Acute Poverty in the Developing World: Robustness and Scope of the Multidimensional Poverty Index," World Development, Elsevier, vol. 59(C), pages 251-274.
    22. Matteo Mazziotta & Adriano Pareto, 2018. "Measuring Well-Being Over Time: The Adjusted Mazziotta–Pareto Index Versus Other Non-compensatory Indices," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 136(3), pages 967-976, April.
    23. Amélie Charles & Olivier Darné & Fabien Tripier, 2018. "Uncertainty and the macroeconomy: evidence from an uncertainty composite indicator," Applied Economics, Taylor & Francis Journals, vol. 50(10), pages 1093-1107, February.
    24. Niels Lind, 2019. "A Development of the Human Development Index," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 146(3), pages 409-423, December.
    25. Lee Cronbach, 1951. "Coefficient alpha and the internal structure of tests," Psychometrika, Springer;The Psychometric Society, vol. 16(3), pages 297-334, September.
    26. R. O. Parreiras & P. Ya. Ekel & D. C. Morais, 2012. "Fuzzy Set Based Consensus Schemes for Multicriteria Group Decision making Applied to Strategic Planning," Group Decision and Negotiation, Springer, vol. 21(2), pages 153-183, March.
    27. Sureeyatanapas, Panitas & Sriwattananusart, Kawinpob & Niyamosoth, Thanawath & Sessomboon, Weerapat & Arunyanart, Sirawadee, 2018. "Supplier selection towards uncertain and unavailable information: An extension of TOPSIS method," Operations Research Perspectives, Elsevier, vol. 5(C), pages 69-79.
    28. Paul Dickes & Marie Valentova, 2013. "Construction, Validation and Application of the Measurement of Social Cohesion in 47 European Countries and Regions," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 113(3), pages 827-846, September.
    29. Green, Paul E & Srinivasan, V, 1978. "Conjoint Analysis in Consumer Research: Issues and Outlook," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 5(2), pages 103-123, Se.
    30. Ruiz, Francisco & Cabello, José M. & Pérez-Gladish, Blanca, 2018. "Building Ease-of-Doing-Business synthetic indicators using a double reference point approach," Technological Forecasting and Social Change, Elsevier, vol. 131(C), pages 130-140.
    31. Salvatore Greco & Alessio Ishizaka & Menelaos Tasiou & Gianpiero Torrisi, 2019. "On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation, and Robustness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 141(1), pages 61-94, January.
    32. Giuseppe Munda & Michela Nardo & Michaela Saisana & Tanja Srebotnjak, 2009. "Measuring uncertainties in composite indicators of sustainability," International Journal of Environmental Technology and Management, Inderscience Enterprises Ltd, vol. 11(1/2/3), pages 7-26.
    33. Petr Ekel & Patrícia Bernardes & Gláucia Maria Vasconcellos Vale & Matheus Pereira Libório, 2022. "South American business environment cost index: reforms for Brazil," International Journal of Business Environment, Inderscience Enterprises Ltd, vol. 13(2), pages 212-233.
    34. M. Saisana & A. Saltelli & S. Tarantola, 2005. "Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 168(2), pages 307-323, March.
    35. Geginat, Carolin & Ramalho, Rita, 2018. "Electricity connections and firm performance in 183 countries," Energy Economics, Elsevier, vol. 76(C), pages 344-366.
    36. Samira El Gibari & Trinidad Gómez & Francisco Ruiz, 2019. "Building composite indicators using multicriteria methods: a review," Journal of Business Economics, Springer, vol. 89(1), pages 1-24, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Matheus Pereira Libório & Alexandre Magno Alves Diniz & Hamidreza Rabiei-Dastjerd & Oseias da Silva Martinuci & Carlos Augusto Paiva da Silva Martins & Petr Iakovlevitch Ekel, 2023. "A Decision Framework for Identifying Methods to Construct Stable Composite Indicators That Capture the Concept of Multidimensional Social Phenomena: The Case of Social Exclusion," Sustainability, MDPI, vol. 15(7), pages 1-17, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cantone, Giulio Giacomo & Tomaselli, Venera, 2024. "On the Coherence of Composite Indexes: Multiversal Model and Specification Analysis for an Index of Well-Being," MetaArXiv d5y26, Center for Open Science.
    2. Matheus Pereira Libório & Elisa Fusco & Alexandre Magno Alves Diniz & Oséias da Silva Martinuci & Petr Iakovlevitch Ekel, 2024. "A Novel Approach for Multispatial and Multitemporal Analysis of Composite Indicators," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 173(3), pages 783-800, July.
    3. Greco, Salvatore & Ishizaka, Alessio & Tasiou, Menelaos & Torrisi, Gianpiero, 2019. "Sigma-Mu efficiency analysis: A methodology for evaluating units through composite indicators," European Journal of Operational Research, Elsevier, vol. 278(3), pages 942-960.
    4. Carlo Drago, 2021. "The Analysis and the Measurement of Poverty: An Interval-Based Composite Indicator Approach," Economies, MDPI, vol. 9(4), pages 1-17, October.
    5. Matheus Pereira Libório & João Francisco Abreu & Petr Iakovlevitch Ekel & Alexei Manso Correa Machado, 2023. "Effect of sub-indicator weighting schemes on the spatial dependence of multidimensional phenomena," Journal of Geographical Systems, Springer, vol. 25(2), pages 185-211, April.
    6. Milica Maricic & Jose A. Egea & Veljko Jeremic, 2019. "A Hybrid Enhanced Scatter Search—Composite I-Distance Indicator (eSS-CIDI) Optimization Approach for Determining Weights Within Composite Indicators," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 144(2), pages 497-537, July.
    7. Patrícia Bernardes & Petr Iakovlevitch Ekel & Sérgio Fernando Loureiro Rezende & Joel Gomes Pereira Júnior & Angélica Cidália Gouveia Santos & Maurício Andrade Rodrigues Costa & Rafael Lopes Carvalhai, 2022. "Cost of doing business index in Latin America," Quality & Quantity: International Journal of Methodology, Springer, vol. 56(4), pages 2233-2252, August.
    8. Matheus Pereira Libório & Alexandre Magno Alvez Diniz & Angélica Cidália Gouveia Santos & Cristiane Neri Nobre & Douglas Alexandre Gomes Vieira & Hasheem Mannan & Marcos Flávio Silveira Vasconcelos Da, 2024. "Benefit-of-the-Doubt in the Spatial Analysis of Child Well-Being in European Countries," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 17(4), pages 1851-1870, August.
    9. Greco, Salvatore & Ishizaka, Alessio & Tasiou, Menelaos & Torrisi, Gianpiero, 2018. "σ-µ efficiency analysis: A new methodology for evaluating units through composite indices," MPRA Paper 83569, University Library of Munich, Germany.
    10. Chao Shi & Kenneth C. Land, 2021. "The Data Envelopment Analysis and Equal Weights/Minimax Methods of Composite Social Indicator Construction: a Methodological Study of Data Sensitivity and Robustness," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 16(4), pages 1689-1716, August.
    11. Panagiotis Ravanos & Giannis Karagiannis, 2021. "A VEA Benefit-of-the-Doubt Model for the HDI," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 155(1), pages 27-46, May.
    12. Salvatore Greco & Alessio Ishizaka & Menelaos Tasiou & Gianpiero Torrisi, 2019. "On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation, and Robustness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 141(1), pages 61-94, January.
    13. Matheus Pereira Libório & Oseias da Silva Martinuci & Alexei Manso Correa Machado & Renata de Mello Lyrio & Patrícia Bernardes, 2022. "Time–Space Analysis of Multidimensional Phenomena: A Composite Indicator of Social Exclusion Through k-Means," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 159(2), pages 569-591, January.
    14. Marco Marozzi & Mario Bolzan, 2018. "An Index of Household Accessibility to Basic Services: A Study of Italian Regions," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 136(3), pages 1237-1250, April.
    15. Fusco, Elisa, 2023. "Potential improvements approach in composite indicators construction: The Multi-directional Benefit of the Doubt model," Socio-Economic Planning Sciences, Elsevier, vol. 85(C).
    16. Ruiz, Francisco & El Gibari, Samira & Cabello, José M. & Gómez, Trinidad, 2020. "MRP-WSCI: Multiple reference point based weak and strong composite indicators," Omega, Elsevier, vol. 95(C).
    17. Matheus Pereira Libório & Petr Iakovlevitch Ekel & Carlos Augusto Paiva Martins, 2023. "Economic analysis through alternative data and big data techniques: what do they tell about Brazil?," SN Business & Economics, Springer, vol. 3(1), pages 1-16, January.
    18. Mosnier, Claire & Boukhriss, Sanae & Minviel, Jean-Joseph, 2023. "Does pig production improves cattle farm sustainability in the French massif central? A hierarchical constrained directional benefit-of-the-doubt approach," Agricultural Systems, Elsevier, vol. 210(C).
    19. Gaganis, Chrysovalantis & Pasiouras, Fotios & Tasiou, Menelaos & Zopounidis, Constantin, 2021. "CISEF: A composite index of social, environmental and financial performance," European Journal of Operational Research, Elsevier, vol. 291(1), pages 394-409.
    20. Marta Kuc-Czarnecka & Samuele Lo Piano & Andrea Saltelli, 2020. "Quantitative Storytelling in the Making of a Composite Indicator," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 149(3), pages 775-802, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:soinre:v:164:y:2022:i:3:d:10.1007_s11205-022-02989-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.