IDEAS home Printed from https://ideas.repec.org/a/eee/csdana/v115y2017icp122-135.html
   My bibliography  Save this article

A novel method for estimating the common signals for consensus across multiple ranked lists

Author

Listed:
  • Švendová, Vendula
  • Schimek, Michael G.

Abstract

The ranking of objects, such as journals, institutions or biological entities, is broadly used to assess the relative quality or relevance of such objects. A multiple ranking is performed by a number of assessors (humans or machines) and inference about the nature of the observed rankings is desirable for evaluation, business or scientific purposes. The assessors’ decisions are based on some inherent metric scale and depend on judgement and discriminatory ability, data to which we usually do not have access. An indirect inference approach is proposed that allows one to estimate those signal parameters that might be causal for the observed rankings obtained from several assessors, some of which may not necessarily provide the same decision quality. The order of the values represents a consensus ranking across the observed individual rankings. The standard errors of the estimated signal parameters are obtained through a non-parametric bootstrap. Hence, the signal variability can be evaluated object-wise for the purpose of quantifying the stability of the associated rank positions. As a result, such signal estimates can be used in the meta-analysis of conceptually similar evaluation exercises, studies or experiments, and in any data integration task where measurements on the metric scale are either unavailable, or not directly comparable. The suggested approach is validated on simulated rank data as well as on experimental rank data from current molecular medicine. The proposed algorithms were implemented and all calculations performed in the R environment. The source code is provided.

Suggested Citation

  • Švendová, Vendula & Schimek, Michael G., 2017. "A novel method for estimating the common signals for consensus across multiple ranked lists," Computational Statistics & Data Analysis, Elsevier, vol. 115(C), pages 122-135.
    • Handle: RePEc:eee:csdana:v:115:y:2017:i:c:p:122-135
    DOI: 10.1016/j.csda.2017.05.010
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167947317301056
    Download Restriction: Full text for ScienceDirect subscribers only.
    File URL: https://libkey.io/10.1016/j.csda.2017.05.010?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Gourieroux, C & Monfort, A & Renault, E, 1993. "Indirect Inference," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 8(S), pages 85-118, Suppl. De.
    2. Peter Hall & Michael G. Schimek, 2012. "Moderate-Deviation-Based Inference for Random Degeneration in Paired Rank Lists," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 107(498), pages 661-672, June.
    3. Kristen Fortney & Joshua Griesman & Max Kotlyar & Chiara Pastrello & Marc Angeli & Ming Sound-Tsao & Igor Jurisica, 2015. "Prioritizing Therapeutics for Lung Cancer: An Integrative Meta-analysis of Cancer Gene Signatures and Chemogenomic Data," PLOS Computational Biology, Public Library of Science, vol. 11(3), pages 1-17, March.
    4. DeConde Robert P & Hawley Sarah & Falcon Seth & Clegg Nigel & Knudsen Beatrice & Etzioni Ruth, 2006. "Combining Results of Microarray Experiments: A Rank Aggregation Approach," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 5(1), pages 1-25, June.
    5. R. L. Plackett, 1975. "The Analysis of Permutations," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 24(2), pages 193-202, June.
    6. Lin Shili, 2010. "Space Oriented Rank-Based Data Integration," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 9(1), pages 1-25, April.
    7. Shili Lin & Jie Ding, 2009. "Integration of Ranked Lists via Cross Entropy Monte Carlo with Applications to mRNA and microRNA Studies," Biometrics, The International Biometric Society, vol. 65(1), pages 9-18, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Antonella Plaia & Simona Buscemi & Johannes Fürnkranz & Eneldo Loza Mencía, 2022. "Comparing Boosting and Bagging for Decision Trees of Rankings," Journal of Classification, Springer;The Classification Society, vol. 39(1), pages 78-99, March.
    2. Antonio D’Ambrosio & Carmela Iorio & Michele Staiano & Roberta Siciliano, 2019. "Median constrained bucket order rank aggregation," Computational Statistics, Springer, vol. 34(2), pages 787-802, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Antonio D’Ambrosio & Carmela Iorio & Michele Staiano & Roberta Siciliano, 2019. "Median constrained bucket order rank aggregation," Computational Statistics, Springer, vol. 34(2), pages 787-802, June.
    2. Giuseppe Jurman & Samantha Riccadonna & Roberto Visintainer & Cesare Furlanello, 2012. "Algebraic Comparison of Partial Lists in Bioinformatics," PLOS ONE, Public Library of Science, vol. 7(5), pages 1-20, May.
    3. Schimek Michael G. & Budinská Eva & Kugler Karl G. & Švendová Vendula & Ding Jie & Lin Shili, 2015. "TopKLists: a comprehensive R package for statistical inference, stochastic aggregation, and visualization of multiple omics ranked lists," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 14(3), pages 311-316, June.
    4. Yoo, Yeawon & Escobedo, Adolfo R. & Skolfield, J. Kyle, 2020. "A new correlation coefficient for comparing and aggregating non-strict and incomplete rankings," European Journal of Operational Research, Elsevier, vol. 285(3), pages 1025-1041.
    5. Antonio Jiménez-Martín & Eduardo Gallego & Alfonso Mateos & Juan A. Fernández Pozo, 2017. "Restoring a Radionuclide Contaminated Aquatic Ecosystem: A Group Decision Making Problem with Incomplete Information within MAUT Accounting for Veto," Group Decision and Negotiation, Springer, vol. 26(4), pages 653-675, July.
    6. Luisa Cutillo & Annamaria Carissimo & Silvia Figini, 2012. "Network Selection: A Method for Ranked Lists Selection," PLOS ONE, Public Library of Science, vol. 7(8), pages 1-13, August.
    7. Chéron, Arnaud & Hairault, Jean-Olivier & Langot, François, 2004. "Labor Market Institutions and the Employment-Productivity Trade-Off: A Wage Posting Approach," IZA Discussion Papers 1364, Institute of Labor Economics (IZA).
    8. Jessica Foo & Lek-Heng Lim & Ken Sze-Wai Wong, 2017. "Macroeconomics and FinTech: Uncovering Latent Macroeconomic Effects on Peer-to-Peer Lending," Papers 1710.11283, arXiv.org.
    9. Peter Fuleky & Eric Zivot, 2014. "Indirect inference based on the score," Econometrics Journal, Royal Economic Society, vol. 17(3), pages 383-393, October.
    10. Lombardi, Marco J. & Calzolari, Giorgio, 2009. "Indirect estimation of [alpha]-stable stochastic volatility models," Computational Statistics & Data Analysis, Elsevier, vol. 53(6), pages 2298-2308, April.
    11. Coenen, Gunter & Wieland, Volker, 2005. "A small estimated euro area model with rational expectations and nominal rigidities," European Economic Review, Elsevier, vol. 49(5), pages 1081-1104, July.
    12. Thomas D. Tallarini, Jr. & Harold H. Zhang, 2005. "External Habit and the Cyclicality of Expected Stock Returns," The Journal of Business, University of Chicago Press, vol. 78(3), pages 1023-1048, May.
    13. Christopher Taber & Rune Vejlin, 2020. "Estimation of a Roy/Search/Compensating Differential Model of the Labor Market," Econometrica, Econometric Society, vol. 88(3), pages 1031-1069, May.
    14. Calvet, Laurent E. & Czellar, Veronika, 2015. "Through the looking glass: Indirect inference via simple equilibria," Journal of Econometrics, Elsevier, vol. 185(2), pages 343-358.
    15. Yu, Jun, 2012. "Bias in the estimation of the mean reversion parameter in continuous time models," Journal of Econometrics, Elsevier, vol. 169(1), pages 114-122.
    16. Baker, Rose D. & McHale, Ian G., 2014. "A dynamic paired comparisons model: Who is the greatest tennis player?," European Journal of Operational Research, Elsevier, vol. 236(2), pages 677-684.
    17. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    18. Hamilton, James D. & Wu, Jing Cynthia, 2012. "Identification and estimation of Gaussian affine term structure models," Journal of Econometrics, Elsevier, vol. 168(2), pages 315-331.
    19. Jesús Fernández-Villaverde & Juan F. Rubio-Ramirez, 2004. "Estimating nonlinear dynamic equilibrium economies: a likelihood approach," FRB Atlanta Working Paper 2004-1, Federal Reserve Bank of Atlanta.
    20. Corradi, Valentina & Swanson, Norman R., 2005. "Bootstrap specification tests for diffusion processes," Journal of Econometrics, Elsevier, vol. 124(1), pages 117-148, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:csdana:v:115:y:2017:i:c:p:122-135. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/csda .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.