IDEAS home Printed from https://ideas.repec.org/a/plo/pgen00/1001230.html
   My bibliography  Save this article

A Unifying Framework for Evaluating the Predictive Power of Genetic Variants Based on the Level of Heritability Explained

Author

Listed:
  • Hon-Cheong So
  • Pak C Sham

Abstract

An increasing number of genetic variants have been identified for many complex diseases. However, it is controversial whether risk prediction based on genomic profiles will be useful clinically. Appropriate statistical measures to evaluate the performance of genetic risk prediction models are required. Previous studies have mainly focused on the use of the area under the receiver operating characteristic (ROC) curve, or AUC, to judge the predictive value of genetic tests. However, AUC has its limitations and should be complemented by other measures. In this study, we develop a novel unifying statistical framework that connects a large variety of predictive indices together. We showed that, given the overall disease probability and the level of variance in total liability (or heritability) explained by the genetic variants, we can estimate analytically a large variety of prediction metrics, for example the AUC, the mean risk difference between cases and non-cases, the net reclassification improvement (ability to reclassify people into high- and low-risk categories), the proportion of cases explained by a specific percentile of population at the highest risk, the variance of predicted risks, and the risk at any percentile. We also demonstrate how to construct graphs to visualize the performance of risk models, such as the ROC curve, the density of risks, and the predictiveness curve (disease risk plotted against risk percentile). The results from simulations match very well with our theoretical estimates. Finally we apply the methodology to nine complex diseases, evaluating the predictive power of genetic tests based on known susceptibility variants for each trait.Author Summary: Recently many genetic variants have been established for diseases, and the findings have raised hope for risk prediction based on genomic profiles. However, we need to have proper statistical measures to assess the usefulness of such tests. In this study, we developed a statistical framework which enables us to evaluate many predictive indices analytically. It is based on the liability threshold model, which postulates a latent liability that is normally distributed. Affected individuals are assumed to have a liability exceeding a certain threshold. We demonstrated that, given the overall disease probability and variance in liability explained by the genetic markers, we can compute a variety of predictive indices. An example is the area under the receiver operating characteristic (ROC) curve, or AUC, which is very commonly employed. However, the limitations of AUC are often ignored, and we proposed complementing it with other indices. We have therefore also computed other metrics like the average difference in risks between cases and non-cases, the ability of reclassification into high- and low-risk categories, and the proportion of cases accounted for by a certain percentile of population at the highest risk. We also derived how to construct graphs showing the risk distribution in population.

Suggested Citation

  • Hon-Cheong So & Pak C Sham, 2010. "A Unifying Framework for Evaluating the Predictive Power of Genetic Variants Based on the Level of Heritability Explained," PLOS Genetics, Public Library of Science, vol. 6(12), pages 1-13, December.
  • Handle: RePEc:plo:pgen00:1001230
    DOI: 10.1371/journal.pgen.1001230
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosgenetics/article?id=10.1371/journal.pgen.1001230
    Download Restriction: no

    File URL: https://journals.plos.org/plosgenetics/article/file?id=10.1371/journal.pgen.1001230&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pgen.1001230?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Margaret Pepe & Holly Janes & Gary Longton & Wendy Leisenring & Polly Newcomb, 2004. "Limitations of the Odds Ratio in Gauging the Performance of a Diagnostic or Prognostic Marker," UW Biostatistics Working Paper Series 1035, Berkeley Electronic Press.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Emil M. Pedersen & Esben Agerbo & Oleguer Plana-Ripoll & Jette Steinbach & Morten D. Krebs & David M. Hougaard & Thomas Werge & Merete Nordentoft & Anders D. Børglum & Katherine L. Musliner & Andrea G, 2023. "ADuLT: An efficient and robust time-to-event GWAS," Nature Communications, Nature, vol. 14(1), pages 1-12, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Debashis Ghosh & Michael S. Sabel, 2022. "A Weighted Sample Framework to Incorporate External Calculators for Risk Modeling," Statistics in Biosciences, Springer;International Chinese Statistical Association, vol. 14(3), pages 363-379, December.
    2. Aljoscha Benjamin Hwang & Guido Schuepfer & Mario Pietrini & Stefan Boes, 2021. "External validation of EPIC’s Risk of Unplanned Readmission model, the LACE+ index and SQLape as predictors of unplanned hospital readmissions: A monocentric, retrospective, diagnostic cohort study in," PLOS ONE, Public Library of Science, vol. 16(11), pages 1-33, November.
    3. Anna-Karin Ivert & Marie Torstensson Levander & Juan Merlo, 2013. "Adolescents' Utilisation of Psychiatric Care, Neighbourhoods and Neighbourhood Socioeconomic Deprivation: A Multilevel Analysis," PLOS ONE, Public Library of Science, vol. 8(11), pages 1-1, November.
    4. Margaret Sullivan Pepe & Tianxi Cai & Gary Longton, 2006. "Combining Predictors for Classification Using the Area under the Receiver Operating Characteristic Curve," Biometrics, The International Biometric Society, vol. 62(1), pages 221-229, March.
    5. Holly Janes & Margaret S. Pepe, 2008. "Matching in Studies of Classification Accuracy: Implications for Analysis, Efficiency, and Assessment of Incremental Value," Biometrics, The International Biometric Society, vol. 64(1), pages 1-9, March.
    6. Carlos A Labarrere & John R Woods & James W Hardin & Beate R Jaeger & Marian Zembala & Mario C Deng & Ghassan S Kassab, 2014. "Early Inflammatory Markers Are Independent Predictors of Cardiac Allograft Vasculopathy in Heart-Transplant Recipients," PLOS ONE, Public Library of Science, vol. 9(12), pages 1-18, December.
    7. Pia Kjær Kristensen & Raquel Perez-Vicente & George Leckie & Søren Paaske Johnsen & Juan Merlo, 2020. "Disentangling the contribution of hospitals and municipalities for understanding patient level differences in one-year mortality risk after hip-fracture: A cross-classified multilevel analysis in Swed," PLOS ONE, Public Library of Science, vol. 15(6), pages 1-14, June.
    8. Diego Tomassi & Liliana Forzani & Efstathia Bura & Ruth Pfeiffer, 2017. "Sufficient dimension reduction for censored predictors," Biometrics, The International Biometric Society, vol. 73(1), pages 220-231, March.
    9. Osamu Komori, 2011. "A boosting method for maximization of the area under the ROC curve," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 63(5), pages 961-979, October.
    10. Quang Bao Le & Boubaker Dhehibi, 2019. "A Typology-Based Approach for Assessing Qualities and Determinants of Adoption of Sustainable Water Use Technologies in Coping with Context Diversity: The Case of Mechanized Raised-Bed Technology in E," Sustainability, MDPI, vol. 11(19), pages 1-21, September.
    11. Kenichi Hayashi & Shinto Eguchi, 2024. "A new integrated discrimination improvement index via odds," Statistical Papers, Springer, vol. 65(8), pages 4971-4990, October.
    12. Tianle Chen & Yuanjia Wang & Huaihou Chen & Karen Marder & Donglin Zeng, 2014. "Targeted Local Support Vector Machine for Age-Dependent Classification," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(507), pages 1174-1187, September.
    13. Hajime Uno & Tianxi Cai & Lu Tian & L. J. Wei, 2011. "Graphical Procedures for Evaluating Overall and Subject-Specific Incremental Values from New Predictors with Censored Event Time Data," Biometrics, The International Biometric Society, vol. 67(4), pages 1389-1396, December.
    14. Anna Persmark & Maria Wemrell & Sofia Zettermark & George Leckie & S V Subramanian & Juan Merlo, 2019. "Precision public health: Mapping socioeconomic disparities in opioid dispensations at Swedish pharmacies by Multilevel Analysis of Individual Heterogeneity and Discriminatory Accuracy (MAIHDA)," PLOS ONE, Public Library of Science, vol. 14(8), pages 1-21, August.
    15. Rebecca Yates Coley & Aaron J. Fisher & Mufaddal Mamawala & Herbert Ballentine Carter & Kenneth J. Pienta & Scott L. Zeger, 2017. "A Bayesian hierarchical model for prediction of latent health states from multiple data sources with application to active surveillance of prostate cancer," Biometrics, The International Biometric Society, vol. 73(2), pages 625-634, June.
    16. Juan Merlo & Philippe Wagner & Nermin Ghith & George Leckie, 2016. "An Original Stepwise Multilevel Logistic Regression Analysis of Discriminatory Accuracy: The Case of Neighbourhoods and Health," PLOS ONE, Public Library of Science, vol. 11(4), pages 1-31, April.
    17. Michael Lebenbaum & Osvaldo Espin-Garcia & Yi Li & Laura C Rosella, 2018. "Development and validation of a population based risk algorithm for obesity: The Obesity Population Risk Tool (OPoRT)," PLOS ONE, Public Library of Science, vol. 13(1), pages 1-11, January.
    18. Michael King & Louise Marston & Igor Švab & Heidi-Ingrid Maaroos & Mirjam I Geerlings & Miguel Xavier & Vicente Benjamin & Francisco Torres-Gonzalez & Juan Angel Bellon-Saameno & Danica Rotar & Anu Al, 2011. "Development and Validation of a Risk Model for Prediction of Hazardous Alcohol Consumption in General Practice Attendees: The PredictAL Study," PLOS ONE, Public Library of Science, vol. 6(8), pages 1-10, August.
    19. Haleh Yasrebi & Peter Sperisen & Viviane Praz & Philipp Bucher, 2009. "Can Survival Prediction Be Improved By Merging Gene Expression Data Sets?," PLOS ONE, Public Library of Science, vol. 4(10), pages 1-14, October.
    20. Shai Mulinari & Sol Pia Juárez & Philippe Wagner & Juan Merlo, 2015. "Does Maternal Country of Birth Matter for Understanding Offspring’s Birthweight? A Multilevel Analysis of Individual Heterogeneity in Sweden," PLOS ONE, Public Library of Science, vol. 10(5), pages 1-19, May.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pgen00:1001230. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosgenetics (email available below). General contact details of provider: https://journals.plos.org/plosgenetics/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.