IDEAS home Printed from https://ideas.repec.org/a/eee/ecomod/v230y2012icp50-62.html
   My bibliography  Save this article

Metrics for evaluating performance and uncertainty of Bayesian network models

Author

Listed:
  • Marcot, Bruce G.

Abstract

This paper presents a selected set of existing and new metrics for gauging Bayesian network model performance and uncertainty. Selected existing and new metrics are discussed for conducting model sensitivity analysis (variance reduction, entropy reduction, case file simulation); evaluating scenarios (influence analysis); depicting model complexity (numbers of model variables, links, node states, conditional probabilities, and node cliques); assessing prediction performance (confusion tables, covariate- and conditional probability-weighted confusion error rates, area under receiver operating characteristic curves, k-fold cross-validation, spherical payoff, Schwarz’ Bayesian information criterion, true skill statistic, Cohen's kappa); and evaluating uncertainty of model posterior probability distributions (Bayesian credible interval, posterior probability certainty index, certainty envelope, Gini coefficient). Examples are presented of applying the metrics to 3 real-world models of wildlife population analysis and management. Using such metrics can vitally bolster model credibility, acceptance, and appropriate application, particularly when informing management decisions.

Suggested Citation

  • Marcot, Bruce G., 2012. "Metrics for evaluating performance and uncertainty of Bayesian network models," Ecological Modelling, Elsevier, vol. 230(C), pages 50-62.
  • Handle: RePEc:eee:ecomod:v:230:y:2012:i:c:p:50-62
    DOI: 10.1016/j.ecolmodel.2012.01.013
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0304380012000245
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.ecolmodel.2012.01.013?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Monte, Luigi, 2009. "Multi-model approach and evaluation of the uncertainty of model results. Rationale and applications to predict the behaviour of contaminants in the abiotic components of the fresh water environment," Ecological Modelling, Elsevier, vol. 220(12), pages 1469-1480.
    2. Steven C. Amstrup & Eric T. DeWeaver & David C. Douglas & Bruce G. Marcot & George M. Durner & Cecilia M. Bitz & David A. Bailey, 2010. "Greenhouse gas mitigation can reduce sea-ice loss and increase polar bear persistence," Nature, Nature, vol. 468(7326), pages 955-958, December.
    3. Jeremy Cain & Charles Batchelor & Dominic Waughray, 1999. "Belief Networks: A Framework for the Participatory Development of Natural Resource Management Strategies," Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development, Springer, vol. 1(2), pages 123-133, June.
    4. Uusitalo, Laura, 2007. "Advantages and challenges of Bayesian networks in environmental modelling," Ecological Modelling, Elsevier, vol. 203(3), pages 312-318.
    5. Huang, H.-C. & Martinez, F. & Mateu, J. & Montes, F., 2007. "Model comparison and selection for stationary space-time models," Computational Statistics & Data Analysis, Elsevier, vol. 51(9), pages 4577-4596, May.
    6. Thogmartin, Wayne E., 2010. "Sensitivity analysis of North American bird population estimates," Ecological Modelling, Elsevier, vol. 221(2), pages 173-177.
    7. Atkinson, Anthony B., 1970. "On the measurement of inequality," Journal of Economic Theory, Elsevier, vol. 2(3), pages 244-263, September.
    8. David J. Spiegelhalter & Nicola G. Best & Bradley P. Carlin & Angelika Van Der Linde, 2002. "Bayesian measures of model complexity and fit," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 64(4), pages 583-639, October.
    9. Castillo, Enrique & Gutiérrez, José Manuel & Hadi, Ali S., 1998. "Modeling Probabilistic Networks of Discrete and Continuous Variables," Journal of Multivariate Analysis, Elsevier, vol. 64(1), pages 48-65, January.
    10. Mouton, Ans M. & De Baets, Bernard & Goethals, Peter L.M., 2010. "Ecological relevance of performance criteria for species distribution models," Ecological Modelling, Elsevier, vol. 221(16), pages 1995-2002.
    11. Williams, Byron K., 2011. "Resolving structural uncertainty in natural resources management using POMDP approaches," Ecological Modelling, Elsevier, vol. 222(5), pages 1092-1102.
    12. Ascough, J.C. & Maier, H.R. & Ravalico, J.K. & Strudley, M.W., 2008. "Future research challenges for incorporation of uncertainty in environmental and ecological decision-making," Ecological Modelling, Elsevier, vol. 219(3), pages 383-399.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Edilberto Cepeda-Cuervo & Vicente Núñez-Antón, 2013. "Spatial Double Generalized Beta Regression Models," Journal of Educational and Behavioral Statistics, , vol. 38(6), pages 604-628, December.
    2. Fabrizi, Enrico & Trivisano, Carlo, 2016. "Small area estimation of the Gini concentration coefficient," Computational Statistics & Data Analysis, Elsevier, vol. 99(C), pages 223-234.
    3. Nam C. Nguyen & Doug Graham & Helen Ross & Kambiz Maani & Ockie Bosch, 2012. "Educating Systems Thinking for Sustainability: Experience with a Developing Country," Systems Research and Behavioral Science, Wiley Blackwell, vol. 29(1), pages 14-29, January.
    4. Ashantha Ranasinghe & Xuejuan Su, 2023. "When social assistance meets market power: A mixed duopoly view of health insurance in the United States," Economic Inquiry, Western Economic Association International, vol. 61(4), pages 851-869, October.
    5. Buddhavarapu, Prasad & Bansal, Prateek & Prozzi, Jorge A., 2021. "A new spatial count data model with time-varying parameters," Transportation Research Part B: Methodological, Elsevier, vol. 150(C), pages 566-586.
    6. Mumtaz, Haroon & Theodoridis, Konstantinos, 2017. "Common and country specific economic uncertainty," Journal of International Economics, Elsevier, vol. 105(C), pages 205-216.
    7. Christina Leuker & Thorsten Pachur & Ralph Hertwig & Timothy J. Pleskac, 2019. "Do people exploit risk–reward structures to simplify information processing in risky choice?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 5(1), pages 76-94, August.
    8. Eckstein, Zvi & Zilcha, Itzhak, 1994. "The effects of compulsory schooling on growth, income distribution and welfare," Journal of Public Economics, Elsevier, vol. 54(3), pages 339-359, July.
    9. Roobavannan, M. & Kandasamy, J. & Pande, S. & Vigneswaran, S. & Sivapalan, M., 2020. "Sustainability of agricultural basin development under uncertain future climate and economic conditions: A socio-hydrological analysis," Ecological Economics, Elsevier, vol. 174(C).
    10. Alessandro Spiganti, 2022. "Wealth Inequality and the Exploration of Novel Alternatives," Working Papers 2022:02, Department of Economics, University of Venice "Ca' Foscari".
    11. Günther Rehme, 2007. "Education, Economic Growth and Measured Income Inequality," Economica, London School of Economics and Political Science, vol. 74(295), pages 493-514, August.
    12. Junyi Zhu, 2014. "Bracket Creep Revisited - with and without r > g: Evidence from Germany," Journal of Income Distribution, Ad libros publications inc., vol. 23(3), pages 106-158, November.
    13. Rubio, F.J. & Steel, M.F.J., 2011. "Inference for grouped data with a truncated skew-Laplace distribution," Computational Statistics & Data Analysis, Elsevier, vol. 55(12), pages 3218-3231, December.
    14. Juan Antonio Duro & Jordi Teixidó-Figueras & Emilio Padilla, 2017. "The Causal Factors of International Inequality in $$\hbox {CO}_{2}$$ CO 2 Emissions Per Capita: A Regression-Based Inequality Decomposition Analysis," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 67(4), pages 683-700, August.
    15. Jeni Klugman & Francisco Rodríguez & Hyung-Jin Choi, 2011. "The HDI 2010: new controversies, old critiques," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 9(2), pages 249-288, June.
    16. Vani K. Borooah, 2013. "A general measure of the ‘effective’ number of parties in a political system," Chapters, in: Francisco Cabrillo & Miguel A. Puchades-Navarro (ed.), Constitutional Economics and Public Institutions, chapter 8, pages 146-159, Edward Elgar Publishing.
    17. Francesco Andreoli & Eugenio Peluso, 2016. "So close yet so unequal: Reconsidering spatial inequality in U.S. cities," Working Papers 21/2016, University of Verona, Department of Economics.
    18. Andonie, Costel & Kuzmics, Christoph & Rogers, Brian W., 2019. "Efficiency-based measures of inequality," Journal of Mathematical Economics, Elsevier, vol. 85(C), pages 60-69.
    19. repec:dau:papers:123456789/4543 is not listed on IDEAS
    20. Alessandri, Piergiorgio & Mumtaz, Haroon, 2019. "Financial regimes and uncertainty shocks," Journal of Monetary Economics, Elsevier, vol. 101(C), pages 31-46.
    21. Svetlana V. Tishkovskaya & Paul G. Blackwell, 2021. "Bayesian estimation of heterogeneous environments from animal movement data," Environmetrics, John Wiley & Sons, Ltd., vol. 32(6), September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ecomod:v:230:y:2012:i:c:p:50-62. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/ecological-modelling .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.