IDEAS home Printed from https://ideas.repec.org/a/kap/theord/v73y2012i4p671-685.html
   My bibliography  Save this article

The Group Calibration Index: a group-based approach for assessing forecasters’ expertise when external outcome data are missing

Author

Listed:
  • Ilan Fischer
  • Ravid Bogaire

Abstract

The Group Calibration Index (GCI) provides a means of assessing the quality of forecasters’ predictions in situations that lack external feedback or outcome data. The GCI replaces the missing outcome data with aggregated ratings of a well-defined reference group. A simulation study and two experiments show how the GCI classifies forecaster performance and distinguishes between forecasters with restricted information and those with complete information. The results also show that under certain circumstances, where members of the reference group have high-quality information, the new GCI will outperform expert classification that is based on traditional calibration indices. Copyright Springer Science+Business Media, LLC. 2012

Suggested Citation

  • Ilan Fischer & Ravid Bogaire, 2012. "The Group Calibration Index: a group-based approach for assessing forecasters’ expertise when external outcome data are missing," Theory and Decision, Springer, vol. 73(4), pages 671-685, October.
    • Handle: RePEc:kap:theord:v:73:y:2012:i:4:p:671-685
    DOI: 10.1007/s11238-011-9265-4
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s11238-011-9265-4
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s11238-011-9265-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Keren, Gideon, 1987. "Facing uncertainty in the game of bridge: A calibration study," Organizational Behavior and Human Decision Processes, Elsevier, vol. 39(1), pages 98-114, February.
    2. Wagenaar, Willem A. & Keren, Gideon B., 1985. "Calibration of probability assessments by professional blackjack dealers, statistical experts, and lay people," Organizational Behavior and Human Decision Processes, Elsevier, vol. 36(3), pages 406-416, December.
    3. Fischer, Ilan & Budescu, David V., 2005. "When do those who know more also know more about how much they know? The development of confidence and performance in categorical decision tasks," Organizational Behavior and Human Decision Processes, Elsevier, vol. 98(1), pages 39-53, September.
    4. Richard P. Larrick & Jack B. Soll, 2006. "Intuitions About Combining Opinions: Misappreciation of the Averaging Principle," Management Science, INFORMS, vol. 52(1), pages 111-127, January.
    5. Sterman, John., 1994. "Learning in and about complex systems," Working papers 3660-94., Massachusetts Institute of Technology (MIT), Sloan School of Management.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Fischer, Ilan & Budescu, David V., 2005. "When do those who know more also know more about how much they know? The development of confidence and performance in categorical decision tasks," Organizational Behavior and Human Decision Processes, Elsevier, vol. 98(1), pages 39-53, September.
    2. Don A. Moore & Samuel A. Swift & Angela Minster & Barbara Mellers & Lyle Ungar & Philip Tetlock & Heather H. J. Yang & Elizabeth R. Tenney, 2017. "Confidence Calibration in a Multiyear Geopolitical Forecasting Competition," Management Science, INFORMS, vol. 63(11), pages 3552-3565, November.
    3. Lawrence, Michael & Goodwin, Paul & O'Connor, Marcus & Onkal, Dilek, 2006. "Judgmental forecasting: A review of progress over the last 25 years," International Journal of Forecasting, Elsevier, vol. 22(3), pages 493-518.
    4. Lu, Jinfeng & Dimov, Dimo, 2023. "A system dynamics modelling of entrepreneurship and growth within firms," Journal of Business Venturing, Elsevier, vol. 38(3).
    5. Laura Schmitt Olabisi & Amadou Sidibé, 2023. "Observations from a system dynamics modeling field school in Mali," System Dynamics Review, System Dynamics Society, vol. 39(1), pages 80-94, January.
    6. Armstrong, J. Scott & Green, Kesten C. & Graefe, Andreas, 2015. "Golden rule of forecasting: Be conservative," Journal of Business Research, Elsevier, vol. 68(8), pages 1717-1731.
    7. Day Yang Liu & Wen Chun Tsai & Pei Leen Liu & Chung Yi Fang, 2021. "Determinants of sales revenue in innovation diffusion effects of Taiwan sports lottery during the FIFA World Cup 2018," International Journal of Research in Business and Social Science (2147-4478), Center for the Strategic Studies in Business and Finance, vol. 10(4), pages 43-58, June.
    8. Oliva, Rogelio, 2003. "Model calibration as a testing strategy for system dynamics models," European Journal of Operational Research, Elsevier, vol. 151(3), pages 552-568, December.
    9. Hazhir Rahmandad & Nelson Repenning, 2016. "Capability erosion dynamics," Strategic Management Journal, Wiley Blackwell, vol. 37(4), pages 649-672, April.
    10. Katarzyna Tworek & Katarzyna Walecka-Jankowska & Anna Zgrzywa-Ziemak, 2019. "The role of information systems in shaping integrative logic for business sustainability," Operations Research and Decisions, Wroclaw University of Science and Technology, Faculty of Management, vol. 29(4), pages 125-146.
    11. Schilirò, Daniele & Graziano, Mario, 2011. "Scelte e razionalità nei modelli economici: un'analisi multidisciplinare [Choices and rationality in economic models: a multidisciplinary analysis]," MPRA Paper 31910, University Library of Munich, Germany.
    12. Brenner, Lyle & Griffin, Dale & Koehler, Derek J., 2005. "Modeling patterns of probability calibration with random support theory: Diagnosing case-based judgment," Organizational Behavior and Human Decision Processes, Elsevier, vol. 97(1), pages 64-81, May.
    13. Atanasov, Pavel & Witkowski, Jens & Ungar, Lyle & Mellers, Barbara & Tetlock, Philip, 2020. "Small steps to accuracy: Incremental belief updaters are better forecasters," Organizational Behavior and Human Decision Processes, Elsevier, vol. 160(C), pages 19-35.
    14. Natalia Ciobanu & Ali Kerem Saysel, 2021. "Using social–ecological inventory and group model building for resilience assessment to climate change in a network governance setting: a case study from Ikel watershed in Moldova," Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development, Springer, vol. 23(1), pages 1065-1085, January.
    15. McKenzie, Craig R.M. & Liersch, Michael J. & Yaniv, Ilan, 2008. "Overconfidence in interval estimates: What does expertise buy you?," Organizational Behavior and Human Decision Processes, Elsevier, vol. 107(2), pages 179-191, November.
    16. repec:cup:judgdm:v:14:y:2019:i:4:p:395-411 is not listed on IDEAS
    17. Rafael Marcos-Sánchez & Daniel Ferrández & Carlos Morón, 2022. "Systems Thinking for Sustainability Education in Building and Business Administration and Management Degrees," Sustainability, MDPI, vol. 14(19), pages 1-20, September.
    18. Meri Duryan & Dragan Nikolik & Godefridus Merode & Leopold M. G. Curfs, 2015. "Reflecting on the efficacy of cognitive mapping for decision-making in intellectual disability care: a case study," International Journal of Health Planning and Management, Wiley Blackwell, vol. 30(2), pages 127-144, April.
    19. Gino, Francesca, 2008. "Do we listen to advice just because we paid for it? The impact of advice cost on its use," Organizational Behavior and Human Decision Processes, Elsevier, vol. 107(2), pages 234-245, November.
    20. repec:cup:judgdm:v:8:y:2013:i:2:p:91-105 is not listed on IDEAS
    21. Florian Kapmeier, 2020. "Reflections on developing a simulation model on sustainable and healthy diets for decision makers: Comment on the paper by Kopainsky," Systems Research and Behavioral Science, Wiley Blackwell, vol. 37(6), pages 928-935, November.
    22. Biskup, Dirk, 2008. "A state-of-the-art review on scheduling with learning effects," European Journal of Operational Research, Elsevier, vol. 188(2), pages 315-329, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:kap:theord:v:73:y:2012:i:4:p:671-685. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.