IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0157914.html
   My bibliography  Save this article

The Success of Linear Bootstrapping Models: Decision Domain-, Expertise-, and Criterion-Specific Meta-Analysis

Author

Listed:
  • Esther Kaufmann
  • Werner W Wittmann

Abstract

The success of bootstrapping or replacing a human judge with a model (e.g., an equation) has been demonstrated in Paul Meehl’s (1954) seminal work and bolstered by the results of several meta-analyses. To date, however, analyses considering different types of meta-analyses as well as the potential dependence of bootstrapping success on the decision domain, the level of expertise of the human judge, and the criterion for what constitutes an accurate decision have been missing from the literature. In this study, we addressed these research gaps by conducting a meta-analysis of lens model studies. We compared the results of a traditional (bare-bones) meta-analysis with findings of a meta-analysis of the success of bootstrap models corrected for various methodological artifacts. In line with previous studies, we found that bootstrapping was more successful than human judgment. Furthermore, bootstrapping was more successful in studies with an objective decision criterion than in studies with subjective or test score criteria. We did not find clear evidence that the success of bootstrapping depended on the decision domain (e.g., education or medicine) or on the judge’s level of expertise (novice or expert). Correction of methodological artifacts increased the estimated success of bootstrapping, suggesting that previous analyses without artifact correction (i.e., traditional meta-analyses) may have underestimated the value of bootstrapping models.

Suggested Citation

  • Esther Kaufmann & Werner W Wittmann, 2016. "The Success of Linear Bootstrapping Models: Decision Domain-, Expertise-, and Criterion-Specific Meta-Analysis," PLOS ONE, Public Library of Science, vol. 11(6), pages 1-21, June.
    • Handle: RePEc:plo:pone00:0157914
    DOI: 10.1371/journal.pone.0157914
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0157914
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0157914&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0157914?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Sue Duval & Richard Tweedie, 2000. "Trim and Fill: A Simple Funnel-Plot–Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis," Biometrics, The International Biometric Society, vol. 56(2), pages 455-463, June.
    2. Harvey, Nigel & Harries, Clare, 2004. "Effects of judges' forecasting on their later combination of forecasts for the same outcomes," International Journal of Forecasting, Elsevier, vol. 20(3), pages 391-409.
    3. Mear, Ross & Firth, Michael, 1987. "Assessing the accuracy of financial analyst security return predictions," Accounting, Organizations and Society, Elsevier, vol. 12(4), pages 331-340, June.
    4. Esther Kaufmann & Ulf-Dietrich Reips & Werner W Wittmann, 2013. "A Critical Meta-Analysis of Lens Model Studies in Human Judgment and Decision-Making," PLOS ONE, Public Library of Science, vol. 8(12), pages 1-16, December.
    5. Stewart, Thomas R. & Roebber, Paul J. & Bosart, Lance F., 1997. "The Importance of the Task in Analyzing Expert Judgment," Organizational Behavior and Human Decision Processes, Elsevier, vol. 69(3), pages 205-219, March.
    6. Ashton, Ah, 1982. "An Empirical-Study Of Budget-Related Predictions Of Corporate-Executives," Journal of Accounting Research, Wiley Blackwell, vol. 20(2), pages 440-449.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Roland W. Scholz, 2017. "Managing complexity: from visual perception to sustainable transitions—contributions of Brunswik’s Theory of Probabilistic Functionalism," Environment Systems and Decisions, Springer, vol. 37(4), pages 381-409, December.
    2. Rebitschek, Felix G. & Gigerenzer, Gerd & Wagner, Gert G., 2021. "People underestimate the errors made by algorithms for credit scoring and recidivism prediction but accept even fewer errors," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 11, pages 1-11.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Esther Kaufmann & Ulf-Dietrich Reips & Werner W Wittmann, 2013. "A Critical Meta-Analysis of Lens Model Studies in Human Judgment and Decision-Making," PLOS ONE, Public Library of Science, vol. 8(12), pages 1-16, December.
    2. Anqiang Huang & Han Qiao & Shouyang Wang & John Liu, 2016. "Improving Forecasting Performance by Exploiting Expert Knowledge: Evidence from Guangzhou Port," International Journal of Information Technology & Decision Making (IJITDM), World Scientific Publishing Co. Pte. Ltd., vol. 15(02), pages 387-401, March.
    3. Anqiang Huang & Kin Keung Lai & Han Qiao & Shouyang Wang & Zhenji Zhang, 2018. "Does Interval Knowledge Sharpen Forecasting Models? Evidence from China’s Typical Ports," International Journal of Information Technology & Decision Making (IJITDM), World Scientific Publishing Co. Pte. Ltd., vol. 17(02), pages 467-483, March.
    4. Qian Li & Yan Chen & Shikun Sun & Muyuan Zhu & Jing Xue & Zihan Gao & Jinfeng Zhao & Yihe Tang, 2022. "Research on Crop Irrigation Schedules Under Deficit Irrigation—A Meta-analysis," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 36(12), pages 4799-4817, September.
    5. Bart Verkuil & Serpil Atasayi & Marc L Molendijk, 2015. "Workplace Bullying and Mental Health: A Meta-Analysis on Cross-Sectional and Longitudinal Data," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-16, August.
    6. Damiano Pizzol & Mike Trott & Igor Grabovac & Mario Antunes & Anna Claudia Colangelo & Simona Ippoliti & Cristian Petre Ilie & Anne Carrie & Nicola Veronese & Lee Smith, 2021. "Laparoscopy in Low-Income Countries: 10-Year Experience and Systematic Literature Review," IJERPH, MDPI, vol. 18(11), pages 1-11, May.
    7. Wolfgang Goymann & John C. Wingfield, 2014. "Male-to-female testosterone ratios, dimorphism, and life history—what does it really tell us?," Behavioral Ecology, International Society for Behavioral Ecology, vol. 25(4), pages 685-699.
    8. Gehrig, Thomas & Güth, Werner & Leví0nský, René & Popova, Vera, 2010. "On the evolution of professional consulting," Journal of Economic Behavior & Organization, Elsevier, vol. 76(1), pages 113-126, October.
    9. Alderotti, Giammarco & Rapallini, Chiara & Traverso, Silvio, 2023. "The Big Five personality traits and earnings: A meta-analysis," Journal of Economic Psychology, Elsevier, vol. 94(C).
    10. Ünal, Zehra E. & Kartal, Gamze & Ulusoy, Serra & Ala, Aslı M. & Yilmaz, Munube & Geary, David C., 2023. "Relative contributions of g and basic domain-specific mathematics skills to complex mathematics competencies," Intelligence, Elsevier, vol. 101(C).
    11. Daniele Zago & Maria Eugênia Andrighetto Canozzi & Júlio Otávio Jardim Barcellos, 2020. "Pregnant beef cow’s nutrition and its effects on postnatal weight and carcass quality of their progeny," PLOS ONE, Public Library of Science, vol. 15(8), pages 1-20, August.
    12. Viktoria Maria Baumeister & Leonie Petra Kuen & Maike Bruckes & Gerhard Schewe, 2021. "The Relationship of Work-Related ICT Use With Well-being, Incorporating the Role of Resources and Demands: A Meta-Analysis," SAGE Open, , vol. 11(4), pages 21582440211, November.
    13. Gundula Krack, 2019. "How to make value-based health insurance designs more effective? A systematic review and meta-analysis," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 20(6), pages 841-856, August.
    14. Angélica Pigola & Priscila Rezende Costa, 2022. "In search of understanding about knowledge and learning on innovation performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 3995-4022, July.
    15. Christopher Snyder & Ran Zhuo, 2018. "Sniff Tests as a Screen in the Publication Process: Throwing out the Wheat with the Chaff," NBER Working Papers 25058, National Bureau of Economic Research, Inc.
    16. Chuang Yuan & Jing Wang & Michael Ying, 2016. "Predictive Value of Carotid Distensibility Coefficient for Cardiovascular Diseases and All-Cause Mortality: A Meta-Analysis," PLOS ONE, Public Library of Science, vol. 11(4), pages 1-15, April.
    17. Francisco Javier Blanco-Encomienda & Rocío García-Cantero & María José Latorre-Medina, 2020. "Association between Work-Related Rumination, Work Environment and Employee Well-Being: A Meta-Analytic Study of Main and Moderator Effects," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 150(3), pages 887-910, August.
    18. Andrew J. Rosman, 2011. "Auditors' going‐concern judgments: rigid, adaptive, or both?," Review of Accounting and Finance, Emerald Group Publishing Limited, vol. 10(1), pages 30-45, February.
    19. Leitner, Johannes & Leopold-Wildburger, Ulrike, 2011. "Experiments on forecasting behavior with several sources of information - A review of the literature," European Journal of Operational Research, Elsevier, vol. 213(3), pages 459-469, September.
    20. Pedro L. Cosio & Manuel Crespo-Posadas & Álvaro Velarde-Sotres & Mireia Pelaez, 2021. "Effect of Chronic Resistance Training on Circulating Irisin: Systematic Review and Meta-Analysis of Randomized Controlled Trials," IJERPH, MDPI, vol. 18(5), pages 1-16, March.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0157914. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.