IDEAS home Printed from https://ideas.repec.org/a/pal/jintbs/v52y2021i1d10.1057_s41267-020-00385-z.html
   My bibliography  Save this article

The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews

Author

Listed:
  • Piers Steel

    (University of Calgary)

  • Sjoerd Beugelsdijk

    (University of Groningen)

  • Herman Aguinis

    (The George Washington University)

Abstract

Meta-analyses summarize a field’s research base and are therefore highly influential. Despite their value, the standards for an excellent meta-analysis, one that is potentially award-winning, have changed in the last decade. Each step of a meta-analysis is now more formalized, from the identification of relevant articles to coding, moderator analysis, and reporting of results. What was exemplary a decade ago can be somewhat dated today. Using the award-winning meta-analysis by Stahl et al. (Unraveling the effects of cultural diversity in teams: A meta-analysis of research on multicultural work groups. Journal of International Business Studies, 41(4):690–709, 2010) as an exemplar, we adopted a multi-disciplinary approach (e.g., management, psychology, health sciences) to summarize the anatomy (i.e., fundamental components) of a modern meta-analysis, focusing on: (1) data collection (i.e., literature search and screening, coding), (2) data preparation (i.e., treatment of multiple effect sizes, outlier identification and management, publication bias), (3) data analysis (i.e., average effect sizes, heterogeneity of effect sizes, moderator search), and (4) reporting (i.e., transparency and reproducibility, future research directions). In addition, we provide guidelines and a decision-making tree for when even foundational and highly cited meta-analyses should be updated. Based on the latest evidence, we summarize what journal editors and reviewers should expect, authors should provide, and readers (i.e., other researchers, practitioners, and policymakers) should consider about meta-analytic reviews.

Suggested Citation

  • Piers Steel & Sjoerd Beugelsdijk & Herman Aguinis, 2021. "The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 52(1), pages 23-44, February.
    • Handle: RePEc:pal:jintbs:v:52:y:2021:i:1:d:10.1057_s41267-020-00385-z
    DOI: 10.1057/s41267-020-00385-z
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1057/s41267-020-00385-z
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1057/s41267-020-00385-z?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Steel, Piers & Johnson, Jeff W. & Jeanneret, P. Richard & Scherbaum, Charles A. & Hoffman, Calvin C. & Foster, Jeff, 2010. "At Sea With Synthetic Validity," Industrial and Organizational Psychology, Cambridge University Press, vol. 3(3), pages 371-383, September.
    2. Klaus E Meyer & Arjen Witteloostuijn & Sjoerd Beugelsdijk, 2017. "What’s in a p? Reassessing best practices for conducting and reporting hypothesis-testing research," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 48(5), pages 535-551, July.
    3. Benjamin D K Wood & Rui Müller & Annette N Brown, 2018. "Push button replication: Is impact evaluation evidence for international development verifiable?," PLOS ONE, Public Library of Science, vol. 13(12), pages 1-15, December.
    4. Aguinis, Herman & Banks, George C. & Rogelberg, Steven G. & Cascio, Wayne F., 2020. "Actionable recommendations for narrowing the science-practice gap in open science," Organizational Behavior and Human Decision Processes, Elsevier, vol. 158(C), pages 27-35.
    5. Johnson, Blair T. & Hennessy, Emily A., 2019. "Systematic reviews and meta-analyses in the health sciences: Best practice methods for research syntheses," Social Science & Medicine, Elsevier, vol. 233(C), pages 237-251.
    6. Lutz Bornmann & Rüdiger Mutz, 2015. "Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2215-2222, November.
    7. Sjoerd Beugelsdijk & Björn Ambos & Phillip C Nell, 2018. "Conceptualizing and measuring distance in international business research: Recurring questions and best practice guidelines," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 49(9), pages 1113-1137, December.
    8. David Moher & Alessandro Liberati & Jennifer Tetzlaff & Douglas G Altman & The PRISMA Group, 2009. "Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement," PLOS Medicine, Public Library of Science, vol. 6(7), pages 1-6, July.
    9. Tomáš Havránek & T. D. Stanley & Hristos Doucouliagos & Pedro Bom & Jerome Geyer‐Klingeberg & Ichiro Iwasaki & W. Robert Reed & Katja Rost & R. C. M. van Aert, 2020. "Reporting Guidelines For Meta‐Analysis In Economics," Journal of Economic Surveys, Wiley Blackwell, vol. 34(3), pages 469-475, July.
    10. Günter K Stahl & Martha L Maznevski & Andreas Voigt & Karsten Jonsen, 2010. "Unraveling the effects of cultural diversity in teams: A meta-analysis of research on multicultural work groups," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 41(4), pages 690-709, May.
    11. Bowen, Frances E. & Rostami, Mahdi & Steel, Piers, 2010. "Timing is everything: A meta-analysis of the relationships between organizational performance and innovation," Journal of Business Research, Elsevier, vol. 63(11), pages 1179-1185, November.
    12. Robbert Maseland & Douglas Dow & Piers Steel, 2018. "The Kogut and Singh national cultural distance index: Time to start using it as a springboard rather than a crutch," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 49(9), pages 1154-1166, December.
    13. Tatiana Kostova & Sjoerd Beugelsdijk & W. Richard Scott & Vincent E. Kunst & Chei Hwee Chua & Marc Essen, 2020. "The construct of institutional distance through the lens of different institutional perspectives: Review, analysis, and recommendations," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 51(4), pages 467-497, June.
    14. Sjoerd Beugelsdijk & Arjen Witteloostuijn & Klaus E. Meyer, 2020. "A new approach to data access and research transparency (DART)," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 51(6), pages 887-905, August.
    15. Huw T. O. Davies & Sandra M. Nutley & Peter C. Smith, 1999. "Viewpoint: Editorial: What Works? The Role of Evidence in Public Sector Policy and Practice," Public Money & Management, Taylor & Francis Journals, vol. 19(1), pages 3-5, January.
    16. Viechtbauer, Wolfgang, 2010. "Conducting Meta-Analyses in R with the metafor Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 36(i03).
    17. Esther Maassen & Marcel A L M van Assen & Michèle B Nuijten & Anton Olsson-Collentine & Jelte M Wicherts, 2020. "Reproducibility of individual effect sizes in meta-analyses in psychology," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-18, May.
    18. Marcus R. Munafò & Brian A. Nosek & Dorothy V. M. Bishop & Katherine S. Button & Christopher D. Chambers & Nathalie Percie du Sert & Uri Simonsohn & Eric-Jan Wagenmakers & Jennifer J. Ware & John P. A, 2017. "A manifesto for reproducible science," Nature Human Behaviour, Nature, vol. 1(1), pages 1-9, January.
    19. Vas Taras & Piers Steel, 2009. "Beyond Hofstede: Challenging the Ten Commandments of Cross-Cultural Research," Palgrave Macmillan Books, in: Cheryl Nakata (ed.), Beyond Hofstede, chapter 3, pages 40-60, Palgrave Macmillan.
    20. Sjoerd Beugelsdijk & Tatiana Kostova & Vincent E. Kunst & Ettore Spadafora & Marc van Essen, 2018. "Cultural Distance and Firm Internationalization," Post-Print hal-02312065, HAL.
    21. Donald D. Bergh & Herman Aguinis & Ciaran Heavey & David J. Ketchen & Brian K. Boyd & Peiran Su & Cubie L. L. Lau & Harry Joo, 2016. "Using meta-analytic structural equation modeling to advance strategic management research: Guidelines and an empirical illustration via the strategic leadership-performance relationship," Strategic Management Journal, Wiley Blackwell, vol. 37(3), pages 477-497, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tatiana Kostova & Sjoerd Beugelsdijk, 2021. "Integrating Diversity into Distance Research for Added Rigor, Parsimony, and Relevance," Journal of Management Studies, Wiley Blackwell, vol. 58(6), pages 1669-1689, September.
    2. Sjoerd Beugelsdijk, 2022. "Capitalizing on the uniqueness of international business: Towards a theory of place, space, and organization," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 53(9), pages 2050-2067, December.
    3. Tatiana Kostova & Sjoerd Beugelsdijk & W. Richard Scott & Vincent E. Kunst & Chei Hwee Chua & Marc Essen, 2020. "The construct of institutional distance through the lens of different institutional perspectives: Review, analysis, and recommendations," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 51(4), pages 467-497, June.
    4. Christian Theres & Stefan Strohmeier, 2024. "Consolidating the theoretical foundations of digital human resource management acceptance and use research: a meta-analytic validation of UTAUT," Management Review Quarterly, Springer, vol. 74(4), pages 2683-2715, December.
    5. Zhang, Wei & He, Xinming & Wang, Tao & Wang, Kang, 2023. "Institutional distance and the international market entry mode: A meta-analysis," Journal of International Management, Elsevier, vol. 29(1).
    6. Sasa Ding & Frank McDonald & Yingqi Wei, 2021. "Is Internationalization Beneficial to Innovation? Evidence from a Meta-analysis," Management International Review, Springer, vol. 61(4), pages 469-519, August.
    7. Francesca Talamini & Gianmarco Altoè & Barbara Carretti & Massimo Grassi, 2017. "Musicians have better memory than nonmusicians: A meta-analysis," PLOS ONE, Public Library of Science, vol. 12(10), pages 1-21, October.
    8. Palitha Konara & Alexander Mohr, 2019. "Why We Should Stop Using the Kogut and Singh Index," Management International Review, Springer, vol. 59(3), pages 335-354, June.
    9. Miriam Flickinger & Miriam Zschoche, 2023. "The Interplay of Subsidiary Performance and Cultural Distance in International Downsizing Decisions," Schmalenbach Journal of Business Research, Springer, vol. 75(2), pages 127-147, June.
    10. Christopher Hansen & Holger Steinmetz & Jörn Block, 2022. "How to conduct a meta-analysis in eight steps: a practical guide," Management Review Quarterly, Springer, vol. 72(1), pages 1-19, February.
    11. Moalla, Emna & Mayrhofer, Ulrike, 2020. "How does distance affect market entry mode choice? Evidence from French companies," European Management Journal, Elsevier, vol. 38(1), pages 135-145.
    12. Esther Maassen & Marcel A L M van Assen & Michèle B Nuijten & Anton Olsson-Collentine & Jelte M Wicherts, 2020. "Reproducibility of individual effect sizes in meta-analyses in psychology," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-18, May.
    13. Schmid, David & Morschett, Dirk, 2020. "Decades of research on foreign subsidiary divestment: What do we really know about its antecedents?," International Business Review, Elsevier, vol. 29(4).
    14. Jeoung Yul Lee & Vasyl Taras & Alfredo Jiménez & Byungchul Choi & Chinmay Pattnaik, 2020. "Ambidextrous Knowledge Sharing within R&D Teams and Multinational Enterprise Performance: The Moderating Effects of Cultural Distance in Uncertainty Avoidance," Management International Review, Springer, vol. 60(3), pages 387-425, June.
    15. Oded Shenkar & Stephen B Tallman & Hao Wang & Jie Wu, 2022. "National culture and international business: A path forward," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 53(3), pages 516-533, April.
    16. Simon Hartmann & Thomas Lindner & Jakob Müllner & Jonas Puck, 2022. "Beyond the nation-state: Anchoring supranational institutions in international business research," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 53(6), pages 1282-1306, August.
    17. Augusteijn, Hilde Elisabeth Maria & van Aert, Robbie Cornelis Maria & van Assen, Marcel A. L. M., 2021. "Posterior Probabilities of Effect Sizes and Heterogeneity in Meta-Analysis: An Intuitive Approach of Dealing with Publication Bias," OSF Preprints avkgj, Center for Open Science.
    18. Gechert, Sebastian & Heimberger, Philipp, 2022. "Do corporate tax cuts boost economic growth?," European Economic Review, Elsevier, vol. 147(C).
    19. Tang, Ryan W., 2023. "Institutional unpredictability and foreign exit−reentry dynamics: The moderating role of foreign ownership," Journal of World Business, Elsevier, vol. 58(2).
    20. Mattia Filomena & Matteo Picchio, 2023. "Retirement and health outcomes in a meta‐analytical framework," Journal of Economic Surveys, Wiley Blackwell, vol. 37(4), pages 1120-1155, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pal:jintbs:v:52:y:2021:i:1:d:10.1057_s41267-020-00385-z. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.palgrave-journals.com/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.