IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0233625.html
   My bibliography  Save this article

Algorithmic bias in social research: A meta-analysis

Author

Listed:
  • Alrik Thiem
  • Lusine Mkrtchyan
  • Tim Haesebrouck
  • David Sanchez

Abstract

Both the natural and the social sciences are currently facing a deep “reproducibility crisis”. Two important factors in this crisis have been the selective reporting of results and methodological problems. In this article, we examine a fusion of these two factors. More specifically, we demonstrate that the uncritical import of Boolean optimization algorithms from electrical engineering into some areas of the social sciences in the late 1980s has induced algorithmic bias on a considerable scale over the last quarter century. Potentially affected are all studies that have used a method nowadays known as Qualitative Comparative Analysis (QCA). Drawing on replication material for 215 peer-reviewed QCA articles from across 109 high-profile management, political science and sociology journals, we estimate the extent this problem has assumed in empirical work. Our results suggest that one in three studies is affected, one in ten severely so. More generally, our article cautions scientists against letting methods and algorithms travel too easily across disparate disciplines without sufficient prior evaluation of their suitability for the context in hand.

Suggested Citation

  • Alrik Thiem & Lusine Mkrtchyan & Tim Haesebrouck & David Sanchez, 2020. "Algorithmic bias in social research: A meta-analysis," PLOS ONE, Public Library of Science, vol. 15(6), pages 1-16, June.
  • Handle: RePEc:plo:pone00:0233625
    DOI: 10.1371/journal.pone.0233625
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0233625
    Download Restriction: no

    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0233625&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pone.0233625?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    2. Cartwright,Nancy, 2007. "Hunting Causes and Using Them," Cambridge Books, Cambridge University Press, number 9780521860819, January.
    3. Hoover, Kevin D., 1990. "The Logic of Causal Inference: Econometrics and the Conditional Analysis of Causation," Economics and Philosophy, Cambridge University Press, vol. 6(2), pages 207-234, October.
    4. Michael Baumgartner & Alrik Thiem, 2020. "Often Trusted but Never (Properly) Tested: Evaluating Qualitative Comparative Analysis," Sociological Methods & Research, , vol. 49(2), pages 279-311, May.
    5. Martin Jones & Robert Sugden, 2001. "Positive confirmation bias in the acquisition of information," Theory and Decision, Springer, vol. 50(1), pages 59-99, February.
    6. Cartwright,Nancy, 2007. "Hunting Causes and Using Them," Cambridge Books, Cambridge University Press, number 9780521677981, January.
    7. Marcus R. Munafò & Brian A. Nosek & Dorothy V. M. Bishop & Katherine S. Button & Christopher D. Chambers & Nathalie Percie du Sert & Uri Simonsohn & Eric-Jan Wagenmakers & Jennifer J. Ware & John P. A, 2017. "A manifesto for reproducible science," Nature Human Behaviour, Nature, vol. 1(1), pages 1-9, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Akter, Shahriar & Dwivedi, Yogesh K. & Sajib, Shahriar & Biswas, Kumar & Bandara, Ruwan J. & Michael, Katina, 2022. "Algorithmic bias in machine learning-based marketing models," Journal of Business Research, Elsevier, vol. 144(C), pages 201-216.
    2. Mangirdas Morkunas & Elzė Rudienė & Lukas Giriūnas & Laura Daučiūnienė, 2020. "Assessment of Factors Causing Bias in Marketing- Related Publications," Publications, MDPI, vol. 8(4), pages 1-16, October.
    3. Lankoski, Jussi & Thiem, Alrik, 2020. "Linkages between agricultural policies, productivity and environmental sustainability," Ecological Economics, Elsevier, vol. 178(C).
    4. Rafael Quintana, 2023. "Embracing complexity in social science research," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(1), pages 15-38, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Paul Shaffer, 2018. "Causal pluralism and mixed methods in the analysis of poverty dynamics," WIDER Working Paper Series wp-2018-115, World Institute for Development Economic Research (UNU-WIDER).
    2. Priscilla Álamos-Concha & Valérie Pattyn & Benoît Rihoux & Benjamin Schalembier & Derek Beach & Bart Cambré, 2022. "Conservative solutions for progress: on solution types when combining QCA with in-depth Process-Tracing," Quality & Quantity: International Journal of Methodology, Springer, vol. 56(4), pages 1965-1997, August.
    3. Paul Shaffer, 2018. "Causal pluralism and mixed methods in the analysis of poverty dynamics," WIDER Working Paper Series 115, World Institute for Development Economic Research (UNU-WIDER).
    4. Fels, Katja M., 2021. "Who nudges whom? Field experiments with public partners," Ruhr Economic Papers 906, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    5. Lankoski, Jussi & Thiem, Alrik, 2020. "Linkages between agricultural policies, productivity and environmental sustainability," Ecological Economics, Elsevier, vol. 178(C).
    6. Andrew Gelman & Guido Imbens, 2013. "Why ask Why? Forward Causal Inference and Reverse Causal Questions," NBER Working Papers 19614, National Bureau of Economic Research, Inc.
    7. Mouchart, Michel & Russo, Federica & Wunsch, Guillaume, 2011. "Inferring causal relations by modelling structures : Article de recherche," LIDAM Discussion Papers ISBA 2011007, Université catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA).
    8. Galbács Peter, 2021. "What did it take for Lucas to set up ‘useful’ analogue systems in monetary business cycle theory?," Economics and Business Review, Sciendo, vol. 7(3), pages 61-82, September.
    9. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    10. Padró, R. & Marco, I. & Font, C. & Tello, E., 2019. "Beyond Chayanov: A sustainable agroecological farm reproductive analysis of peasant domestic units and rural communities (Sentmenat; Catalonia, 1860)," Ecological Economics, Elsevier, vol. 160(C), pages 227-239.
    11. Bernhard Voelkl & Lucile Vogt & Emily S Sena & Hanno Würbel, 2018. "Reproducibility of preclinical animal research improves with heterogeneity of study samples," PLOS Biology, Public Library of Science, vol. 16(2), pages 1-13, February.
    12. Deirdre Nansen McCloskey, 2013. "Why Economics cannot Explain the Modern World," The Economic Record, The Economic Society of Australia, vol. 89, pages 8-22, June.
    13. Hünermund Paul & Louw Beyers & Caspi Itamar, 2023. "Double machine learning and automated confounder selection: A cautionary tale," Journal of Causal Inference, De Gruyter, vol. 11(1), pages 1-12, January.
    14. Magdalena Osinska, 2011. "On the Interpretation of Causality in Granger’s Sense," Dynamic Econometric Models, Uniwersytet Mikolaja Kopernika, vol. 11, pages 129-140.
    15. Kieran P. Donaghy, 2022. "A Circular Economy Model of Economic Growth with Circular and Cumulative Causation and Trade," Networks and Spatial Economics, Springer, vol. 22(3), pages 461-488, September.
    16. Kieran P. Donaghy, 2014. "Walter Isard’s Evolving Sense of the Scientific in Regional Science," International Regional Science Review, , vol. 37(1), pages 78-95, January.
    17. Gil, Olga, 2022. "Accountability in Artificial Intelligence," SocArXiv wckuf, Center for Open Science.
    18. Mangirdas Morkunas & Elzė Rudienė & Lukas Giriūnas & Laura Daučiūnienė, 2020. "Assessment of Factors Causing Bias in Marketing- Related Publications," Publications, MDPI, vol. 8(4), pages 1-16, October.
    19. Li, Sufang & Tu, Dalun & Zeng, Yan & Gong, Chenggang & Yuan, Di, 2022. "Does geopolitical risk matter in crude oil and stock markets? Evidence from disaggregated data," Energy Economics, Elsevier, vol. 113(C).
    20. George F. DeMartino, 2021. "The specter of irreparable ignorance: counterfactuals and causality in economics," Review of Evolutionary Political Economy, Springer, vol. 2(2), pages 253-276, July.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0233625. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.