IDEAS home Printed from https://ideas.repec.org/a/inm/orisre/v32y2021i3p713-735.html
   My bibliography  Save this article

Augmenting Medical Diagnosis Decisions? An Investigation into Physicians’ Decision-Making Process with Artificial Intelligence

Author

Listed:
  • Ekaterina Jussupow

    (Business School, Area Information Systems, Chair of General Management and Information Systems, University of Mannheim, 68161 Mannheim, Germany)

  • Kai Spohrer

    (Business School, Area Information Systems, Chair of General Management and Information Systems, University of Mannheim, 68161 Mannheim, Germany)

  • Armin Heinzl

    (Business School, Area Information Systems, Chair of General Management and Information Systems, University of Mannheim, 68161 Mannheim, Germany)

  • Joshua Gawlitza

    (Institute of Diagnostic and Interventional Radiology, Thoracic Imaging, University Hospital Rechts der Isar, Technical University Munich, 81675 Munich, Germany)

Abstract

Systems based on artificial intelligence (AI) increasingly support physicians in diagnostic decisions. Compared with rule-based systems, however, these systems are less transparent and their errors less predictable. Much research currently aims to improve AI technologies and debates their societal implications. Surprisingly little effort is spent on understanding the cognitive challenges of decision augmentation with AI-based systems although these systems make it more difficult for decision makers to evaluate the correctness of system advice and to decide whether to reject or accept it. As little is known about the cognitive mechanisms that underlie such evaluations, we take an inductive approach to understand how AI advice influences physicians’ decision-making process. We conducted experiments with a total of 68 novice and 12 experienced physicians who diagnosed patient cases with an AI-based system that provided both correct and incorrect advice. Based on qualitative data from think-aloud protocols, interviews, and questionnaires, we elicit five decision-making patterns and develop a process model of medical diagnosis decision augmentation with AI advice. We show that physicians use second-order cognitive processes, namely metacognitions, to monitor and control their reasoning while assessing AI advice. These metacognitions determine whether physicians are able to reap the full benefits of AI or not. Specifically, wrong diagnostic decisions often result from shortcomings in utilizing metacognitions related to decision makers’ own reasoning (self-monitoring) and metacognitions related to the AI-based system (system monitoring). As a result, physicians fall for decisions based on beliefs rather than actual data or engage in unsuitably superficial information search. Our findings provide a first perspective on the metacognitive mechanisms that decision makers use to evaluate system advice. Overall, our study sheds light on an overlooked facet of decision augmentation with AI, namely, the crucial role of human actors in compensating for technological errors.

Suggested Citation

  • Ekaterina Jussupow & Kai Spohrer & Armin Heinzl & Joshua Gawlitza, 2021. "Augmenting Medical Diagnosis Decisions? An Investigation into Physicians’ Decision-Making Process with Artificial Intelligence," Information Systems Research, INFORMS, vol. 32(3), pages 713-735, September.
    • Handle: RePEc:inm:orisre:v:32:y:2021:i:3:p:713-735
    DOI: 10.1287/isre.2020.0980
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/isre.2020.0980
    Download Restriction: no
    File URL: https://libkey.io/10.1287/isre.2020.0980?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Chuan-Hoo Tan & Hock-Hai Teo & Izak Benbasat, 2010. "Assessing Screening and Evaluation Decision Support Systems: A Resource-Matching Approach," Information Systems Research, INFORMS, vol. 21(2), pages 305-326, June.
    2. Chiara Longoni & Andrea Bonezzi & Carey K Morewedge, 2019. "Resistance to Medical Artificial Intelligence," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 46(4), pages 629-650.
    3. Diana J. Burgess, 2010. "Are Providers More Likely to Contribute to Healthcare Disparities Under High Levels of Cognitive Load? How Features of the Healthcare Setting May Lead to Biases in Medical Decision Making," Medical Decision Making, , vol. 30(2), pages 246-257, March.
    4. Arun Rai, 2020. "Explainable AI: from black box to glass box," Journal of the Academy of Marketing Science, Springer, vol. 48(1), pages 137-141, January.
    5. Gawlitza, Joshua & Sturm, Timo & Spohrer, Kai & Henzler, Thomas & Akin, Ibrahim & Schönberg, Stefan & Borggrefe, Martin & Haubenreisser, Holger & Trinkmann, Frederik, 2019. "Predicting Pulmonary Function Testing from Quantified Computed Tomography Using Machine Learning Algorithms in Patients with COPD," Publications of Darmstadt Technical University, Institute for Business Studies (BWL) 113226, Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL).
    6. Peter Todd & Izak Benbasat, 1991. "An Experimental Investigation of the Impact of Computer Based Decision Aids on Decision Making Strategies," Information Systems Research, INFORMS, vol. 2(2), pages 87-115, June.
    7. Logg, Jennifer M. & Minson, Julia A. & Moore, Don A., 2019. "Algorithm appreciation: People prefer algorithmic to human judgment," Organizational Behavior and Human Decision Processes, Elsevier, vol. 151(C), pages 90-103.
    8. Jon Kleinberg & Himabindu Lakkaraju & Jure Leskovec & Jens Ludwig & Sendhil Mullainathan, 2018. "Human Decisions and Machine Predictions," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 133(1), pages 237-293.
    9. Mehmet Eren Ahsen & Mehmet Ulvi Saygi Ayvaci & Srinivasan Raghunathan, 2019. "When Algorithmic Predictions Use Human-Generated Data: A Bias-Aware Classification Algorithm for Breast Cancer Diagnosis," Service Science, INFORMS, vol. 30(1), pages 97-116, March.
    10. Bo Xiao & Izak Benbasat, 2015. "Designing Warning Messages for Detecting Biased Online Product Recommendations: An Empirical Investigation," Information Systems Research, INFORMS, vol. 26(4), pages 793-811, December.
    11. Gediminas Adomavicius & Jesse C. Bockstedt & Shawn P. Curley & Jingjing Zhang, 2013. "Do Recommender Systems Manipulate Consumer Preferences? A Study of Anchoring Effects," Information Systems Research, INFORMS, vol. 24(4), pages 956-975, December.
    12. Berkeley J. Dietvorst & Joseph P. Simmons & Cade Massey, 2018. "Overcoming Algorithm Aversion: People Will Use Imperfect Algorithms If They Can (Even Slightly) Modify Them," Management Science, INFORMS, vol. 64(3), pages 1155-1170, March.
    13. repec:cup:judgdm:v:3:y:2008:i::p:229-243 is not listed on IDEAS
    14. Bonaccio, Silvia & Dalal, Reeshad S., 2006. "Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences," Organizational Behavior and Human Decision Processes, Elsevier, vol. 101(2), pages 127-151, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Erdem Dogukan Yilmaz & Christian Peukert, 2024. "Who Benefits from AI? Project-Level Evidence on Labor Demand, Operations and Profitability," CESifo Working Paper Series 11321, CESifo.
    2. Mi, Yunlong & Wang, Zongrun & Quan, Pei & Shi, Yong, 2024. "A semi-supervised concept-cognitive computing system for dynamic classification decision making with limited feedback information," European Journal of Operational Research, Elsevier, vol. 315(3), pages 1123-1138.
    3. Christoph Riedl & Eric Bogert, 2024. "Effects of AI Feedback on Learning, the Skill Gap, and Intellectual Diversity," Papers 2409.18660, arXiv.org.
    4. Kevin Bauer & Moritz von Zahn & Oliver Hinz, 2023. "Expl(AI)ned: The Impact of Explainable Artificial Intelligence on Users’ Information Processing," Information Systems Research, INFORMS, vol. 34(4), pages 1582-1602, December.
    5. Sullivan, Yulia & Fosso Wamba, Samuel, 2024. "Artificial intelligence and adaptive response to market changes: A strategy to enhance firm performance and innovation," Journal of Business Research, Elsevier, vol. 174(C).
    6. Kevin Bauer & Andrej Gill, 2024. "Mirror, Mirror on the Wall: Algorithmic Assessments, Transparency, and Self-Fulfilling Prophecies," Information Systems Research, INFORMS, vol. 35(1), pages 226-248, March.
    7. Goutier, Marc & Diebel, Christopher & Adam, Martin & Benlian, Alexander, 2024. "Proactive and Reactive Help from Intelligent Agents in Identity-Relevant Tasks," Publications of Darmstadt Technical University, Institute for Business Studies (BWL) 142985, Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL).
    8. Tiwari, Manisha & Bryde, David J. & Stavropoulou, Foteini & Dubey, Rameshwar & Kumari, Sushma & Foropon, Cyril, 2024. "Modelling supply chain Visibility, digital Technologies, environmental dynamism and healthcare supply chain Resilience: An organisation information processing theory perspective," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 188(C).
    9. Singh, Nidhi & Jain, Monika & Kamal, Muhammad Mustafa & Bodhi, Rahul & Gupta, Bhumika, 2024. "Technological paradoxes and artificial intelligence implementation in healthcare. An application of paradox theory," Technological Forecasting and Social Change, Elsevier, vol. 198(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kevin Bauer & Andrej Gill, 2024. "Mirror, Mirror on the Wall: Algorithmic Assessments, Transparency, and Self-Fulfilling Prophecies," Information Systems Research, INFORMS, vol. 35(1), pages 226-248, March.
    2. Keding, Christoph & Meissner, Philip, 2021. "Managerial overreliance on AI-augmented decision-making processes: How the use of AI-based advisory systems shapes choice behavior in R&D investment decisions," Technological Forecasting and Social Change, Elsevier, vol. 171(C).
    3. Chugunova, Marina & Sele, Daniela, 2022. "We and It: An interdisciplinary review of the experimental evidence on how humans interact with machines," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 99(C).
    4. Pascal Oliver Heßler & Jella Pfeiffer & Sebastian Hafenbrädl, 2022. "When Self-Humanization Leads to Algorithm Aversion," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 64(3), pages 275-292, June.
    5. Bauer, Kevin & von Zahn, Moritz & Hinz, Oliver, 2022. "Expl(AI)ned: The impact of explainable Artificial Intelligence on cognitive processes," SAFE Working Paper Series 315, Leibniz Institute for Financial Research SAFE, revised 2022.
    6. Zhu, Yimin & Zhang, Jiemin & Wu, Jifei & Liu, Yingyue, 2022. "AI is better when I'm sure: The influence of certainty of needs on consumers' acceptance of AI chatbots," Journal of Business Research, Elsevier, vol. 150(C), pages 642-652.
    7. Merle, Aurélie & St-Onge, Anik & Sénécal, Sylvain, 2022. "Does it pay to be honest? The effect of retailer-provided negative feedback on consumers’ product choice and shopping experience," Journal of Business Research, Elsevier, vol. 147(C), pages 532-543.
    8. Benedikt Berger & Martin Adam & Alexander Rühr & Alexander Benlian, 2021. "Watch Me Improve—Algorithm Aversion and Demonstrating the Ability to Learn," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(1), pages 55-68, February.
    9. repec:cup:judgdm:v:15:y:2020:i:3:p:449-451 is not listed on IDEAS
    10. Scott Schanke & Gordon Burtch & Gautam Ray, 2021. "Estimating the Impact of “Humanizing” Customer Service Chatbots," Information Systems Research, INFORMS, vol. 32(3), pages 736-751, September.
    11. Peng, Leiqing & Luo, Mengting & Guo, Yulang, 2023. "Deposit AI as the “invisible hand†to make the resale easier: A moderated mediation model," Journal of Retailing and Consumer Services, Elsevier, vol. 75(C).
    12. Christiane B. Haubitz & Cedric A. Lehmann & Andreas Fügener & Ulrich W. Thonemann, 2021. "The Risk of Algorithm Transparency: How Algorithm Complexity Drives the Effects on Use of Advice," ECONtribute Discussion Papers Series 078, University of Bonn and University of Cologne, Germany.
    13. Bauer, Kevin & Gill, Andrej, 2021. "Mirror, mirror on the wall: Machine predictions and self-fulfilling prophecies," SAFE Working Paper Series 313, Leibniz Institute for Financial Research SAFE.
    14. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    15. Said Kaawach & Oskar Kowalewski & Oleksandr Talavera, 2023. "Automatic vs Manual Investing: Role of Past Performance," Discussion Papers 23-04, Department of Economics, University of Birmingham.
    16. Sarah Spiekermann & Hanna Krasnova & Oliver Hinz & Annika Baumann & Alexander Benlian & Henner Gimpel & Irina Heimbach & Antonia Köster & Alexander Maedche & Björn Niehaves & Marten Risius & Manuel Tr, 2022. "Values and Ethics in Information Systems," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 64(2), pages 247-264, April.
    17. Martin Adam & Konstantin Roethke & Alexander Benlian, 2023. "Human vs. Automated Sales Agents: How and Why Customer Responses Shift Across Sales Stages," Information Systems Research, INFORMS, vol. 34(3), pages 1148-1168, September.
    18. Fumagalli, Elena & Rezaei, Sarah & Salomons, Anna, 2022. "OK computer: Worker perceptions of algorithmic recruitment," Research Policy, Elsevier, vol. 51(2).
    19. Wang, Cuicui & Li, Yiyang & Fu, Weizhong & Jin, Jia, 2023. "Whether to trust chatbots: Applying the event-related approach to understand consumers’ emotional experiences in interactions with chatbots in e-commerce," Journal of Retailing and Consumer Services, Elsevier, vol. 73(C).
    20. Bansak, Kirk & Paulson, Elisabeth, 2023. "Public Opinion on Fairness and Efficiency for Algorithmic and Human Decision-Makers," OSF Preprints pghmx, Center for Open Science.
    21. Hermann, Erik & Puntoni, Stefano, 2024. "Artificial intelligence and consumer behavior: From predictive to generative AI," Journal of Business Research, Elsevier, vol. 180(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:orisre:v:32:y:2021:i:3:p:713-735. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.