IDEAS home Printed from https://ideas.repec.org/p/esm/wpaper/esmt-22-02.html
   My bibliography  Save this paper

Is your machine better than you? You may never know

Author

Listed:
  • Francis de Véricourt

    (ESMT European School of Management and Technology GmbH)

  • Huseyin Gurkan

    (ESMT European School of Management and Technology GmbH)

Abstract

Artificial intelligence systems are increasingly demonstrating their capacity to make better predictions than human experts. Yet, recent studies suggest that professionals sometimes doubt the quality of these systems and overrule machine-based prescriptions. This paper explores the extent to which a decision maker (DM) supervising a machine to make high-stake decisions can properly assess whether the machine produces better recommendations. To that end, we study a set-up, in which a machine performs repeated decision tasks (e.g., whether to perform a biopsy) under the DM’s supervision. Because stakes are high, the DM primarily focuses on making the best choice for the task at hand. Nonetheless, as the DM observes the correctness of the machine’s prescriptions across tasks, she updates her belief about the machine. However, the DM observes the machine’s correctness only if she ultimately decides to act on the task. Further, the DM sometimes overrides the machine depending on her belief, which affects learning. In this set-up, we characterize the evolution of the DM’s belief and overruling decisions over time. We identify situations under which the DM hesitates forever whether the machine is better, i.e., she never fully ignores but regularly overrules it. Moreover, the DM sometimes wrongly believes with positive probability that the machine is better. We fully characterize the conditions under which these learning failures occur and explore how mistrusting the machine affects them. Our results highlight some fundamental limitations in determining whether machines make better decisions than experts and provide a novel explanation for human-machine complementarity.

Suggested Citation

  • Francis de Véricourt & Huseyin Gurkan, 2022. "Is your machine better than you? You may never know," ESMT Research Working Papers ESMT-22-02, ESMT European School of Management and Technology.
  • Handle: RePEc:esm:wpaper:esmt-22-02
    as

    Download full text from publisher

    File URL: http://static.esmt.org/publications/workingpapers/ESMT-22-02.pdf
    File Function: First version, 2022
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Fudenberg, Drew & Romanyuk, Gleb & Strack, Philipp, 2017. "Active learning with a misspecified prior," Theoretical Economics, Econometric Society, vol. 12(3), September.
    2. Karel H. van Donselaar & Vishal Gaur & Tom van Woensel & Rob A. C. M. Broekmeulen & Jan C. Fransoo, 2010. "Ordering Behavior in Retail Stores and Implications for Automated Replenishment," Management Science, INFORMS, vol. 56(5), pages 766-784, May.
    3. Alexander Coutts, 2019. "Good news and bad news are still news: experimental evidence on belief updating," Experimental Economics, Springer;Economic Science Association, vol. 22(2), pages 369-395, June.
    4. J. Michael Harrison & N. Bora Keskin & Assaf Zeevi, 2012. "Bayesian Dynamic Pricing Policies: Learning and Earning Under a Binary Prior Distribution," Management Science, INFORMS, vol. 58(3), pages 570-586, March.
    5. Berkeley J. Dietvorst & Joseph P. Simmons & Cade Massey, 2018. "Overcoming Algorithm Aversion: People Will Use Imperfect Algorithms If They Can (Even Slightly) Modify Them," Management Science, INFORMS, vol. 64(3), pages 1155-1170, March.
    6. Tamer Boyaci, & Caner Canyakmaz, & Francis de Véricourt,, 2020. "Human and machine: The impact of machine input on decision-making under cognitive limitations," ESMT Research Working Papers ESMT-20-02, ESMT European School of Management and Technology.
    7. Omar Besbes & Assaf Zeevi, 2009. "Dynamic Pricing Without Knowing the Demand Function: Risk Bounds and Near-Optimal Algorithms," Operations Research, INFORMS, vol. 57(6), pages 1407-1420, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. William L. Cooper & Tito Homem-de-Mello & Anton J. Kleywegt, 2015. "Learning and Pricing with Models That Do Not Explicitly Incorporate Competition," Operations Research, INFORMS, vol. 63(1), pages 86-103, February.
    2. Xiao, Baichun & Yang, Wei, 2021. "A Bayesian learning model for estimating unknown demand parameter in revenue management," European Journal of Operational Research, Elsevier, vol. 293(1), pages 248-262.
    3. N. Bora Keskin & Assaf Zeevi, 2014. "Dynamic Pricing with an Unknown Demand Model: Asymptotically Optimal Semi-Myopic Policies," Operations Research, INFORMS, vol. 62(5), pages 1142-1167, October.
    4. Thomas Loots & Arnoud V. den Boer, 2023. "Data‐driven collusion and competition in a pricing duopoly with multinomial logit demand," Production and Operations Management, Production and Operations Management Society, vol. 32(4), pages 1169-1186, April.
    5. Josef Broder & Paat Rusmevichientong, 2012. "Dynamic Pricing Under a General Parametric Choice Model," Operations Research, INFORMS, vol. 60(4), pages 965-980, August.
    6. Yash Kanoria & Hamid Nazerzadeh, 2020. "Dynamic Reserve Prices for Repeated Auctions: Learning from Bids," Papers 2002.07331, arXiv.org.
    7. Huashuai Qu & Ilya O. Ryzhov & Michael C. Fu & Eric Bergerson & Megan Kurka & Ludek Kopacek, 2020. "Learning Demand Curves in B2B Pricing: A New Framework and Case Study," Production and Operations Management, Production and Operations Management Society, vol. 29(5), pages 1287-1306, May.
    8. Yuqing Zhang & Neil Walton, 2019. "Adaptive Pricing in Insurance: Generalized Linear Models and Gaussian Process Regression Approaches," Papers 1907.05381, arXiv.org.
    9. Stefanus Jasin, 2014. "Reoptimization and Self-Adjusting Price Control for Network Revenue Management," Operations Research, INFORMS, vol. 62(5), pages 1168-1178, October.
    10. Mira Frick & Ryota Iijima & Yuhta Ishii, 2021. "Welfare Comparisons for Biased Learning," Cowles Foundation Discussion Papers 2274, Cowles Foundation for Research in Economics, Yale University.
    11. Gur, Yonatan & Macnamara, Gregory & Saban, Daniela, 2020. "On the Disclosure of Promotion Value in Platforms with Learning Sellers," Research Papers 3865, Stanford University, Graduate School of Business.
    12. Yang, Chaolin & Xiong, Yi, 2020. "Nonparametric advertising budget allocation with inventory constraint," European Journal of Operational Research, Elsevier, vol. 285(2), pages 631-641.
    13. den Boer, Arnoud V., 2015. "Tracking the market: Dynamic pricing and learning in a changing environment," European Journal of Operational Research, Elsevier, vol. 247(3), pages 914-927.
    14. Ruben Geer & Arnoud V. Boer & Christopher Bayliss & Christine S. M. Currie & Andria Ellina & Malte Esders & Alwin Haensel & Xiao Lei & Kyle D. S. Maclean & Antonio Martinez-Sykora & Asbjørn Nilsen Ris, 2019. "Dynamic pricing and learning with competition: insights from the dynamic pricing challenge at the 2017 INFORMS RM & pricing conference," Journal of Revenue and Pricing Management, Palgrave Macmillan, vol. 18(3), pages 185-203, June.
    15. Omar Besbes & Assaf Zeevi, 2015. "On the (Surprising) Sufficiency of Linear Models for Dynamic Pricing with Demand Learning," Management Science, INFORMS, vol. 61(4), pages 723-739, April.
    16. N. Bora Keskin & Assaf Zeevi, 2017. "Chasing Demand: Learning and Earning in a Changing Environment," Mathematics of Operations Research, INFORMS, vol. 42(2), pages 277-307, May.
    17. Arnoud V. den Boer & Bert Zwart, 2014. "Simultaneously Learning and Optimizing Using Controlled Variance Pricing," Management Science, INFORMS, vol. 60(3), pages 770-783, March.
    18. Wang Chi Cheung & David Simchi-Levi & He Wang, 2017. "Technical Note—Dynamic Pricing and Demand Learning with Limited Price Experimentation," Operations Research, INFORMS, vol. 65(6), pages 1722-1731, December.
    19. Arnoud V. den Boer & N. Bora Keskin, 2020. "Discontinuous Demand Functions: Estimation and Pricing," Management Science, INFORMS, vol. 66(10), pages 4516-4534, October.
    20. Benjamin R. Handel & Kanishka Misra, 2015. "Robust New Product Pricing," Marketing Science, INFORMS, vol. 34(6), pages 864-881, November.

    More about this item

    Keywords

    machine accuracy; decision making; human-in-the-loop; algorithm aversion; dynamic learning;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:esm:wpaper:esmt-22-02. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ESMT Faculty Publications (email available below). General contact details of provider: https://edirc.repec.org/data/emstbde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.