IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2410.06932.html
   My bibliography  Save this paper

Reproducing and Extending Experiments in Behavioral Strategy with Large Language Models

Author

Listed:
  • Daniel Albert
  • Stephan Billinger

Abstract

In this study, we propose LLM agents as a novel approach in behavioral strategy research, complementing simulations and laboratory experiments to advance our understanding of cognitive processes in decision-making. Specifically, we reproduce a human laboratory experiment in behavioral strategy using large language model (LLM) generated agents and investigate how LLM agents compare to observed human behavior. Our results show that LLM agents effectively reproduce search behavior and decision-making comparable to humans. Extending our experiment, we analyze LLM agents' simulated "thoughts," discovering that more forward-looking thoughts correlate with favoring exploitation over exploration to maximize wealth. We show how this new approach can be leveraged in behavioral strategy research and address limitations.

Suggested Citation

  • Daniel Albert & Stephan Billinger, 2024. "Reproducing and Extending Experiments in Behavioral Strategy with Large Language Models," Papers 2410.06932, arXiv.org.
    • Handle: RePEc:arx:papers:2410.06932
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2410.06932
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Richard A. Bettis & Constance E. Helfat & J. Myles Shaver & Richard A. Bettis & Constance E. Helfat & J. Myles Shaver, 2016. "The necessity, logic, and forms of replication," Strategic Management Journal, Wiley Blackwell, vol. 37(11), pages 2193-2203, November.
    2. Giovanni Gavetti & Daniel A. Levinthal & Jan W. Rivkin, 2005. "Strategy making in novel and complex worlds: the power of analogy," Strategic Management Journal, Wiley Blackwell, vol. 26(8), pages 691-712, August.
    3. James G. March, 1991. "Exploration and Exploitation in Organizational Learning," Organization Science, INFORMS, vol. 2(1), pages 71-87, February.
    4. Farid Anvari & Stephan Billinger & Pantelis P. Analytis & Vithor Rosa Franco & Davide Marchiori, 2024. "Testing the convergent validity, domain generality, and temporal stability of selected measures of people’s tendency to explore," Nature Communications, Nature, vol. 15(1), pages 1-23, December.
    5. Michael J. Leiblein & Jeffrey J. Reuer & Todd Zenger, 2018. "What Makes a Decision Strategic?," Strategy Science, INFORMS, vol. 3(4), pages 558-573, December.
    6. Heckman, James, 2013. "Sample selection bias as a specification error," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 31(3), pages 129-137.
    7. Meissner, Philip & Poensgen, Christian & Wulf, Torsten, 2021. "How hot cognition can lead us astray: The effect of anger on strategic decision making," European Management Journal, Elsevier, vol. 39(4), pages 434-444.
    8. Sheen S. Levine & Mark Bernard & Rosemarie Nagel, 2018. "Strategic intelligence: The cognitive capability to anticipate competitor behaviour," Strategic Management Journal, Wiley Blackwell, vol. 39(2), pages 527-527, February.
    9. Lisa Messeri & M. J. Crockett, 2024. "Artificial intelligence and illusions of understanding in scientific research," Nature, Nature, vol. 627(8002), pages 49-58, March.
    10. Sarah Kaplan, 2011. "Research in Cognition and Strategy: Reflections on Two Decades of Progress and a Look to the Future," Journal of Management Studies, Wiley Blackwell, vol. 48(3), pages 665-695, May.
    11. Daniella Laureiro‐Martínez & Stefano Brusoni, 2018. "Cognitive flexibility and adaptive decision‐making: Evidence from a laboratory study of expert decision makers," Strategic Management Journal, Wiley Blackwell, vol. 39(4), pages 1031-1058, April.
    12. John J. Horton, 2023. "Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?," NBER Working Papers 31122, National Bureau of Economic Research, Inc.
    13. Jan W. Rivkin, 2000. "Imitation of Complex Strategies," Management Science, INFORMS, vol. 46(6), pages 824-844, June.
    14. John J. Horton, 2023. "Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?," Papers 2301.07543, arXiv.org.
    15. Peiyao Li & Noah Castelo & Zsolt Katona & Miklos Sarvary, 2024. "Frontiers: Determining the Validity of Large Language Models for Automated Perceptual Analysis," Marketing Science, INFORMS, vol. 43(2), pages 254-266, March.
    16. William Ocasio, 2011. "Attention to Attention," Organization Science, INFORMS, vol. 22(5), pages 1286-1296, October.
    17. Daniel A. Levinthal, 1997. "Adaptation on Rugged Landscapes," Management Science, INFORMS, vol. 43(7), pages 934-950, July.
    18. William M. Tracy & Dmitri G. Markovitch & Lois S. Peters & B. V. Phani & Deepu Philip, 2017. "Algorithmic Representations of Managerial Search Behavior," Computational Economics, Springer;Society for Computational Economics, vol. 49(3), pages 343-361, March.
    19. Felipe A. Csaszar & Nicolaj Siggelkow, 2010. "How Much to Copy? Determinants of Effective Imitation Breadth," Organization Science, INFORMS, vol. 21(3), pages 661-676, June.
    20. Felipe A. Csaszar & Daniel A. Levinthal, 2016. "Mental representation and the discovery of new strategies," Strategic Management Journal, Wiley Blackwell, vol. 37(10), pages 2031-2049, October.
    21. Giovanni Gavetti & Jan W. Rivkin, 2007. "On the Origin of Strategy: Action and Cognition over Time," Organization Science, INFORMS, vol. 18(3), pages 420-439, June.
    22. Michael J. Lenox & Scott F. Rockart & Arie Y. Lewin, 2006. "Interdependency, Competition, and the Distribution of Firm and Industry Profits," Management Science, INFORMS, vol. 52(5), pages 757-772, May.
    23. Stephan Billinger & Nils Stieglitz & Terry R. Schumacher, 2014. "Search on Rugged Landscapes: An Experimental Study," Organization Science, INFORMS, vol. 25(1), pages 93-108, February.
    24. J. P. Eggers & Sarah Kaplan, 2009. "Cognition and Renewal: Comparing CEO and Organizational Effects on Incumbent Adaptation to Technical Change," Organization Science, INFORMS, vol. 20(2), pages 461-477, April.
    25. Dongil D. Keum & Kelly E. See, 2017. "The Influence of Hierarchy on Idea Generation and Selection in the Innovation Process," Organization Science, INFORMS, vol. 28(4), pages 653-669, August.
    26. Stephan Billinger & Kannan Srikanth & Nils Stieglitz & Terry R. Schumacher, 2021. "Exploration and exploitation in complex search tasks: How feedback influences whether and where human agents search," Strategic Management Journal, Wiley Blackwell, vol. 42(2), pages 361-385, February.
    27. James W. A. Strachan & Dalila Albergo & Giulia Borghini & Oriana Pansardi & Eugenio Scaliti & Saurabh Gupta & Krati Saxena & Alessandro Rufo & Stefano Panzeri & Guido Manzi & Michael S. A. Graziano & , 2024. "Testing theory of mind in large language models and humans," Nature Human Behaviour, Nature, vol. 8(7), pages 1285-1295, July.
    28. William Ocasio, 1997. "Towards An Attention‐Based View Of The Firm," Strategic Management Journal, Wiley Blackwell, vol. 18(S1), pages 187-206, July.
    29. Juanjuan Meng, 2024. "AI emerges as the frontier in behavioral science," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 121(10), pages 2401336121-, March.
    30. Qiaozhu Mei & Yutong Xie & Walter Yuan & Matthew O. Jackson, 2024. "A Turing test of whether AI chatbots are behaviorally similar to humans," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 121(9), pages 2313925121-, February.
    31. John Joseph & Alex J. Wilson, 2018. "The growth of the firm: An attention‐based view," Strategic Management Journal, Wiley Blackwell, vol. 39(6), pages 1779-1800, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Stephan Billinger & Kannan Srikanth & Nils Stieglitz & Terry R. Schumacher, 2021. "Exploration and exploitation in complex search tasks: How feedback influences whether and where human agents search," Strategic Management Journal, Wiley Blackwell, vol. 42(2), pages 361-385, February.
    2. Michael J. Leiblein & Jeffrey J. Reuer & Todd Zenger, 2018. "What Makes a Decision Strategic?," Strategy Science, INFORMS, vol. 3(4), pages 558-573, December.
    3. Cha Li & Felipe A. Csaszar, 2019. "Government as Landscape Designer: A Behavioral View of Industrial Policy," Strategy Science, INFORMS, vol. 4(3), pages 175-192, September.
    4. Felipe A. Csaszar, 2018. "A note on how NK landscapes work," Journal of Organization Design, Springer;Organizational Design Community, vol. 7(1), pages 1-6, December.
    5. Isin Guler, 2018. "Pulling the Plug: The Capability to Terminate Unsuccessful Projects and Firm Performance," Strategy Science, INFORMS, vol. 3(3), pages 481-497, September.
    6. Ganco, Martin, 2017. "NK model as a representation of innovative search," Research Policy, Elsevier, vol. 46(10), pages 1783-1800.
    7. Felipe A. Csaszar & Daniel A. Levinthal, 2016. "Mental representation and the discovery of new strategies," Strategic Management Journal, Wiley Blackwell, vol. 37(10), pages 2031-2049, October.
    8. Oliver Baumann, 2015. "Models of complex adaptive systems in strategy and organization research," Mind & Society: Cognitive Studies in Economics and Social Sciences, Springer;Fondazione Rosselli, vol. 14(2), pages 169-183, November.
    9. Giovanni Gavetti, 2012. "PERSPECTIVE—Toward a Behavioral Theory of Strategy," Organization Science, INFORMS, vol. 23(1), pages 267-285, February.
    10. Lorenz Graf-Vlachy & Jonathan Bundy & Donald C. Hambrick, 2020. "Effects of an Advancing Tenure on CEO Cognitive Complexity," Organization Science, INFORMS, vol. 31(4), pages 936-959, July.
    11. Hart E. Posen & Sangyoon Yi & Jeho Lee, 2020. "A contingency perspective on imitation strategies: When is “benchmarking” ineffective?," Strategic Management Journal, Wiley Blackwell, vol. 41(2), pages 198-221, February.
    12. Vivek Tandon & Puay Khoon Toh, 2022. "Who deviates? Technological opportunities, career concern, and inventor's distant search," Strategic Management Journal, Wiley Blackwell, vol. 43(4), pages 724-757, April.
    13. Juha Uotila, 2018. "Punctuated equilibrium or ambidexterity: dynamics of incremental and radical organizational change over time," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 27(1), pages 131-148.
    14. Varshney, Mayank, 2023. "Learning-by-hiring: How do rival firms learn from focal firm's hiring," Research Policy, Elsevier, vol. 52(2).
    15. Gino Cattani & Daniel Sands & Joe Porac & Jason Greenberg, 2018. "Competitive Sensemaking in Value Creation and Capture," Strategy Science, INFORMS, vol. 3(4), pages 632-657, December.
    16. Ron Adner & Felipe A. Csaszar & Peter B. Zemsky, 2014. "Positioning on a Multiattribute Landscape," Management Science, INFORMS, vol. 60(11), pages 2794-2815, November.
    17. Madeline K. Kneeland & Melissa A. Schilling & Barak S. Aharonson, 2020. "Exploring Uncharted Territory: Knowledge Search Processes in the Origination of Outlier Innovation," Organization Science, INFORMS, vol. 31(3), pages 535-557, May.
    18. Scott C. Ganz, 2020. "Hyperopic Search: Organizations Learning About Managers Learning About Strategies," Organization Science, INFORMS, vol. 31(4), pages 821-838, July.
    19. Choi, Jaeho & Rhee, Mooweon & Kim, Young-Choon, 2019. "Performance feedback and problemistic search: The moderating effects of managerial and board outsiderness," Journal of Business Research, Elsevier, vol. 102(C), pages 21-33.
    20. Felipe A. Csaszar & Harsh Ketkar & Hyunjin Kim, 2024. "Artificial Intelligence and Strategic Decision-Making: Evidence from Entrepreneurs and Investors," Papers 2408.08811, arXiv.org.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2410.06932. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.