IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2411.01067.html
   My bibliography  Save this paper

Randomized Controlled Trials for Security Copilot for IT Administrators

Author

Listed:
  • James Bono
  • Alec Xu

Abstract

As generative AI (GAI) tools become increasingly integrated into workplace environments, it is essential to measure their impact on productivity across specific domains. This study evaluates the effects of Microsoft's Security Copilot ("Copilot") on information technology administrators ("IT admins") through randomized controlled trials. Participants were divided into treatment and control groups, with the former granted access to Copilot within Microsoft's Entra and Intune admin centers. Across three IT admin scenarios - sign-in troubleshooting, device policy management, and device troubleshooting - Copilot users demonstrated substantial improvements in both accuracy and speed. Across all scenarios and tasks, Copilot subjects experienced a 34.53% improvement in overall accuracy and a 29.79% reduction in task completion time. We also find that the productivity benefits vary by task type, with more complex tasks showing greater improvement. In free response tasks, Copilot users identified 146.07% more relevant facts and reduced task completion time by 61.14%. Subject satisfaction with Copilot was high, with participants reporting reduced effort and a strong preference for using the tool in future tasks. These findings suggest that GAI tools like Copilot can significantly enhance the productivity and efficiency of IT admins, especially in scenarios requiring information synthesis and complex decision-making.

Suggested Citation

  • James Bono & Alec Xu, 2024. "Randomized Controlled Trials for Security Copilot for IT Administrators," Papers 2411.01067, arXiv.org, revised Nov 2024.
    • Handle: RePEc:arx:papers:2411.01067
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2411.01067
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Erik Brynjolfsson & Danielle Li & Lindsey Raymond, 2023. "Generative AI at Work," Papers 2304.11771, arXiv.org, revised Nov 2024.
    2. Parteka, Aleksandra & Kordalska, Aleksandra, 2023. "Artificial intelligence and productivity: global evidence from AI patent and bibliometric data," Technovation, Elsevier, vol. 125(C).
    3. Alexander Bick & Adam Blandin & David Deming, 2023. "The Rapid Adoption of Generative AI," On the Economy 98843, Federal Reserve Bank of St. Louis.
    4. Ajay Agrawal & Joshua S. Gans & Avi Goldfarb, 2019. "Artificial Intelligence: The Ambiguous Labor Market Impact of Automating Prediction," Journal of Economic Perspectives, American Economic Association, vol. 33(2), pages 31-50, Spring.
    5. Tyna Eloundou & Sam Manning & Pamela Mishkin & Daniel Rock, 2023. "GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models," Papers 2303.10130, arXiv.org, revised Aug 2023.
    6. Daron Acemoglu, 2024. "The Simple Macroeconomics of AI," NBER Working Papers 32487, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Carvajal, Daniel & Franco, Catalina & Isaksson, Siri, 2024. "Will Artificial Intelligence Get in the Way of Achieving Gender Equality?," Discussion Paper Series in Economics 3/2024, Norwegian School of Economics, Department of Economics, revised 20 Mar 2025.
    2. Amali Matharaarachchi & Wishmitha Mendis & Kanishka Randunu & Daswin De Silva & Gihan Gamage & Harsha Moraliyage & Nishan Mills & Andrew Jennings, 2024. "Optimizing Generative AI Chatbots for Net-Zero Emissions Energy Internet-of-Things Infrastructure," Energies, MDPI, vol. 17(8), pages 1-19, April.
    3. Becker, Dominik & Deck, Luca & Feulner, Simon & Gutheil, Niklas & Schüll, Moritz & Decker, Stefan & Eymann, Torsten & Gimpel, Henner & Pippow, Andreas & Röglinger, Maximilian & Urbach, Nils, 2024. "Lohnt sich Microsoft 365 Copilot? Eine Potenzialanalyse für Unternehmen und Bildungseinrichtungen," Bayreuth Reports on Information Systems Management 72, University of Bayreuth, Chair of Information Systems Management.
    4. Harry Moroz & Mariana Viollaz, 2024. "The Future of Work in Central America and the Dominican Republic," World Bank Publications - Reports 42043, The World Bank Group.
    5. Christian Peukert & Florian Abeillon & Jérémie Haese & Franziska Kaiser & Alexander Staub, 2024. "Strategic Behavior and AI Training Data," CESifo Working Paper Series 11099, CESifo.
    6. Tatsuru Kikuchi, 2024. "Impact Evaluation on the European Privacy Laws governing generative-AI models -- Evidence in Relation between Internet Censorship and the Ban of ChatGPT in Italy," Papers 2407.06495, arXiv.org.
    7. Fossen, Frank M. & McLemore, Trevor & Sorgner, Alina, 2024. "Artificial Intelligence and Entrepreneurship," IZA Discussion Papers 17055, Institute of Labor Economics (IZA).
    8. Kreitmeir, David & Raschky, Paul Anton, 2023. "The Unintended Consequences of Censoring Digital Technology - Evidence from Italy's ChatGPT Ban," SocArXiv v3cgs, Center for Open Science.
    9. Dandan Qiao & Huaxia Rui & Qian Xiong, 2023. "AI and Jobs: Has the Inflection Point Arrived? Evidence from an Online Labor Platform," Papers 2312.04180, arXiv.org, revised Aug 2024.
    10. Albanesi, Stefania & Dias da Silva, Antonio & Jimeno, Juan Francisco & Lamo, Ana & Wabitsch, Alena, 2023. "New Technologies and Jobs in Europe," CEPR Discussion Papers 18220, C.E.P.R. Discussion Papers.
    11. Ozge Demirci & Jonas Hannane & Xinrong Zhu, 2024. "Who Is AI Replacing? The Impact of Generative AI on Online Freelancing Platforms," CESifo Working Paper Series 11276, CESifo.
    12. Daniel Goller & Christian Gschwendt & Stefan C. Wolter, 2023. ""This time it's different" Generative Artificial Intelligence and Occupational Choice," Economics of Education Working Paper Series 0209, University of Zurich, Department of Business Administration (IBW).
    13. Walkowiak, Emmanuelle, 2023. "Task-interdependencies between Generative AI and Workers," Economics Letters, Elsevier, vol. 231(C).
    14. Arai Koki, 2024. "Law and Economics of Language Model Development: Empirical Examination of Corporate Strategies and Vaporware Claims," Asian Journal of Law and Economics, De Gruyter, vol. 15(1), pages 31-53, April.
    15. Masayuki MORIKAWA, 2024. "Use of Artificial Intelligence and Productivity: Evidence from firm and worker surveys," Discussion papers 24074, Research Institute of Economy, Trade and Industry (RIETI).
    16. Jason P Davis & Jian Bai Li, 2024. "Early Adoption of Generative AI by Global Business Leaders: Insights from an INSEAD Alumni Survey," Papers 2404.04543, arXiv.org.
    17. Masayuki MORIKAWA, 2024. "Macroeconomic Impact of Artificial Intelligence on Productivity: An estimate from a survey," Discussion papers 24084, Research Institute of Economy, Trade and Industry (RIETI).
    18. Tina Highfill & David Wasshausen & Gregory Prunchak, 2025. "Concepts and Challenges of Measuring Production of Artificial Intelligence in the U.S. Economy," BEA Papers 0134, Bureau of Economic Analysis.
    19. Siliang Tong & Nan Jia & Xueming Luo & Zheng Fang, 2021. "The Janus face of artificial intelligence feedback: Deployment versus disclosure effects on employee performance," Strategic Management Journal, Wiley Blackwell, vol. 42(9), pages 1600-1631, September.
    20. Christoph Riedl & Eric Bogert, 2024. "Effects of AI Feedback on Learning, the Skill Gap, and Intellectual Diversity," Papers 2409.18660, arXiv.org.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2411.01067. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.