IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2411.01067.html
   My bibliography  Save this paper

Randomized Controlled Trials for Security Copilot for IT Administrators

Author

Listed:
  • James Bono
  • Alec Xu

Abstract

As generative AI (GAI) tools become increasingly integrated into workplace environments, it is essential to measure their impact on productivity across specific domains. This study evaluates the effects of Microsoft's Security Copilot ("Copilot") on information technology administrators ("IT admins") through randomized controlled trials. Participants were divided into treatment and control groups, with the former granted access to Copilot within Microsoft's Entra and Intune admin centers. Across three IT admin scenarios - sign-in troubleshooting, device policy management, and device troubleshooting - Copilot users demonstrated substantial improvements in both accuracy and speed. Across all scenarios and tasks, Copilot subjects experienced a 34.53% improvement in overall accuracy and a 29.79% reduction in task completion time. We also find that the productivity benefits vary by task type, with more complex tasks showing greater improvement. In free response tasks, Copilot users identified 146.07% more relevant facts and reduced task completion time by 61.14%. Subject satisfaction with Copilot was high, with participants reporting reduced effort and a strong preference for using the tool in future tasks. These findings suggest that GAI tools like Copilot can significantly enhance the productivity and efficiency of IT admins, especially in scenarios requiring information synthesis and complex decision-making.

Suggested Citation

  • James Bono & Alec Xu, 2024. "Randomized Controlled Trials for Security Copilot for IT Administrators," Papers 2411.01067, arXiv.org, revised Nov 2024.
    • Handle: RePEc:arx:papers:2411.01067
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2411.01067
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Erik Brynjolfsson & Danielle Li & Lindsey Raymond, 2023. "Generative AI at Work," Papers 2304.11771, arXiv.org, revised Nov 2024.
    2. Parteka, Aleksandra & Kordalska, Aleksandra, 2023. "Artificial intelligence and productivity: global evidence from AI patent and bibliometric data," Technovation, Elsevier, vol. 125(C).
    3. Alexander Bick & Adam Blandin & David Deming, 2023. "The Rapid Adoption of Generative AI," On the Economy 98843, Federal Reserve Bank of St. Louis.
    4. Ajay Agrawal & Joshua S. Gans & Avi Goldfarb, 2019. "Artificial Intelligence: The Ambiguous Labor Market Impact of Automating Prediction," Journal of Economic Perspectives, American Economic Association, vol. 33(2), pages 31-50, Spring.
    5. Tyna Eloundou & Sam Manning & Pamela Mishkin & Daniel Rock, 2023. "GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models," Papers 2303.10130, arXiv.org, revised Aug 2023.
    6. Daron Acemoglu, 2024. "The Simple Macroeconomics of AI," NBER Working Papers 32487, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Piyush Gulati & Arianna Marchetti & Phanish Puranam & Victoria Sevcenko, 2025. "Generative AI Adoption and Higher Order Skills," Papers 2503.09212, arXiv.org.
    2. Christoph Riedl & Eric Bogert, 2024. "Effects of AI Feedback on Learning, the Skill Gap, and Intellectual Diversity," Papers 2409.18660, arXiv.org.
    3. Carvajal, Daniel & Franco, Catalina & Isaksson, Siri, 2024. "Will Artificial Intelligence Get in the Way of Achieving Gender Equality?," Discussion Paper Series in Economics 3/2024, Norwegian School of Economics, Department of Economics, revised 20 Mar 2025.
    4. Leonardo Banh & Gero Strobel, 2023. "Generative artificial intelligence," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-17, December.
    5. Amali Matharaarachchi & Wishmitha Mendis & Kanishka Randunu & Daswin De Silva & Gihan Gamage & Harsha Moraliyage & Nishan Mills & Andrew Jennings, 2024. "Optimizing Generative AI Chatbots for Net-Zero Emissions Energy Internet-of-Things Infrastructure," Energies, MDPI, vol. 17(8), pages 1-19, April.
    6. Becker, Dominik & Deck, Luca & Feulner, Simon & Gutheil, Niklas & Schüll, Moritz & Decker, Stefan & Eymann, Torsten & Gimpel, Henner & Pippow, Andreas & Röglinger, Maximilian & Urbach, Nils, 2024. "Lohnt sich Microsoft 365 Copilot? Eine Potenzialanalyse für Unternehmen und Bildungseinrichtungen," Bayreuth Reports on Information Systems Management 72, University of Bayreuth, Chair of Information Systems Management.
    7. Harry Moroz & Mariana Viollaz, 2024. "The Future of Work in Central America and the Dominican Republic," World Bank Publications - Reports 42043, The World Bank Group.
    8. Christian Peukert & Florian Abeillon & Jérémie Haese & Franziska Kaiser & Alexander Staub, 2024. "Strategic Behavior and AI Training Data," CESifo Working Paper Series 11099, CESifo.
    9. Tatsuru Kikuchi, 2024. "Impact Evaluation on the European Privacy Laws governing generative-AI models -- Evidence in Relation between Internet Censorship and the Ban of ChatGPT in Italy," Papers 2407.06495, arXiv.org.
    10. Fossen, Frank M. & McLemore, Trevor & Sorgner, Alina, 2024. "Artificial Intelligence and Entrepreneurship," IZA Discussion Papers 17055, Institute of Labor Economics (IZA).
    11. Kreitmeir, David & Raschky, Paul Anton, 2023. "The Unintended Consequences of Censoring Digital Technology - Evidence from Italy's ChatGPT Ban," SocArXiv v3cgs, Center for Open Science.
    12. Dandan Qiao & Huaxia Rui & Qian Xiong, 2023. "AI and Jobs: Has the Inflection Point Arrived? Evidence from an Online Labor Platform," Papers 2312.04180, arXiv.org, revised Aug 2024.
    13. Ramaul, Laavanya & Ritala, Paavo & Ruokonen, Mika, 2024. "Creational and conversational AI affordances: How the new breed of chatbots is revolutionizing knowledge industries," Business Horizons, Elsevier, vol. 67(5), pages 615-627.
    14. Albanesi, Stefania & Dias da Silva, Antonio & Jimeno, Juan Francisco & Lamo, Ana & Wabitsch, Alena, 2023. "New Technologies and Jobs in Europe," CEPR Discussion Papers 18220, C.E.P.R. Discussion Papers.
    15. Ozge Demirci & Jonas Hannane & Xinrong Zhu, 2024. "Who Is AI Replacing? The Impact of Generative AI on Online Freelancing Platforms," CESifo Working Paper Series 11276, CESifo.
    16. Daniel Goller & Christian Gschwendt & Stefan C. Wolter, 2023. ""This time it's different" Generative Artificial Intelligence and Occupational Choice," Economics of Education Working Paper Series 0209, University of Zurich, Department of Business Administration (IBW).
    17. Pawel Gmyrek & Hernan Winkler & Santiago Garganta, 2024. "Buffer or Bottleneck? Employment Exposure to Generative AI and the Digital Divide in Latin America," CEDLAS, Working Papers 0340, CEDLAS, Universidad Nacional de La Plata.
    18. Yan Liu & He Wang, 2024. "Who on Earth Is Using Generative AI ?," Policy Research Working Paper Series 10870, The World Bank.
    19. Frantz Rowe & Raphaël Suire & Myriam Raymond & Florence Jacob, 2024. "Beliefs, Controversies, and Innovation Diffusion: The case of Beliefs, Controversies, and Innovation Diffusion: The case of Generative AI in a Large Technological Firm Generative AI in a Large Technol," Post-Print hal-04973613, HAL.
    20. Walkowiak, Emmanuelle, 2023. "Task-interdependencies between Generative AI and Workers," Economics Letters, Elsevier, vol. 231(C).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2411.01067. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.