IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2411.12128.html
   My bibliography  Save this paper

The Role of Accuracy and Validation Effectiveness in Conversational Business Analytics

Author

Listed:
  • Adem Alparslan

Abstract

This study examines conversational business analytics, an approach that utilizes AI to address the technical competency gaps that hinder end users from effectively using traditional self-service analytics. By facilitating natural language interactions, conversational business analytics aims to empower end users to independently retrieve data and generate insights. The analysis focuses on Text-to-SQL as a representative technology for translating natural language requests into SQL statements. Developing theoretical models grounded in expected utility theory, this study identifies the conditions under which conversational business analytics, through partial or full support, can outperform delegation to human experts. The results indicate that partial support, focusing solely on information generation by AI, is viable when the accuracy of AI-generated SQL queries leads to a profit that surpasses the performance of a human expert. In contrast, full support includes not only information generation but also validation through explanations provided by the AI, and requires sufficiently high validation effectiveness to be reliable. However, user-based validation presents challenges, such as misjudgment and rejection of valid SQL queries, which may limit the effectiveness of conversational business analytics. These challenges underscore the need for robust validation mechanisms, including improved user support, automated processes, and methods for assessing quality independent of the technical competency of end users.

Suggested Citation

  • Adem Alparslan, 2024. "The Role of Accuracy and Validation Effectiveness in Conversational Business Analytics," Papers 2411.12128, arXiv.org, revised Nov 2024.
    • Handle: RePEc:arx:papers:2411.12128
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2411.12128
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Dursun Delen & Sudha Ram, 2018. "Research challenges and opportunities in business analytics," Journal of Business Analytics, Taylor & Francis Journals, vol. 1(1), pages 2-12, January.
    2. Paul Alpar & Michael Schulz, 2016. "Self-Service Business Intelligence," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 58(2), pages 151-155, April.
    3. Soroosh Tayebi Arasteh & Tianyu Han & Mahshad Lotfinia & Christiane Kuhl & Jakob Nikolas Kather & Daniel Truhn & Sven Nebelung, 2024. "Large language models streamline automated machine learning for clinical studies," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ron Berman & Ayelet Israeli, 2022. "The Value of Descriptive Analytics: Evidence from Online Retailers," Marketing Science, INFORMS, vol. 41(6), pages 1074-1096, November.
    2. Stinshoff, Volker, 2020. "Selbst gemacht ist gut gemacht? Der Einfluss von Self-Service Reporting auf die Qualität von Managemententscheidungen," Junior Management Science (JUMS), Junior Management Science e. V., vol. 5(2), pages 223-245.
    3. Jasmien Lismont & Tine Van Calster & María Óskarsdóttir & Seppe vanden Broucke & Bart Baesens & Wilfried Lemahieu & Jan Vanthienen, 2019. "Closing the Gap Between Experts and Novices Using Analytics-as-a-Service: An Experimental Study," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 61(6), pages 679-693, December.
    4. Nicolas Prat, 2019. "Augmented Analytics," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 61(3), pages 375-380, June.
    5. Jens Passlick & Lukas Grützner & Michael Schulz & Michael H. Breitner, 2023. "Self-service business intelligence and analytics application scenarios: A taxonomy for differentiation," Information Systems and e-Business Management, Springer, vol. 21(1), pages 159-191, March.
    6. Tarkan Tan & M. Hakan Akyüz & Bengisu Urlu & Santiago Ruiz, 2024. "Stop Auditing and Start to CARE: Paradigm Shift in Assessing and Improving Supplier Sustainability," Interfaces, INFORMS, vol. 54(3), pages 241-263, May.
    7. Steffen Kurpiela & Frank Teuteberg, 2024. "Linking business analytics affordances to corporate strategic planning and decision making outcomes," Information Systems and e-Business Management, Springer, vol. 22(1), pages 33-60, March.
    8. Luqman, Adeel & Wang, Liangyu & Katiyar, Gagan & Agarwal, Reeti & Mohapatra, Amiya Kumar, 2024. "Unpacking associations between positive-negative valence and ambidexterity of big data. Implications for firm performance," Technological Forecasting and Social Change, Elsevier, vol. 200(C).
    9. Cankaya, Burak & Topuz, Kazim & Delen, Dursun & Glassman, Aaron, 2023. "Evidence-based managerial decision-making with machine learning: The case of Bayesian inference in aviation incidents," Omega, Elsevier, vol. 120(C).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2411.12128. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.