IDEAS home Printed from https://ideas.repec.org/a/eee/ijrema/v41y2024i3p468-488.html
   My bibliography  Save this article

Generative AI for scalable feedback to multimodal exercises

Author

Listed:
  • Jürgensmeier, Lukas
  • Skiera, Bernd

Abstract

Detailed feedback on exercises helps learners become proficient but is time-consuming for educators and, thus, hardly scalable. This manuscript evaluates how well Generative Artificial Intelligence (AI) provides automated feedback on complex multimodal exercises requiring coding, statistics, and economic reasoning. Besides providing this technology through an easily accessible web application, this article evaluates the technology’s performance by comparing the quantitative feedback (i.e., points achieved) from Generative AI models with human expert feedback for 4,349 solutions to marketing analytics exercises. The results show that automated feedback produced by Generative AI (GPT-4) provides almost unbiased evaluations while correlating highly with (r = 0.94) and deviating only 6 % from human evaluations. GPT-4 performs best among seven Generative AI models, albeit at the highest cost. Comparing the models’ performance with costs shows that GPT-4, Mistral Large, Claude 3 Opus, and Gemini 1.0 Pro dominate three other Generative AI models (Claude 3 Sonnet, GPT-3.5, and Gemini 1.5 Pro). Expert assessment of the qualitative feedback (i.e., the AI’s textual response) indicates that it is mostly correct, sufficient, and appropriate for learners. A survey of marketing analytics learners shows that they highly recommend the app and its Generative AI feedback. An advantage of the app is its subject-agnosticism—it does not require any subject- or exercise-specific training. Thus, it is immediately usable for new exercises in marketing analytics and other subjects.

Suggested Citation

  • Jürgensmeier, Lukas & Skiera, Bernd, 2024. "Generative AI for scalable feedback to multimodal exercises," International Journal of Research in Marketing, Elsevier, vol. 41(3), pages 468-488.
    • Handle: RePEc:eee:ijrema:v:41:y:2024:i:3:p:468-488
    DOI: 10.1016/j.ijresmar.2024.05.005
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167811624000430
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijresmar.2024.05.005?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Peres, Renana & Schreier, Martin & Schweidel, David & Sorescu, Alina, 2023. "On ChatGPT and beyond: How generative artificial intelligence may affect research, teaching, and practice," International Journal of Research in Marketing, Elsevier, vol. 40(2), pages 269-275.
    2. Ali Goli & Amandeep Singh, 2024. "Frontiers: Can Large Language Models Capture Human Preferences?," Marketing Science, INFORMS, vol. 43(4), pages 709-722, July.
    3. Bernd Skiera & Lukas Jürgensmeier, 2024. "Teaching marketing analytics: a pricing case study for quantitative and substantive marketing skills," Journal of Marketing Analytics, Palgrave Macmillan, vol. 12(2), pages 209-226, June.
    4. Albers, Sönke, 2012. "Optimizable and implementable aggregate response modeling for marketing decision support," International Journal of Research in Marketing, Elsevier, vol. 29(2), pages 111-122.
    5. Martin Reisenbichler & Thomas Reutterer & David A. Schweidel & Daniel Dan, 2022. "Frontiers: Supporting Content Marketing with Natural Language Generation," Marketing Science, INFORMS, vol. 41(3), pages 441-452, May.
    6. Germann, Frank & Lilien, Gary L. & Rangaswamy, Arvind, 2013. "Performance implications of deploying marketing analytics," International Journal of Research in Marketing, Elsevier, vol. 30(2), pages 114-128.
    7. Peiyao Li & Noah Castelo & Zsolt Katona & Miklos Sarvary, 2024. "Frontiers: Determining the Validity of Large Language Models for Automated Perceptual Analysis," Marketing Science, INFORMS, vol. 43(2), pages 254-266, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Acar, Oguz A., 2024. "Commentary: Reimagining marketing education in the age of generative AI," International Journal of Research in Marketing, Elsevier, vol. 41(3), pages 489-495.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hermann, Erik & Puntoni, Stefano, 2024. "Artificial intelligence and consumer behavior: From predictive to generative AI," Journal of Business Research, Elsevier, vol. 180(C).
    2. Kirshner, Samuel N., 2024. "GPT and CLT: The impact of ChatGPT's level of abstraction on consumer recommendations," Journal of Retailing and Consumer Services, Elsevier, vol. 76(C).
    3. Yuan Gao & Dokyun Lee & Gordon Burtch & Sina Fazelpour, 2024. "Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina," Papers 2410.19599, arXiv.org, revised Jan 2025.
    4. Jan Ole Krugmann & Jochen Hartmann, 2024. "Sentiment Analysis in the Age of Generative AI," Customer Needs and Solutions, Springer;Institute for Sustainable Innovation and Growth (iSIG), vol. 11(1), pages 1-19, December.
    5. Arpan Kumar Kar & P. S. Varsha & Shivakami Rajan, 2023. "Unravelling the Impact of Generative Artificial Intelligence (GAI) in Industrial Applications: A Review of Scientific and Grey Literature," Global Journal of Flexible Systems Management, Springer;Global Institute of Flexible Systems Management, vol. 24(4), pages 659-689, December.
    6. Gary L. Lilien & John H. Roberts & Venkatesh Shankar, 2013. "Effective Marketing Science Applications: Insights from the ISMS-MSI Practice Prize Finalist Papers and Projects," Marketing Science, INFORMS, vol. 32(2), pages 229-245, March.
    7. Robertson, Jeandri & Ferreira, Caitlin & Botha, Elsamari & Oosthuizen, Kim, 2024. "Game changers: A generative AI prompt protocol to enhance human-AI knowledge co-construction," Business Horizons, Elsevier, vol. 67(5), pages 499-510.
    8. Celiktutan, Begum & Klesse, Anne-Kathrin & Tuk, Mirjam A., 2024. "Acceptability lies in the eye of the beholder: Self-other biases in GenAI collaborations," International Journal of Research in Marketing, Elsevier, vol. 41(3), pages 496-512.
    9. Frank Germann & Gary L. Lilien & Christine Moorman & Lars Fiedler & Till Groβmaβ, 2020. "Driving Customer Analytics From the Top," Customer Needs and Solutions, Springer;Institute for Sustainable Innovation and Growth (iSIG), vol. 7(3), pages 43-61, October.
    10. Samuel Adomako & Kwabena Frimpong & Joseph Amankwah-Amoah & Francis Donbesuur & Robert A. Opoku, 2021. "Strategic Decision Speed and International Performance: The Roles of Competitive Intensity, Resource Flexibility, and Structural Organicity," Management International Review, Springer, vol. 61(1), pages 27-55, March.
    11. Johannes Habel & Sascha Alavi & Nicolas Heinitz, 2023. "A theory of predictive sales analytics adoption," AMS Review, Springer;Academy of Marketing Science, vol. 13(1), pages 34-54, June.
    12. Casenave, Eric & Klarmann, Martin, 2020. "The accountability paradox: How holding marketers accountable hinders alignment with short-term marketing goals," Journal of Business Research, Elsevier, vol. 112(C), pages 95-108.
    13. Mian, G. Mujtaba & Sharma, Piyush & Gul, Ferdinand A., 2018. "Investor sentiment and advertising expenditure," International Journal of Research in Marketing, Elsevier, vol. 35(4), pages 611-627.
    14. Brüns, Jasper David & Meißner, Martin, 2024. "Do you create your content yourself? Using generative artificial intelligence for social media content creation diminishes perceived brand authenticity," Journal of Retailing and Consumer Services, Elsevier, vol. 79(C).
    15. Hofmann, Julian & Clement, Michel & Völckner, Franziska & Hennig-Thurau, Thorsten, 2017. "Empirical generalizations on the impact of stars on the economic success of movies," International Journal of Research in Marketing, Elsevier, vol. 34(2), pages 442-461.
    16. Eachempati, Prajwal & Srivastava, Praveen Ranjan & Kumar, Ajay & Muñoz de Prat, Javier & Delen, Dursun, 2022. "Can customer sentiment impact firm value? An integrated text mining approach," Technological Forecasting and Social Change, Elsevier, vol. 174(C).
    17. Clemens Koob, 2021. "Determinants of content marketing effectiveness: Conceptual framework and empirical findings from a managerial perspective," PLOS ONE, Public Library of Science, vol. 16(4), pages 1-25, April.
    18. Walsh, Gianfranco & Northington, William Magnus & Hille, Patrick & Dose, David, 2015. "Service employees' willingness to report complaints scale: Cross-country application and replication," Journal of Business Research, Elsevier, vol. 68(3), pages 500-506.
    19. Shuaiyu Chen & T. Clifton Green & Huseyin Gulen & Dexin Zhou, 2024. "What Does ChatGPT Make of Historical Stock Returns? Extrapolation and Miscalibration in LLM Stock Return Forecasts," Papers 2409.11540, arXiv.org.
    20. Kimberly A. Whitler & Ben Lee & Ryan Krause & Neil A. Morgan, 2021. "Upper echelons research in marketing," Journal of the Academy of Marketing Science, Springer, vol. 49(1), pages 198-219, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ijrema:v:41:y:2024:i:3:p:468-488. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/international-journal-of-research-in-marketing/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.