IDEAS home Printed from https://ideas.repec.org/a/spr/jcsosc/v8y2025i1d10.1007_s42001-024-00345-9.html
   My bibliography  Save this article

Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning

Author

Listed:
  • Meysam Alizadeh

    (University of Zurich)

  • Maël Kubli

    (University of Zurich)

  • Zeynab Samei

    (Institute for Fundamental Research)

  • Shirin Dehghani

    (Allameh Tabataba’i University)

  • Mohammadmasiha Zahedivafa

    (Iran University of Science and Technology)

  • Juan D. Bermeo

    (University of Zurich)

  • Maria Korobeynikova

    (University of Zurich)

  • Fabrizio Gilardi

    (University of Zurich)

Abstract

This paper studies the performance of open-source Large Language Models (LLMs) in text classification tasks typical for political science research. By examining tasks like stance, topic, and relevance classification, we aim to guide scholars in making informed decisions about their use of LLMs for text analysis and to establish a baseline performance benchmark that demonstrates the models’ effectiveness. Specifically, we conduct an assessment of both zero-shot and fine-tuned LLMs across a range of text annotation tasks using news articles and tweets datasets. Our analysis shows that fine-tuning improves the performance of open-source LLMs, allowing them to match or even surpass zero-shot GPT $$-$$ - 3.5 and GPT-4, though still lagging behind fine-tuned GPT $$-$$ - 3.5. We further establish that fine-tuning is preferable to few-shot training with a relatively modest quantity of annotated text. Our findings show that fine-tuned open-source LLMs can be effectively deployed in a broad spectrum of text annotation applications. We provide a Python notebook facilitating the application of LLMs in text annotation for other researchers.

Suggested Citation

  • Meysam Alizadeh & Maël Kubli & Zeynab Samei & Shirin Dehghani & Mohammadmasiha Zahedivafa & Juan D. Bermeo & Maria Korobeynikova & Fabrizio Gilardi, 2025. "Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning," Journal of Computational Social Science, Springer, vol. 8(1), pages 1-25, February.
  • Handle: RePEc:spr:jcsosc:v:8:y:2025:i:1:d:10.1007_s42001-024-00345-9
    DOI: 10.1007/s42001-024-00345-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s42001-024-00345-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s42001-024-00345-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Arthur Spirling, 2023. "Why open-source generative AI models are an ethical way forward for science," Nature, Nature, vol. 616(7957), pages 413-413, April.
    2. Barberá, Pablo & Boydstun, Amber E. & Linn, Suzanna & McMahon, Ryan & Nagler, Jonathan, 2021. "Automated Text Classification of News Articles: A Practical Guide," Political Analysis, Cambridge University Press, vol. 29(1), pages 19-42, January.
    3. Eva A. M. van Dis & Johan Bollen & Willem Zuidema & Robert van Rooij & Claudi L. Bockting, 2023. "ChatGPT: five priorities for research," Nature, Nature, vol. 614(7947), pages 224-226, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Evangelos Katsamakas & Oleg V. Pavlov & Ryan Saklad, 2024. "Artificial intelligence and the transformation of higher education institutions," Papers 2402.08143, arXiv.org.
    2. Celiktutan, Begum & Klesse, Anne-Kathrin & Tuk, Mirjam A., 2024. "Acceptability lies in the eye of the beholder: Self-other biases in GenAI collaborations," International Journal of Research in Marketing, Elsevier, vol. 41(3), pages 496-512.
    3. Emilio Ferrara, 2024. "GenAI against humanity: nefarious applications of generative artificial intelligence and large language models," Journal of Computational Social Science, Springer, vol. 7(1), pages 549-569, April.
    4. Shahad Al-Khalifa & Fatima Alhumaidhi & Hind Alotaibi & Hend S. Al-Khalifa, 2023. "ChatGPT across Arabic Twitter: A Study of Topics, Sentiments, and Sarcasm," Data, MDPI, vol. 8(11), pages 1-19, November.
    5. Nur Ashfaraliana Abd Hadi & Faizah Mohamad & Elia Md Johar & Zaemah Abdul Kadir, 2024. "Exploring the Acceptance of ChatGPT as an Assisting Tool in Academic Writing among ESL Undergraduate Students," International Journal of Research and Innovation in Social Science, International Journal of Research and Innovation in Social Science (IJRISS), vol. 8(10), pages 2886-2901, October.
    6. Peres, Renana & Schreier, Martin & Schweidel, David & Sorescu, Alina, 2023. "On ChatGPT and beyond: How generative artificial intelligence may affect research, teaching, and practice," International Journal of Research in Marketing, Elsevier, vol. 40(2), pages 269-275.
    7. Fabio Motoki & Valdemar Pinho Neto & Victor Rodrigues, 2024. "More human than human: measuring ChatGPT political bias," Public Choice, Springer, vol. 198(1), pages 3-23, January.
    8. Feng, Jianghong & Ning, Yu & Wang, Zhaohua & Li, Guo & Xiu Xu, Su, 2024. "ChatGPT-enabled two-stage auctions for electric vehicle battery recycling," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 183(C).
    9. Yanwei You & Yuquan Chen & Yujun You & Qi Zhang & Qiang Cao, 2023. "Evolutionary Game Analysis of Artificial Intelligence Such as the Generative Pre-Trained Transformer in Future Education," Sustainability, MDPI, vol. 15(12), pages 1-12, June.
    10. Christopher J. Lynch & Erik J. Jensen & Virginia Zamponi & Kevin O’Brien & Erika Frydenlund & Ross Gore, 2023. "A Structured Narrative Prompt for Prompting Narratives from Large Language Models: Sentiment Assessment of ChatGPT-Generated Narratives and Real Tweets," Future Internet, MDPI, vol. 15(12), pages 1-36, November.
    11. Ma, Xiaoyue & Huo, Yudi, 2023. "Are users willing to embrace ChatGPT? Exploring the factors on the acceptance of chatbots from the perspective of AIDUA framework," Technology in Society, Elsevier, vol. 75(C).
    12. Giordano, Vito & Spada, Irene & Chiarello, Filippo & Fantoni, Gualtiero, 2024. "The impact of ChatGPT on human skills: A quantitative study on twitter data," Technological Forecasting and Social Change, Elsevier, vol. 203(C).
    13. Hauke Licht & Ronja Sczepanksi, 2024. "Who are They Talking About? Detecting Mentions of Social Groups in Political Texts with Supervised Learning," ECONtribute Discussion Papers Series 277, University of Bonn and University of Cologne, Germany.
    14. Viglia, Giampaolo & Adler, Susanne J. & Miltgen, Caroline Lancelot & Sarstedt, Marko, 2024. "The use of synthetic data in tourism," Annals of Tourism Research, Elsevier, vol. 108(C).
    15. Ghio, Alessandro, 2024. "Democratizing academic research with Artificial Intelligence: The misleading case of language," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 98(C).
    16. Alin ZAMFIROIU & Denisa VASILE & Daniel SAVU, 2023. "ChatGPT – A Systematic Review of Published Research Papers," Informatica Economica, Academy of Economic Studies - Bucharest, Romania, vol. 27(1), pages 5-16.
    17. Roberto Araya, 2023. "Connecting Classrooms with Online Interclass Tournaments: A Strategy to Imitate, Recombine and Innovate Teaching Practices," Sustainability, MDPI, vol. 15(10), pages 1-25, May.
    18. Ching-Sheng Lin & Chung-Nan Tsai & Shao-Tang Su & Jung-Sing Jwo & Cheng-Hsiung Lee & Xin Wang, 2023. "Predictive Prompts with Joint Training of Large Language Models for Explainable Recommendation," Mathematics, MDPI, vol. 11(20), pages 1-12, October.
    19. Kuldip Singh & Elizabeth Caroline Agustine & Nur Farhani Binti Samasu & Nur Aida Binti Kipli, 2024. "Exploring Students’ Perceptions of ChatGPT: A Study of Public University in Malaysia," International Journal of Research and Innovation in Social Science, International Journal of Research and Innovation in Social Science (IJRISS), vol. 8(11), pages 1387-1397, November.
    20. Nuortimo, Kalle & Harkonen, Janne & Breznik, Kristijan, 2024. "Global, regional, and local acceptance of solar power," Renewable and Sustainable Energy Reviews, Elsevier, vol. 193(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:jcsosc:v:8:y:2025:i:1:d:10.1007_s42001-024-00345-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.