Exploring the applicability of large language models to citation context analysis

Unlike traditional citation analysis, which assumes that all citations in a paper are equivalent, citation context analysis considers the contextual information of individual citations. However, citation context analysis requires creating a large amount of data through annotation, which hinders its widespread use. This study explored the applicability of Large Language Models (LLM)—particularly Generative Pre-trained Transformer (GPT)—to citation context analysis by comparing LLM and human annotation results. The results showed that LLM annotation is as good as or better than human annotation in terms of consistency but poor in terms of its predictive performance. Thus, having LLM immediately replace human annotators in citation context analysis is inappropriate. However, the annotation results obtained by LLM can be used as reference information when narrowing the annotation results obtained by multiple human annotators down to one; alternatively, the LLM can be used as an annotator when it is difficult to prepare sufficient human annotators. This study provides basic findings important for the future development of citation context analysis.