A novel text clustering model based on topic modelling and social network analysis

Document clustering is a well-known text-mining method that assists in the categorization and comprehension of textual data. Document clustering is vital in areas like information retrieval, knowledge management, and marketing, underscoring the need for a highly accurate clustering model. Current models in document clustering face significant hurdles, such as dealing with sparse, high-dimensional representations based on the bag-of-words (BOW) approach, which are not only computationally demanding on large datasets but also lack in capturing the semantic nuances of documents. Additionally, these models struggle with determining the ideal number of clusters and managing datasets with overlapping elements. To overcome these issues, this paper introduces a novel co-clustering strategy that merges community detection methods from social network analysis with advanced text analysis techniques. The proposed method transforms documents into a network structure, where each document is a node and connections (edges) are formed between documents that are most similar. Community detection algorithms are then employed to identify clusters within this network of documents. The study explores various document representation methods, including topic modelling and sentence embedding, to provide a rich contextual understanding of the documents. An extensive evaluation is carried out, examining different combinations of community detection algorithms, clustering methodologies, and document representation strategies, particularly focusing on their efficacy in handling overlapping and non-overlapping datasets. The findings demonstrate that the Element-Centric evaluation measure is effective in enabling community detection algorithms to autonomously ascertain the most suitable number of clusters, yielding promising results for both overlapping and non-overlapping datasets. The LCD model shows remarkable performance in addressing overlapping datasets. Furthermore, the research reveals that innovative document representation approaches significantly enhance the performance of the models. Additionally, the use of topic modelling in conjunction with co-clustering algorithms proves effective in clearly depicting the themes within the clusters.