Improving Medical Abstract Classification Using PEFT-LoRA Fine-Tuned Large and Small Language Models

Designing intelligent systems to classify text in the medical domain is a challenging task. There is a shortage of openly available medical datasets (due to HIPPA-related strict regulations on protected health information for patients). In this paper, we explore the application of Open Source Medical LLMs (such as Meditron LLM), generic Large Language Models (such as LLAMA2), and Small Language Models (such as Phi2) on medical text classification (medical abstract dataset). We show that PEFT approaches such as LoRA can perform very well in classifying medical text, which involves interpreting patient conditions and symptoms and determining what medical problems the patients have. These approaches (based on Large and Small Language Models) have outperformed the current state of the results on medical abstracts corpus. In addition to medical LLMs, the open-source generic LLMs can be adapted to solving classification tasks on medical text and perform nearly as well as the specialized medical LLMs. SLMs can be a serious contender for solving domain-specific classification tasks (e.g., medical literature). This shows that carefully selecting the training data and fine-tuning positively impacts classification accuracy, precision, and recall. Generic Language Models such as LLAMA2 (LLM) and Phi2 (SLM) weren’t specifically trained with medical text. Medical LLMs such as Meditron outperform LLAMA2 and Phi2 in precision and accuracy. This is quite evident as Meditron was originally trained on medical text. The (micro averaged) F1 score for the fine-tuned Meditron model is 0.64. This is superior compared to fined-tuned LLAMA2 of 0.58 and Phi2 of 0.62. We see that Phi2 can outperform LLAMA2 with fewer number of parameters. The approaches used in this work can be extended to other medical text classification problems in the medical domain.