Deep learning in automated text classification: a case study using toxicological abstracts

Machine learning technology has been widely adopted as a cost-saving document prioritization approach in systematic literature reviews related to human health risk assessments. Supervised approaches use a training dataset, a relatively small set of documents with human-annotated labels indicating the topic of each document, to build models that automatically predict the labels of a much larger set of unlabelled documents. Deep learning algorithms form a branch of machine learning that relies on complex neural network architectures to learn the features of the object to be classified. Although deep learning algorithms have till recently mainly been applied for image, video, and audio classification, they are increasingly being deployed on text classification problems. To explore the potential advantages and practicalities of using deep learning algorithms in the document prioritization step of systematic literature reviews, we compare the performance of the most commonly used deep learning architectures with more traditional machine learning models using a dataset of approximately 7000 abstracts from the scientific literature related to the chemical arsenic. The dataset was previously annotated by subject matter experts with regard to relevance to toxicological mode of action. We examine the relative performance of each algorithm type at alternative levels of training by sequentially expanding the training dataset to generate a learning curve. We find that deep learning offers increased performance in some instances but also requires more data to train algorithms, increased model training time, increased computational power, and more labor-intensive algorithm tuning compared to baseline traditional machine learning algorithms.