Deep and handcrafted features from clinical images combined with patient information for skin cancer diagnosis

Skin lesions diagnostic is a challenging problem due to the variety of visual aspects of the lesions. The clinical analysis of skin lesions relies on the visual information as well as on the complementary information provided by the patient. Since dermatologists make use of visual cues and patient clinical information, we investigate if the combination of features from convolutional neural networks (CNN), handcrafted features and patient clinical information can improve the performance of automated diagnosis of skin cancer. Most works on skin lesion diagnosis in the literature use dermoscopic images without patient clinical information. In order to address this problem, we used a clinical image dataset of skin lesions with patient information collected via smartphone named PAD-UFES-20. With the proposed fusion architecture we show that the results using clinical features as a complement to the CNN and handcrafted features improve the classification in terms of balanced accuracy by 7.1 % for cancer and by 3.2 % for melanoma as compared with only features extracted from a CNN. In addition, our findings show that combining only handcrafted features with deep features did not improve the results, indicating the importance of using clinical metadata for skin lesion classification.