Recommendation by Multiscale Semantic-Visual Analysis of User Reviews and Product Images

Recommendation for items that users have not interacted with is often based on the predicted ratings. However, predicting an item rating can be challenging due to a fact that the user-item paired data are not easily. Many studies addressed this problem by modeling latent factors of users (and/or items) using semantic reviews, as they are available on most ecommerce websites. However, this approach ignored the visual features of items, or product images, which reflect users’ personal preference. In this paper, we present a novel method that exploits multiscale semantic-visual representations (MSVR) to perform rating predictions with two paralleled submodules. The first submodule identifies the review features with respect to various aspects through word-aware and scale-aware attention mechanisms. The second submodule extracts block-level visual features at multi-scales with a pre-trained deep net, and then embeds the features with visual word vocabularies followed by a projection layer to reduce dimensions and an attention layer to re-weight multiscale blocks. Extensive experiments performed on 22 Amazon datasets demonstrates that our model significantly outperforms several state-of-the-art recommendation methods.