Please be polite to your peers: a multi-task model for assessing the tone and objectivity of critiques of peer review comments

The peer-review process plays a pivotal role in maintaining the quality and credibility of scientific publications. However, in recent times, there has been an increase in unhelpful and overly critical reviews, which can be detrimental to the process. This surge in unconstructive reviews can be attributed to a higher volume of paper submissions and the inclusion of inexperienced reviewers. Consequently, authors are left with limited valuable feedback, compromising the effectiveness of peer review. Peer review feedback must be not only objective but also delivered politely and constructively. Our study introduces a novel approach to assessing the constructiveness and tone of peer reviews. We propose a two-fold taxonomy that categorizes reviews into five labels for constructiveness and three labels for politeness. To facilitate this research, we have created a corpus of 2716 review sentences, which have been manually annotated with a high inter-annotation agreement of 88.27% for constructiveness and 83.49% for politeness, offering a valuable resource for the scientific community. Furthermore, we present a multi-task model named “Multi-Label Critique (MLC)”that leverages ToxicBERT representations and deep neural attention mechanisms. This model adeptly evaluates the constructiveness and politeness of review sentences, outperforming competitive baseline models with an impressive accuracy of 87.4%. Our paper includes an extensive analysis of the MLC model and its variations. Our research is a significant step towards contributing to the development of constructive peer-review reports.