A Comment on Identity Effects in Social Media - Taylor et al. (2023)

Taylor et al. (2023) explored the impact of identity cues on online behavior, employing a large-scale field experiment on a social news aggregation website. Findings reveal that identity cues significantly influence how individuals form opinions and engage with online content, accounting for 28% to 61% of variation in voting associated with commenters' production, reputation, and reciprocity. The results highlight the role of identity cues in perpetuating social content evaluation disparities and suggest anonymized content votes could enhance overall content quality on social platforms. In the replication analysis of this study, we utilized the provided script on the same data, which was provided by the paper's author following non-disclosure agreements. Further the robustness of the results was also tested after applying a mixed effects model instead of the linear probability model. Our replication confirmed the overall reproducibility of the results using the provided script, but there were notable changes in the estimates. In our analysis, the variation in individuals forming opinions and engaging with online content, as measured by voting associated with commenters' production, reputation, and reciprocity, ranged from 15% to 60% due to identity cues. This indicates that a few effects are somewhat smaller than in the original study. Moreover, when using our alternative analytic approach, the results remained generally robust, but there were exceptions. Specifically, the model assessing the impact of identity cues on individuals in voting associated with commenters' production yielded different results: We generally found stronger evidence in form of higher statistical significance for the claims of the authors.