Understanding the role of subpopulations and reliability in between-group studies

The replication crisis has led to a renewed discussion about the impacts of measurement quality on the precision of psychology research. High measurement quality is associated with low measurement error, yet the relationship between reliability and measurement error is not yet well understood. In this study, we attempt to understand the role of reliability in the replication research through its relationship with power. We outline a latent variable framework to understand this nuanced relationship through their equations. We find that, under the influence of the latent variable variance, reliability and power can be negatively associated. We proceed to demonstrate the impact of this relationship between reliability and power in replication research by imitating different scenarios of large-scale replications. We find negative correlations between reliability and power when there are sizable differences in the latent variable variance and negligible differences in the other parameters across studies. Finally, we apply our conclusions to the data from replications of the ego depletion effect (Hagger et al., 2016) and the replications of the grammatical aspect effect (Eerland et al., 2016). The applications show that a negative relationship between reliability and power is a realistic possibility, which eliminates doubts one might have about the applicability of our results to the applied work.