Resampling-Based Inference Methods for Comparing Two Coefficients Alpha

The two-sample problem for Cronbach’s coefficient $$\alpha _C$$ α C , as an estimate of test or composite score reliability, has attracted little attention compared to the extensive treatment of the one-sample case. It is necessary to compare the reliability of a test for different subgroups, for different tests or the short and long forms of a test. In this paper, we study statistical procedures of comparing two coefficients $$\alpha _{C,1}$$ α C , 1 and $$\alpha _{C,2}$$ α C , 2 . The null hypothesis of interest is $$H_0 : \alpha _{C,1} = \alpha _{C,2}$$ H 0 : α C , 1 = α C , 2 , which we test against one-or two-sided alternatives. For this purpose, resampling-based permutation and bootstrap tests are proposed for two-group multivariate non-normal models under the general asymptotically distribution-free (ADF) setting. These statistical tests ensure a better control of the type-I error, in finite or very small sample sizes, when the state-of-affairs ADF large-sample test may fail to properly attain the nominal significance level. By proper choice of a studentized test statistic, the resampling tests are modified in order to be valid asymptotically even in non-exchangeable data frameworks. Moreover, extensions of this approach to other designs and reliability measures are discussed as well. Finally, the usefulness of the proposed resampling-based testing strategies is demonstrated in an extensive simulation study and illustrated by real data applications.