Investigating the reliability of second-order formative measurement in information systems research

Recently, a few studies empirically explored the stability of first-order formative measurement, and raised concerns with its estimation reliability. Interpretational confounding, the disparity in the nominal and empirical meaning of a formatively measured construct, is at the center stage of the concern. Our study examines the issue in the context of the higher-order abstraction, focusing on the formatively defined relationship between the second-order construct and its indicators (i.e., first-order latent variables). Although the second-order formative abstraction is a widely accepted practice in structural equation modeling, the estimation results have been given a blind faith with no attempt to evaluate their integrity. Our empirical test, therefore, constitutes an attempt to fill the void. This study observed moderations of the theoretical relationship between reflectively designed first-order constructs and formatively defined second-order constructs when there is a change of endogenous variables. For this, two different formatively defined second-order constructs (i.e., IT management capabilities and IT personnel expertise) are utilized for the empirical testing. The estimation reveals that, while there was a considerable moderation of weights between IT management capabilities and its first-order constructs, those between IT personnel expertise and its first-order constructs remained relatively stable. These results demonstrate that the formatively defined relationship between the first- and second-order constructs can be precarious depending on the choice of the dependent variables. The analysis, therefore, revealed a significant presence of interpretational confounding and a higher chance of Type 1 error in model estimation. This implies that it becomes difficult to retain the construct validity and external validity of a formatively defined second-order construct. Thus, researchers are encouraged to exercise caution in mobilizing the formatively defined second-order measurement.