An evaluation of Monte-Carlo logic and logicFS motivated by a study of the regulation of gene expression in heart failure

Monte-Carlo (MC) Logic and Logic Feature Selection (logicFS) identify binary predictors of outcome using repeated iterations of logic regression, a variable selection method that identifies Boolean combinations of predictors. Both methods compute the frequency with which predictors appear in the model with the output of the logicFS program providing specific summaries of predictor form. We sought to identify variables related to transcription factor-related regulation of gene expression differences in a study of failing and non-failing hearts. Results based broadly on the frequency of occurrence of predictors into the MC Logic or logicFS models were similar. However key to logicFS are variable importance measures (VIMs), which augment the frequency metrics and seek to evaluate a predictor's contribution to classification. Analytic work and simulation studies indicate that the VIM vary as a function of the joint prevalence of outcome and predictor. Thus, findings from logicFS have limited generalizability, particularly with respect to case-control studies where the prevalence of outcome is determined by study design. Interpretation of VIM for those variables with near-zero or negative values is particularly ambiguous. Additional issues with interpretability arise because the VIM are strongly affected by other variables selected into the model but logicFS does not explicitly identify these variables in its output.