Statistical Reliability of a Diet-Disease Association Meta-analysis

Risk ratios or p-values from multiple, independent studies – observational or randomized – can be pooled to address a common research question in meta-analysis. However, reliability of independent studies should not be assumed as claimed risk factor−disease relationships may fail to reproduce. An independent evaluation was undertaken of a published meta-analysis of cohort studies examining diet−disease associations; specifically between red and processed meat and six disease outcomes (all-cause mortality, cardiovascular mortality, all cancer mortality, breast cancer incidence, colorectal cancer incidence, type 2 diabetes incidence). The number of hypotheses examined were counted in 15 random base papers (14%) of 105 used in the meta-analysis. Test statistics (relative risk values with 95% confidence limits) for 125 results used in the meta-analysis were converted to p-values; p-value plots were used to examine the effect heterogeneity of the p-values. The possible number of hypotheses examined in the 15 base papers was large, median = 20,736 (interquartile range = 1,728–331,776). Each p-value plot for selected health effects showed either a random pattern (p-values > 0.05), or a two-component mixture (small p-values < 0.001 while other p-values appeared random). Given potentially large numbers of hypotheses examined in the base studies, questionable research practices cannot be ruled out as explanations for some test statistics with small p-values. Like the original findings of the published meta-analysis, our independent evaluation concludes that base papers used in the meta-analysis do not support evidence for an association between red and processed meat and the six health effects investigated.