P > 0.05 is Good: The NORD-h Protocol for Several Hypotheses Analysis Based on Known Risks, Costs, and Benefits

Statistical testing in public health is a controversial and often misunderstood topic. Despite decades of efforts by organizations, associations, and renowned international experts, fallacies of nullism, magnitude, and significance dichotomy are still extremely widespread. Nevertheless, our work highlights and addresses another common interpretive and cognitive error: the fallacy of surprise, understood as the mistaken habit of primarily considering and seeking findings with a low p-value (or a high s-value). Indeed, there are hypotheses (e.g., the efficacy of a drug) for which a high p-value is an optimal and desirable outcome. Therefore, this manuscript proposes a method to address the above situations based on comparing the statistical result with multiple hypotheses rather than just the null. Additionally, the concept of interval hypothesis is formalized based on costs, risks, and benefits known a priori. The objective is to assess the consistency of the data with various interval hypotheses of relevance simultaneously (including study limitations), in order to have a comprehensive understanding of the result reliability in relation to the research purposes. In this regard, the incompatibility graph (or surprisal graph) is introduced to make the reading of data simple and intuitive. As a general rule, unless meta-analyses with systematic review or other solid evidence are considered, these methods aim at providing a non-inferential and non-decisional descriptive overview of the scientific scenario.