Practical alternatives to the “significant/non-significant” statistical dichotomy

Science is a discipline that selects, collects, and summarizes the best available information on a given phenomenon in order to model it. It is closer to a bet with the highest probability of winning than to the search for truth. The dichotomous categorization of significance (e.g., acceptance vs. rejection of objective hypotheses) is unsuitable for the scientific nature of a statistical investigation and inevitably leads to dangerous exaggerations of the state of evidence (especially on single studies). Indeed, no study can prove that a result is (not) significant (not even at the clinical level) since uncertainty is and always will be part of scientific action. At most, it can be stated that a certain result, in terms of both statistical surprise and effect size, together with other evidence (e.g., biological, psychological, etc.), provides a certain degree of information “against” or “in favor of” substantial consequences in the clinical field. Therefore, based on the above considerations and an analysis of the (function of) costs and benefits, it can be decided whether a certain therapy meets the threshold of scientific evidence required for its approval. In this regard, this manuscript proposes and discusses alternative concepts to statistical dichotomy such as ranges of significance (or, better, ranges of compatibility with hypotheses) or the recommended evaluation of results in terms of statistical surprise (by comparison with obtaining S consecutive heads when flipping an unbiased coin). Furthermore, it emphasizes the absolute necessity to investigate all relevant objective hypotheses (not just the null one) as well as all model assumptions (the violation of which can make significance/compatibility/surprise uninformative for the scientific goal). Finally, it proposes a compact framework for a complete presentation of results.