Valid p-values and expectations of p-values revisited

We focus on valid definitions of p-values. A valid p-value (VpV) statistic can be used to make a prefixed level- $$ \alpha $$ α decision. In this context, Kolmogorov–Smirnov goodness-of-fit tests and the normal two-sample problem are considered. We examine an issue regarding the goodness-of-fit testability based on a single observation. We exemplify constructions of new test procedures, advocating practical reasons to implement VpV mechanisms. The VpV framework induces an extension of the conventional expected p-value (EPV) tool for measuring the performance of a test. Associating the EPV concept with the receiver operating characteristic (ROC) curve methodology, a well-established biostatistical approach, we propose a Youden’s index-based optimality to derive critical values of tests. In these terms, the significance level $$ \alpha = 0.05 $$ α = 0.05 is suggested. We introduce partial EPV’s to characterize properties of tests including their unbiasedness. We provide the intrinsic relationship between the Bayes Factor (BF) test statistic and the BF of test statistics.