Small-Sample Accuracy of Approximate Distributions of Functions of Observed Probabilities From t Tests

The observed probability p is the social scientist’s primary tool for evaluating the outcomes of statistical hypothesis tests. Functions of p s are used in tests of “combined significance,†meta-analytic summaries based on sample probability values. This study examines the nonnull asymptotic distributions of several functions of one-tailed sample probability values (from t tests). Normal approximations were based on the asymptotic distributions of z(p), the standard normal deviate associated with the one-sided p value; of ln(p), the natural logarithm of the probability value; and of several modifications of ln(p). Two additional approximations, based on variance-stabilizing transformations of ln(p) and z(p), were derived. Approximate cumulative distribution functions (cdfs) were compared to the computed exact cdf of the p associated with the one-sample t test. Approximations to the distribution of z(p) appeared quite accurate even for very small samples, while other approximations were inaccurate unless sample sizes or effect sizes were very large. Approximations based on variance-stabilizing transformations were not much more accurate than those based on ln(p) and z(p). Generalizations of the results are discussed, and implications for use of the approximations conclude the article.