Lies, Damned Lies and More Statistics: The Neglected Issue of Multiplicity in Accounting Research

The problem of ‘multiplicity’ (and selection) is considered by many statisticians to be among the most difficult and important problems they face. It includes such areas as multiple tests, variable selection in regression analysis, subgroup analysis and data mining. It undermines statistical inference by inflating type I errors well above reported levels of significance. The basic issue confronting researchers is how to adjust inferences to allow for multiplicity, particularly in exploratory or model-building analyses where standard textbook probability calculations associated with estimation and hypothesis testing do not apply. Despite its importance, and unlike what occurs in other disciplines, the multiplicity problem receives little or no attention in accounting. This situation is of concern because survey results suggest that serious type I error inflation is a real possibility in management accounting. In attempting to remedy this situation, the paper adopts a classical statistical framework for the purpose of examining the key issues underlying the problem and to present some strategies for dealing with it. These strategies offer the researcher a reasoned approach that recognises the possibility that the observed results may be due to chance, as well as the possibility that they are real. The discussion also highlights the fact that the only way to deal with the issue of multiplicity successfully is to repeat results on new data.