Comparing Discrete Distributions: Survey Validation and Survey Experiments

Field survey experiments often measure amorphous concepts in discretely ordered categories, with postsurvey analytics that fail to account for the discrete attributes of the data. This article demonstrates the use of discrete distribution tests, specifically the chi-square test and the discrete Kolmogorov—Smirnov (KS) test, as simple devices for comparing and analyzing ordered responses typically found in surveys. In Monte Carlo simulations, we find the discrete KS test to have more power than the chi-square test when distributions are right or left skewed, regardless of the sample size or the number of alternatives. The discrete KS test has at least as much power as the chi-square, and sometimes more so, when distributions are bi-modal or approximately uniform and samples are small. After deriving rules of usage for the two tests, we implement them in two cases typical of survey analysis. Using our own data collected after Hurricanes Katrina and Rita, we employ our rules to both validate and assess treatment effects in a natural experimental setting.