My App is an Experiment: Experience from User Studies in Mobile App Stores

Experiments are a cornerstone of HCI research. Mobile distribution channels such as Apple’s App Store and Google’s Android Market have created the opportunity to bring experiments to the end user. Hardly any experience exists on how to conduct such experiments successfully. This article reports on five experiments that were conducted by publishing Apps in the Android Market. The Apps are freely available and have been installed more than 30,000 times. The outcomes of the experiments range from failure to valuable insights. Based on these outcomes, the authors identified factors that account for the success of experiments using mobile application stores. When generalizing findings it must be considered that smartphone users are a non-representative sample of the world’s population. Most participants can be obtained by informing users about the study when the App had been started for the first time. Because Apps are often used for a short time only, data should be collected as early as possible. To collect valuable qualitative feedback other channels than user comments and email have to be used. Finally, the interpretation of collected data has to consider unpredicted usage patterns to provide valid conclusions.