User Evaluation of Mobile Devices: In-Situ versus Laboratory Experiments

Nowadays, mobile devices features are often linked up to the context of usage. As a consequence, researchers must consider not only the user and the device, but also the surrounding environment when designing effective user study evaluations. Two opposite experimental setups are possible: in-situ and in the laboratory. There is no consensus on their respective benefits, for instance with regard to the number of usability issues detected. In this paper, we isolate independent variables that could contribute to evaluation biases by proposing a taxonomy that splits the in-situ experimental setups into two new setups. We describe the concept of the “Uncertainty Principle” to emphasize the dilemma between precise observation and bias minimization and introduce the “Trojan Horse” technique to partially overcome the consequences of the uncertainty principle. As a conclusion, a methodology using both laboratory and in-situ experiments in a complementary way is proposed.