Probability-based A/B testing with Adaptive Minimax Regret (AMR) criterion for long-term customer metrics

In consumer research and psychological experiments, subjects' states (attitudes) are manipulated by means of stimulus treatment in order to examine the effects of the subjects' states (attitudes) on the target variable. The interest here is not the effect of the treatment (stimulus) itself, but the effect on the target variable of the difference in state produced as a result of the treatment. Therefore, a manipulation check is usually performed to establish the validity of the experimental design, i.e., whether the stimulus produced the intended difference in state. When the manipulation-check variable (state) is directly associated with the target variable, one encounters the problem of confounding that affects both variables. To eliminate this problem, randomized controlled trials (RCTs) are used, but two weaknesses exist: first, only a discrete, binary effect of the presence or absence of an treatment on the target variable can be uncovered. Second, the incompleteness of the experimental design, in which the state induced by the treatment (stimulus) varies from subject to subject, resulting in different effects on the target variable, cannot be taken into account. In this study, we propose an approach that can correctly estimate the effect, which relates the manipulation-check variable to the target variable, even when unobserved confounding factors are present. By accounting for imperfections in the experimental design, the effect of the state variable becomes statistically more efficient than the effect of the experimental approach. The simulation analysis confirms that, for the same sample size, our instrumental variable approach is more significant than the usual experimental approach.