IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2310.17496.html
   My bibliography  Save this paper

Tackling Interference Induced by Data Training Loops in A/B Tests: A Weighted Training Approach

Author

Listed:
  • Nian Si

Abstract

In modern recommendation systems, the standard pipeline involves training machine learning models on historical data to predict user behaviors and improve recommendations continuously. However, these data training loops can introduce interference in A/B tests, where data generated by control and treatment algorithms, potentially with different distributions, are combined. To address these challenges, we introduce a novel approach called weighted training. This approach entails training a model to predict the probability of each data point appearing in either the treatment or control data and subsequently applying weighted losses during model training. We demonstrate that this approach achieves the least variance among all estimators that do not cause shifts in the training distributions. Through simulation studies, we demonstrate the lower bias and variance of our approach compared to other methods.

Suggested Citation

  • Nian Si, 2023. "Tackling Interference Induced by Data Training Loops in A/B Tests: A Weighted Training Approach," Papers 2310.17496, arXiv.org, revised Apr 2024.
    • Handle: RePEc:arx:papers:2310.17496
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2310.17496
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Eckles Dean & Karrer Brian & Ugander Johan, 2017. "Design and Analysis of Experiments in Networks: Reducing Bias from Interference," Journal of Causal Inference, De Gruyter, vol. 5(1), pages 1-23, March.
    2. Ariel Boyarsky & Hongseok Namkoong & Jean Pouget-Abadie, 2023. "Modeling Interference Using Experiment Roll-out," Papers 2305.10728, arXiv.org, revised Aug 2023.
    3. Ugander Johan & Yin Hao, 2023. "Randomized graph cluster randomization," Journal of Causal Inference, De Gruyter, vol. 11(1), pages 1-53, January.
    4. Yuchen Hu & Stefan Wager, 2022. "Switchback Experiments under Geometric Mixing," Papers 2209.00197, arXiv.org, revised Apr 2024.
    5. Hudgens, Michael G. & Halloran, M. Elizabeth, 2008. "Toward Causal Inference With Interference," Journal of the American Statistical Association, American Statistical Association, vol. 103, pages 832-842, June.
    6. Guillaume W Basse & Yi Ding & Panos Toulis, 2023. "Minimax designs for causal effects in temporal experiments with treatment habituation," Biometrika, Biometrika Trust, vol. 110(1), pages 155-168.
    7. Stefan Wager & Kuang Xu, 2021. "Experimenting in Equilibrium," Management Science, INFORMS, vol. 67(11), pages 6694-6715, November.
    8. Eckles Dean & Karrer Brian & Ugander Johan, 2017. "Design and Analysis of Experiments in Networks: Reducing Bias from Interference," Journal of Causal Inference, De Gruyter, vol. 5(1), pages 1-23, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jinglong Zhao, 2024. "Experimental Design For Causal Inference Through An Optimization Lens," Papers 2408.09607, arXiv.org, revised Aug 2024.
    2. Ruoxuan Xiong & Alex Chin & Sean J. Taylor, 2024. "Data-Driven Switchback Experiments: Theoretical Tradeoffs and Empirical Bayes Designs," Papers 2406.06768, arXiv.org.
    3. Evan Munro & David Jones & Jennifer Brennan & Roland Nelet & Vahab Mirrokni & Jean Pouget-Abadie, 2023. "Causal Estimation of User Learning in Personalized Systems," Papers 2306.00485, arXiv.org.
    4. Ozan Candogan & Chen Chen & Rad Niazadeh, 2024. "Correlated Cluster-Based Randomized Experiments: Robust Variance Minimization," Management Science, INFORMS, vol. 70(6), pages 4069-4086, June.
    5. Mohsen Bayati & Yuwei Luo & William Overman & Sadegh Shirani & Ruoxuan Xiong, 2024. "Higher-Order Causal Message Passing for Experimentation with Complex Interference," Papers 2411.00945, arXiv.org.
    6. Supriya Tiwari & Pallavi Basu, 2024. "Quasi-randomization tests for network interference," Papers 2403.16673, arXiv.org, revised Oct 2024.
    7. Luofeng Liao & Christian Kroer, 2024. "Statistical Inference and A/B Testing in Fisher Markets and Paced Auctions," Papers 2406.15522, arXiv.org, revised Aug 2024.
    8. Han, Kevin & Basse, Guillaume & Bojinov, Iavor, 2024. "Population interference in panel experiments," Journal of Econometrics, Elsevier, vol. 238(1).
    9. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," Working Papers hal-03455978, HAL.
    10. Shuze Chen & David Simchi-Levi & Chonghuan Wang, 2024. "Experimenting on Markov Decision Processes with Local Treatments," Papers 2407.19618, arXiv.org, revised Oct 2024.
    11. Zhaonan Qu & Ruoxuan Xiong & Jizhou Liu & Guido Imbens, 2021. "Semiparametric Estimation of Treatment Effects in Observational Studies with Heterogeneous Partial Interference," Papers 2107.12420, arXiv.org, revised Jun 2024.
    12. Ariel Boyarsky & Hongseok Namkoong & Jean Pouget-Abadie, 2023. "Modeling Interference Using Experiment Roll-out," Papers 2305.10728, arXiv.org, revised Aug 2023.
    13. Shaina J. Alexandria & Michael G. Hudgens & Allison E. Aiello, 2023. "Assessing intervention effects in a randomized trial within a social network," Biometrics, The International Biometric Society, vol. 79(2), pages 1409-1419, June.
    14. Vivek F. Farias & Andrew A. Li & Tianyi Peng & Andrew Zheng, 2022. "Markovian Interference in Experiments," Papers 2206.02371, arXiv.org, revised Jun 2022.
    15. Michael P. Leung, 2022. "Causal Inference Under Approximate Neighborhood Interference," Econometrica, Econometric Society, vol. 90(1), pages 267-293, January.
    16. Luofeng Liao & Christian Kroer, 2023. "Statistical Inference and A/B Testing for First-Price Pacing Equilibria," Papers 2301.02276, arXiv.org, revised Jun 2023.
    17. Stefan Wager & Kuang Xu, 2021. "Experimenting in Equilibrium," Management Science, INFORMS, vol. 67(11), pages 6694-6715, November.
    18. Davide Viviano & Jess Rudder, 2020. "Policy design in experiments with unknown interference," Papers 2011.08174, arXiv.org, revised May 2024.
    19. Elizabeth L. Ogburn & Ilya Shpitser & Youjin Lee, 2020. "Causal inference, social networks and chain graphs," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(4), pages 1659-1676, October.
    20. Davide Viviano & Lihua Lei & Guido Imbens & Brian Karrer & Okke Schrijvers & Liang Shi, 2023. "Causal clustering: design of cluster experiments under network interference," Papers 2310.14983, arXiv.org, revised Jan 2024.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2310.17496. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.