IDEAS home Printed from https://ideas.repec.org/a/inm/ormksc/v43y2024i3p590-614.html
   My bibliography  Save this article

A Bias Correction Approach for Interference in Ranking Experiments

Author

Listed:
  • Ali Goli

    (University of Washington, Seattle, Washington 98195)

  • Anja Lambrecht

    (London Business School, London NW1 4SA, United Kingdom)

  • Hema Yoganarasimhan

    (University of Washington, Seattle, Washington 98195)

Abstract

Online marketplaces use ranking algorithms to determine the rank-ordering of items sold on their websites. The standard practice is to determine the optimal algorithm using A/B tests. We present a theoretical framework to characterize the total average treatment effect (TATE) of a ranking algorithm in an A/B test and show that naive TATE estimates can be biased because of interference. We propose a bias-correction approach that can recover the TATE of a ranking algorithm based on past A/B tests even if those tests suffer from a combination of interference issues. Our solution leverages data across multiple experiments and identifies observations in partial equilibrium in each experiment, that is, items close to their positions under the true counterfactual equilibrium of interest. We apply our framework to data from a travel website and present comprehensive evidence for interference bias in this setting. Next, we use our solution concept to build a customized deep learning model to predict the true TATE of the main algorithm of interest in our data. Counterfactual estimates from our model show that naive TATE estimates of click and booking rates can be biased by as much as 15% and 29%, respectively.

Suggested Citation

  • Ali Goli & Anja Lambrecht & Hema Yoganarasimhan, 2024. "A Bias Correction Approach for Interference in Ranking Experiments," Marketing Science, INFORMS, vol. 43(3), pages 590-614, May.
    • Handle: RePEc:inm:ormksc:v:43:y:2024:i:3:p:590-614
    DOI: 10.1287/mksc.2022.0046
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mksc.2022.0046
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mksc.2022.0046?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormksc:v:43:y:2024:i:3:p:590-614. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.