IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v66y2020i8p3412-3424.html
   My bibliography  Save this article

Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments

Author

Listed:
  • Duncan Simester

    (Massachusetts Institute of Technology, Cambridge, Massachusetts 02139)

  • Artem Timoshenko

    (Massachusetts Institute of Technology, Cambridge, Massachusetts 02139)

  • Spyros I. Zoumpoulis

    (INSEAD, 77300 Fontainebleau, France)

Abstract

Champion versus challenger field experiments are widely used to compare the performance of different targeting policies. These experiments randomly assign customers to receive marketing actions recommended by either the existing (champion) policy or the new (challenger) policy, and then compare the aggregate outcomes. We recommend an alternative experimental design and propose an alternative estimation approach to improve the evaluation of targeting policies. The recommended experimental design randomly assigns customers to marketing actions. This allows evaluation of any targeting policy without requiring an additional experiment, including policies designed after the experiment is implemented. The proposed estimation approach identifies customers for whom different policies recommend the same action and recognizes that for these customers there is no difference in performance. This allows for a more precise comparison of the policies. We illustrate the advantages of the experimental design and estimation approach using data from an actual field experiment. We also demonstrate that the grouping of customers, which is the foundation of our estimation approach, can help to improve the training of new targeting policies.

Suggested Citation

  • Duncan Simester & Artem Timoshenko & Spyros I. Zoumpoulis, 2020. "Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments," Management Science, INFORMS, vol. 66(8), pages 3412-3424, August.
    • Handle: RePEc:inm:ormnsc:v:66:y:2020:i:8:p:3412-3424
    DOI: 10.1287/mnsc.2019.3379
    as

    Download full text from publisher

    File URL: https://doi.org/10.1287/mnsc.2019.3379
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mnsc.2019.3379?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-838, May.
    2. Duncan I. Simester & Peng Sun & John N. Tsitsiklis, 2006. "Dynamic Catalog Mailing Policies," Management Science, INFORMS, vol. 52(5), pages 683-696, May.
    3. Bernd Skiera & Nadia Abou Nabout, 2013. "Practice Prize Paper ---PROSAD: A Bidding Decision Support System for Profit Optimizing Search Engine Advertising," Marketing Science, INFORMS, vol. 32(2), pages 213-220, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Alex Chin & Dean Eckles & Johan Ugander, 2022. "Evaluating Stochastic Seeding Strategies in Networks," Management Science, INFORMS, vol. 68(3), pages 1714-1736, March.
    2. Günter J. Hitsch & Sanjog Misra & Walter W. Zhang, 2024. "Heterogeneous treatment effects and optimal targeting policy evaluation," Quantitative Marketing and Economics (QME), Springer, vol. 22(2), pages 115-168, June.
    3. Carlos Fernández-Loría & Foster Provost, 2022. "Causal Decision Making and Causal Effect Estimation Are Not the Same…and Why It Matters," INFORMS Joural on Data Science, INFORMS, vol. 1(1), pages 4-16, April.
    4. Ngai, Eric W.T. & Wu, Yuanyuan, 2022. "Machine learning in marketing: A literature review, conceptual framework, and research agenda," Journal of Business Research, Elsevier, vol. 145(C), pages 35-48.
    5. Leif Nelson & Duncan Simester & K. Sudhir, 2020. "Introduction to the Special Issue on Marketing Science and Field Experiments," Marketing Science, INFORMS, vol. 39(6), pages 1033-1038, November.
    6. Arun Gopalakrishnan & Young-Hoon Park, 2021. "The Impact of Coupons on the Visit-to-Purchase Funnel," Marketing Science, INFORMS, vol. 40(1), pages 48-61, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dominik Schreyer, 2019. "Football spectator no-show behaviour in the German Bundesliga," Applied Economics, Taylor & Francis Journals, vol. 51(45), pages 4882-4901, September.
    2. Fors, Gunnar & Zejan, Mario, 1996. "Overseas R&D by Multinationals in foreign Centers of Excellence," SSE/EFI Working Paper Series in Economics and Finance 111, Stockholm School of Economics.
    3. repec:spo:wpmain:info:hdl:2441/7172 is not listed on IDEAS
    4. MacKinnon, J G, 1989. "Heteroskedasticity-Robust Tests for Structural Change," Empirical Economics, Springer, vol. 14(2), pages 77-92.
    5. Fenech, Jean-Pierre & Skully, Michael & Xuguang, Han, 2014. "Franking credits and market reactions: Evidence from the Australian convertible security market," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 32(C), pages 1-19.
    6. Bliss, Mark A. & Gul, Ferdinand A., 2012. "Political connection and leverage: Some Malaysian evidence," Journal of Banking & Finance, Elsevier, vol. 36(8), pages 2344-2350.
    7. Gu, Chen & Kurov, Alexander & Wolfe, Marketa Halova, 2018. "Relief Rallies after FOMC Announcements as a Resolution of Uncertainty," Journal of Empirical Finance, Elsevier, vol. 49(C), pages 1-18.
    8. Son K. Lam & Thomas E. DeCarlo & Ashish Sharma, 2019. "Salesperson ambidexterity in customer engagement: do customer base characteristics matter?," Journal of the Academy of Marketing Science, Springer, vol. 47(4), pages 659-680, July.
    9. David A. Volkman, 1999. "Market Volatility And Perverse Timing Performance Of Mutual Fund Managers," Journal of Financial Research, Southern Finance Association;Southwestern Finance Association, vol. 22(4), pages 449-470, December.
    10. Goncalves, Silvia & Kilian, Lutz, 2004. "Bootstrapping autoregressions with conditional heteroskedasticity of unknown form," Journal of Econometrics, Elsevier, vol. 123(1), pages 89-120, November.
    11. Daron Acemoglu & Philippe Aghion & Claire Lelarge & John Van Reenen & Fabrizio Zilibotti, 2007. "Technology, Information, and the Decentralization of the Firm," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(4), pages 1759-1799.
    12. Daiki Maki, 2015. "Wild bootstrap tests for unit root in ESTAR models," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 24(3), pages 475-490, September.
    13. Akinyosoye, Vincent O., 2007. "Demand For Dairy Products In Nigeria: Evidence From The Nigerian," Journal of Rural Economics and Development, University of Ibadan, Department of Agricultural Economics, vol. 16, pages 1-14.
    14. Arthur C. Brooks, 2001. "Private Philanthropy and the Economics of Public Radio," Center for Policy Research Working Papers 41, Center for Policy Research, Maxwell School, Syracuse University.
    15. Alfred Garloff & Carsten Pohl & Norbert Schanne, 2013. "Do small labor market entry cohorts reduce unemployment?," Demographic Research, Max Planck Institute for Demographic Research, Rostock, Germany, vol. 29(15), pages 379-406.
    16. Stolowy, Hervé & Jeanjean, Thomas & Erkens, Michael, 2011. "The economic consequences of increasing the international visibility of financial reports," HEC Research Papers Series 957, HEC Paris.
    17. Sylvain Chassang & Erik Snowberg & Ben Seymour & Cayley Bowles, 2015. "Accounting for Behavior in Treatment Effects: New Applications for Blind Trials," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-13, June.
    18. Niklas Potrafke, 2016. "Policies against human trafficking: the role of religion and political institutions," Economics of Governance, Springer, vol. 17(4), pages 353-386, November.
    19. Ito, Akitoshi, 1999. "Profits on technical trading rules and time-varying expected returns: evidence from Pacific-Basin equity markets," Pacific-Basin Finance Journal, Elsevier, vol. 7(3-4), pages 283-330, August.
    20. Oasis Kodila-Tedika & Julius Agbor, 2016. "Does Trust Matter for Entrepreneurship: Evidence from a Cross-Section of Countries," Economies, MDPI, vol. 4(1), pages 1-17, March.
    21. Carlo Rosa & Giovanni Verga, 2006. "The Impact of Central Bank Announcements on Asset Prices in Real Time: Testing the Efficiency of the Euribor Futures Market," CEP Discussion Papers dp0764, Centre for Economic Performance, LSE.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:66:y:2020:i:8:p:3412-3424. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.