IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v66y2020i8p3412-3424.html
   My bibliography  Save this article

Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments

Author

Listed:
  • Duncan Simester

    (Massachusetts Institute of Technology, Cambridge, Massachusetts 02139)

  • Artem Timoshenko

    (Massachusetts Institute of Technology, Cambridge, Massachusetts 02139)

  • Spyros I. Zoumpoulis

    (INSEAD, 77300 Fontainebleau, France)

Abstract

Champion versus challenger field experiments are widely used to compare the performance of different targeting policies. These experiments randomly assign customers to receive marketing actions recommended by either the existing (champion) policy or the new (challenger) policy, and then compare the aggregate outcomes. We recommend an alternative experimental design and propose an alternative estimation approach to improve the evaluation of targeting policies. The recommended experimental design randomly assigns customers to marketing actions. This allows evaluation of any targeting policy without requiring an additional experiment, including policies designed after the experiment is implemented. The proposed estimation approach identifies customers for whom different policies recommend the same action and recognizes that for these customers there is no difference in performance. This allows for a more precise comparison of the policies. We illustrate the advantages of the experimental design and estimation approach using data from an actual field experiment. We also demonstrate that the grouping of customers, which is the foundation of our estimation approach, can help to improve the training of new targeting policies.

Suggested Citation

  • Duncan Simester & Artem Timoshenko & Spyros I. Zoumpoulis, 2020. "Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments," Management Science, INFORMS, vol. 66(8), pages 3412-3424, August.
    • Handle: RePEc:inm:ormnsc:v:66:y:2020:i:8:p:3412-3424
    DOI: 10.1287/mnsc.2019.3379
    as

    Download full text from publisher

    File URL: https://doi.org/10.1287/mnsc.2019.3379
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mnsc.2019.3379?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-838, May.
    2. Duncan I. Simester & Peng Sun & John N. Tsitsiklis, 2006. "Dynamic Catalog Mailing Policies," Management Science, INFORMS, vol. 52(5), pages 683-696, May.
    3. Bernd Skiera & Nadia Abou Nabout, 2013. "Practice Prize Paper ---PROSAD: A Bidding Decision Support System for Profit Optimizing Search Engine Advertising," Marketing Science, INFORMS, vol. 32(2), pages 213-220, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Alex Chin & Dean Eckles & Johan Ugander, 2022. "Evaluating Stochastic Seeding Strategies in Networks," Management Science, INFORMS, vol. 68(3), pages 1714-1736, March.
    2. Günter J. Hitsch & Sanjog Misra & Walter W. Zhang, 2024. "Heterogeneous treatment effects and optimal targeting policy evaluation," Quantitative Marketing and Economics (QME), Springer, vol. 22(2), pages 115-168, June.
    3. Carlos Fernández-Loría & Foster Provost, 2022. "Causal Decision Making and Causal Effect Estimation Are Not the Same…and Why It Matters," INFORMS Joural on Data Science, INFORMS, vol. 1(1), pages 4-16, April.
    4. Ngai, Eric W.T. & Wu, Yuanyuan, 2022. "Machine learning in marketing: A literature review, conceptual framework, and research agenda," Journal of Business Research, Elsevier, vol. 145(C), pages 35-48.
    5. Leif Nelson & Duncan Simester & K. Sudhir, 2020. "Introduction to the Special Issue on Marketing Science and Field Experiments," Marketing Science, INFORMS, vol. 39(6), pages 1033-1038, November.
    6. Arun Gopalakrishnan & Young-Hoon Park, 2021. "The Impact of Coupons on the Visit-to-Purchase Funnel," Marketing Science, INFORMS, vol. 40(1), pages 48-61, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dominik Schreyer, 2019. "Football spectator no-show behaviour in the German Bundesliga," Applied Economics, Taylor & Francis Journals, vol. 51(45), pages 4882-4901, September.
    2. S. Arunachalam & Sridhar N. Ramaswami & Pol Herrmann & Doug Walker, 2018. "Innovation pathway to profitability: the role of entrepreneurial orientation and marketing capabilities," Journal of the Academy of Marketing Science, Springer, vol. 46(4), pages 744-766, July.
    3. Timothy Erickson & Toni M. Whited, 2000. "Measurement Error and the Relationship between Investment and q," Journal of Political Economy, University of Chicago Press, vol. 108(5), pages 1027-1057, October.
    4. Paul W. Miller & Barry R. Chiswick, 2002. "Immigrant earnings: Language skills, linguistic concentrations and the business cycle," Journal of Population Economics, Springer;European Society for Population Economics, vol. 15(1), pages 31-57.
    5. Fors, Gunnar & Zejan, Mario, 1996. "Overseas R&D by Multinationals in foreign Centers of Excellence," SSE/EFI Working Paper Series in Economics and Finance 111, Stockholm School of Economics.
    6. Rodrigo M. S. Moita & Claudio Paiva, 2013. "Political Price Cycles in Regulated Industries: Theory and Evidence," American Economic Journal: Economic Policy, American Economic Association, vol. 5(1), pages 94-121, February.
    7. repec:spo:wpmain:info:hdl:2441/7172 is not listed on IDEAS
    8. Butler, Marty & Leone, Andrew J. & Willenborg, Michael, 2004. "An empirical analysis of auditor reporting and its association with abnormal accruals," Journal of Accounting and Economics, Elsevier, vol. 37(2), pages 139-165, June.
    9. Baiyegunhi, L.J.S. & Oppong, B.B., 2016. "Commercialisation of mopane worm (Imbrasia belina) in rural households in Limpopo Province, South Africa," Forest Policy and Economics, Elsevier, vol. 62(C), pages 141-148.
    10. MacKinnon, J G, 1989. "Heteroskedasticity-Robust Tests for Structural Change," Empirical Economics, Springer, vol. 14(2), pages 77-92.
    11. Fenech, Jean-Pierre & Skully, Michael & Xuguang, Han, 2014. "Franking credits and market reactions: Evidence from the Australian convertible security market," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 32(C), pages 1-19.
    12. François Desmoulins-Lebeault & Jean-François Gajewski & Luc Meunier, 2018. "Personality and Risk Aversion," Economics Bulletin, AccessEcon, vol. 38(1), pages 472-489.
    13. Benjamin M. Blau & Ryan J. Whitby, 2014. "Speculative Trading In Reits," Journal of Financial Research, Southern Finance Association;Southwestern Finance Association, vol. 37(1), pages 55-74, February.
    14. Bliss, Mark A. & Gul, Ferdinand A., 2012. "Political connection and leverage: Some Malaysian evidence," Journal of Banking & Finance, Elsevier, vol. 36(8), pages 2344-2350.
    15. Gu, Chen & Kurov, Alexander & Wolfe, Marketa Halova, 2018. "Relief Rallies after FOMC Announcements as a Resolution of Uncertainty," Journal of Empirical Finance, Elsevier, vol. 49(C), pages 1-18.
    16. Bierens, H.J. & Broersma, L., 1991. "The relation between unemployment and interest rate : some international evidence," Serie Research Memoranda 0112, VU University Amsterdam, Faculty of Economics, Business Administration and Econometrics.
    17. Son K. Lam & Thomas E. DeCarlo & Ashish Sharma, 2019. "Salesperson ambidexterity in customer engagement: do customer base characteristics matter?," Journal of the Academy of Marketing Science, Springer, vol. 47(4), pages 659-680, July.
    18. Gruener Hans Peter & Hayo Bernd & Hefeker Carsten, 2009. "Unions, Wage Setting and Monetary Policy Uncertainty," The B.E. Journal of Macroeconomics, De Gruyter, vol. 9(1), pages 1-25, October.
    19. Sian Owen & Jo-Ann Suchard, 2013. "The impact of venture capital/private equity investment on the performance of IPOs in Australia," Chapters, in: Mario Levis & Silvio Vismara (ed.), Handbook of Research on IPOs, chapter 19, pages 400-420, Edward Elgar Publishing.
    20. Jongmoo Jay Choi & Hoje Jo & Jimi Kim & Moo Sung Kim, 2018. "Business Groups and Corporate Social Responsibility," Journal of Business Ethics, Springer, vol. 153(4), pages 931-954, December.
    21. Michael Fritsch & Viktor Slavtchev, 2007. "What determines the efficiency of regional innovation systems?," Jena Economics Research Papers 2007-006, Friedrich-Schiller-University Jena.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:66:y:2020:i:8:p:3412-3424. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.