IDEAS home Printed from https://ideas.repec.org/a/inm/ormksc/v36y2017i1p43-53.html
   My bibliography  Save this article

When Less Is More: Data and Power in Advertising Experiments

Author

Listed:
  • Garrett A. Johnson

    (Simon Business School, University of Rochester, Rochester, New York 14627)

  • Randall A. Lewis

    (Netflix, Los Gatos, California 95032)

  • David H. Reiley

    (Pandora Media, Oakland, California 94612; University of California at Berkeley, Berkeley, California 94720)

Abstract

Yahoo! Research partnered with a nationwide retailer to study the effects of online display advertising on both online and in-store purchases. We use a randomized field experiment on 3 million Yahoo! users who are also past customers of the retailer. We find statistically significant evidence that the retailer ads increase sales 3.6% relative to the control group. We show that control ads boost measurement precision by identifying and removing the half of in-campaign sales data that are unaffected by the ads. Less data give us 31% more precision in our estimates—equivalent to increasing our sample to 5.3 million users. By contrast, we only improve precision by 5% when we include additional covariate data to reduce the residual variance in our experimental regression. The covariate-adjustment strategy disappoints despite exceptional consumer-level data including demographics, ad exposure levels, and two years’ worth of past purchase history.

Suggested Citation

  • Garrett A. Johnson & Randall A. Lewis & David H. Reiley, 2017. "When Less Is More: Data and Power in Advertising Experiments," Marketing Science, INFORMS, vol. 36(1), pages 43-53, January.
    • Handle: RePEc:inm:ormksc:v:36:y:2017:i:1:p:43-53
    DOI: 10.1287/mksc.2016.0998
    as

    Download full text from publisher

    File URL: https://doi.org/10.1287/mksc.2016.0998
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mksc.2016.0998?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Navdeep S. Sahni, 2015. "Erratum to: Effect of temporal spacing between advertising exposures: Evidence from online field experiments," Quantitative Marketing and Economics (QME), Springer, vol. 13(3), pages 249-250, September.
    2. Duncan Simester & Yu (Jeffrey) Hu & Erik Brynjolfsson & Eric T. Anderson, 2009. "Dynamics Of Retail Advertising: Evidence From A Field Experiment," Economic Inquiry, Western Economic Association International, vol. 47(3), pages 482-499, July.
    3. Randall Lewis & David Reiley, 2014. "Online ads and offline sales: measuring the effect of retail advertising via a controlled experiment on Yahoo!," Quantitative Marketing and Economics (QME), Springer, vol. 12(3), pages 235-266, September.
    4. Thomas Blake & Chris Nosko & Steven Tadelis, 2015. "Consumer Heterogeneity and Paid Search Effectiveness: A Large‐Scale Field Experiment," Econometrica, Econometric Society, vol. 83, pages 155-174, January.
    5. Joseph O. Eastlack, Jr. & Ambar G. Rao, 1989. "Advertising Experiments at the Campbell Soup Company," Marketing Science, INFORMS, vol. 8(1), pages 57-71.
    6. Avi Goldfarb & Catherine Tucker, 2011. "Rejoinder--Implications of "Online Display Advertising: Targeting and Obtrusiveness"," Marketing Science, INFORMS, vol. 30(3), pages 413-415, 05-06.
    7. Navdeep S. Sahni, 2015. "Effect of temporal spacing between advertising exposures: Evidence from online field experiments," Quantitative Marketing and Economics (QME), Springer, vol. 13(3), pages 203-247, September.
    8. Randall Lewis & Dan Nguyen, 2015. "Display advertising’s competitive spillovers to consumer search," Quantitative Marketing and Economics (QME), Springer, vol. 13(2), pages 93-115, June.
    9. Avi Goldfarb & Catherine Tucker, 2011. "Online Display Advertising: Targeting and Obtrusiveness," Marketing Science, INFORMS, vol. 30(3), pages 389-404, 05-06.
    10. Navdeep Sahni, 2015. "Erratum to: Effect of temporal spacing between advertising exposures: Evidence from online field experiments," Quantitative Marketing and Economics (QME), Springer, vol. 13(3), pages 249-250, September.
    11. Navdeep Sahni, 2015. "Effect of temporal spacing between advertising exposures: Evidence from online field experiments," Quantitative Marketing and Economics (QME), Springer, vol. 13(3), pages 203-247, September.
    12. Michael Sinkinson & Amanda Starc, 2015. "Ask Your Doctor? Direct-to-Consumer Advertising of Pharmaceuticals," NBER Working Papers 21045, National Bureau of Economic Research, Inc.
    13. Randall A. Lewis & Justin M. Rao, 2015. "The Unfavorable Economics of Measuring the Returns to Advertising," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 130(4), pages 1941-1973.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jin Wang & Zihan Hong & Hai Long, 2023. "Digital Transformation Empowers ESG Performance in the Manufacturing Industry: From ESG to DESG," SAGE Open, , vol. 13(4), pages 21582440231, October.
    2. Francesco Decarolis & Gabriele Rovigatti, 2021. "From Mad Men to Maths Men: Concentration and Buyer Power in Online Advertising," American Economic Review, American Economic Association, vol. 111(10), pages 3299-3327, October.
    3. Black, Bernard & Hollingsworth, Alex & Nunes, Letícia & Simon, Kosali, 2022. "Simulated power analyses for observational studies: An application to the Affordable Care Act Medicaid expansion," Journal of Public Economics, Elsevier, vol. 213(C).
    4. Wei Zhou & Zidong Wang, 2020. "Competing for Search Traffic in Query Markets: Entry Strategy, Platform Design, and Entrepreneurship," Working Papers 20-12, NET Institute.
    5. Eisenberg, Matthew D. & Avery, Rosemary J. & Cantor, Jonathan H., 2017. "Vitamin panacea: Is advertising fueling demand for products with uncertain scientific benefit?," Journal of Health Economics, Elsevier, vol. 55(C), pages 30-44.
    6. Caio Waisman & Brett R. Gordon, 2023. "Multi-cell experiments for marginal treatment effect estimation of digital ads," Papers 2302.13857, arXiv.org, revised Jan 2024.
    7. Xiao Han & Menghan Zhang & Yixuan Hu & Yuan Huang, 2022. "Study on the Digital Transformation Capability of Cost Consultation Enterprises Based on Maturity Model," Sustainability, MDPI, vol. 14(16), pages 1-18, August.
    8. Osinga, Ernst C. & Zevenbergen, Menno & van Zuijlen, Mark W.G., 2019. "Do mobile banner ads increase sales? Yes, in the offline channel," International Journal of Research in Marketing, Elsevier, vol. 36(3), pages 439-453.
    9. Johannes Hermle & Giorgio Martini, 2022. "Valid and Unobtrusive Measurement of Returns to Advertising through Asymmetric Budget Split," Papers 2207.00206, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Weijia Dai & Hyunjin Kim & Michael Luca, 2023. "Frontiers: Which Firms Gain from Digital Advertising? Evidence from a Field Experiment," Marketing Science, INFORMS, vol. 42(3), pages 429-439, May.
    2. Johannes Hermle & Giorgio Martini, 2022. "Valid and Unobtrusive Measurement of Returns to Advertising through Asymmetric Budget Split," Papers 2207.00206, arXiv.org.
    3. Brett R Gordon & Kinshuk Jerath & Zsolt Katona & Sridhar Narayanan & Jiwoong Shin & Kenneth C Wilbur, 2019. "Inefficiencies in Digital Advertising Markets," Papers 1912.09012, arXiv.org, revised Feb 2020.
    4. Brett R. Gordon & Florian Zettelmeyer & Neha Bhargava & Dan Chapsky, 2019. "A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook," Marketing Science, INFORMS, vol. 38(2), pages 193-225, March.
    5. Stephan Seiler & Song Yao & Wenbo Wang, 2017. "Does Online Word of Mouth Increase Demand? (And How?) Evidence from a Natural Experiment," Marketing Science, INFORMS, vol. 36(6), pages 838-861, November.
    6. Kirthi Kalyanam & John McAteer & Jonathan Marek & James Hodges & Lifeng Lin, 2018. "Cross channel effects of search engine advertising on brick & mortar retail sales: Meta analysis of large scale field experiments on Google.com," Quantitative Marketing and Economics (QME), Springer, vol. 16(1), pages 1-42, March.
    7. Kurt P. Munz & Minah H. Jung & Adam L. Alter, 2020. "Name Similarity Encourages Generosity: A Field Experiment in Email Personalization," Marketing Science, INFORMS, vol. 39(6), pages 1071-1091, November.
    8. Christina Uhl & Nadia Abou Nabout & Klaus Miller, 2020. "How Much Ad Viewability is Enough? The Effect of Display Ad Viewability on Advertising Effectiveness," Papers 2008.12132, arXiv.org.
    9. Navdeep S. Sahni & Dan Zou & Pradeep K. Chintagunta, 2017. "Do Targeted Discount Offers Serve as Advertising? Evidence from 70 Field Experiments," Management Science, INFORMS, vol. 63(8), pages 2688-2705, August.
    10. Brett R. Gordon & Robert Moakler & Florian Zettelmeyer, 2023. "Predictive Incrementality by Experimentation (PIE) for Ad Measurement," Papers 2304.06828, arXiv.org.
    11. Min Tian & Paul R. Hoban & Neeraj Arora, 2024. "What Cookie-Based Advertising Effectiveness Fails to Measure," Marketing Science, INFORMS, vol. 43(2), pages 407-418, March.
    12. Hana Choi & Carl F. Mela & Santiago R. Balseiro & Adam Leary, 2020. "Online Display Advertising Markets: A Literature Review and Future Directions," Information Systems Research, INFORMS, vol. 31(2), pages 556-575, June.
    13. Navdeep S. Sahni & Harikesh S. Nair, 2020. "Sponsorship Disclosure and Consumer Deception: Experimental Evidence from Native Advertising in Mobile Search," Marketing Science, INFORMS, vol. 39(1), pages 5-32, January.
    14. Thomas W. Frick & Rodrigo Belo & Rahul Telang, 2023. "Incentive Misalignments in Programmatic Advertising: Evidence from a Randomized Field Experiment," Management Science, INFORMS, vol. 69(3), pages 1665-1686, March.
    15. Randall Lewis & Dan Nguyen, 2015. "Display advertising’s competitive spillovers to consumer search," Quantitative Marketing and Economics (QME), Springer, vol. 13(2), pages 93-115, June.
    16. Chen He & Tobias J. Klein, 2023. "Advertising as a Reminder: Evidence from the Dutch State Lottery," Marketing Science, INFORMS, vol. 42(5), pages 892-909, September.
    17. Wesley R. Hartmann & Daniel Klapper, 2018. "Super Bowl Ads," Marketing Science, INFORMS, vol. 37(1), pages 78-96, January.
    18. Ruichang Lu & Qiaowei Shen & Tenghui Wang & Xiaojun Zhang, 2022. "Frenemies: Corporate Advertising Under Common Ownership," Management Science, INFORMS, vol. 68(6), pages 4645-4669, June.
    19. Anja Lambrecht & Catherine Tucker & Caroline Wiertz, 2018. "Advertising to Early Trend Propagators: Evidence from Twitter," Marketing Science, INFORMS, vol. 37(2), pages 177-199, March.
    20. Caio Waisman & Navdeep S. Sahni & Harikesh S. Nair & Xiliang Lin, 2019. "Parallel Experimentation and Competitive Interference on Online Advertising Platforms," Papers 1903.11198, arXiv.org, revised Feb 2024.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormksc:v:36:y:2017:i:1:p:43-53. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.