IDEAS home Printed from https://ideas.repec.org/a/inm/ormksc/v30y2011i6p1115-1122.html
   My bibliography  Save this article

The Sense and Non-Sense of Holdout Sample Validation in the Presence of Endogeneity

Author

Listed:
  • Peter Ebbes

    (Fisher College of Business, Ohio State University, Columbus, Ohio 43210)

  • Dominik Papies

    (Institute for Marketing and Media, University of Hamburg, 20354 Hamburg, Germany)

  • Harald J. van Heerde

    (University of Waikato, Hamilton 3240, New Zealand; and Extramural Fellow at CentER, Tilburg University, 5000 LE Tilburg, The Netherlands)

Abstract

Market response models based on field-generated data need to address potential endogeneity in the regressors to obtain consistent parameter estimates. Another requirement is that market response models predict well in a holdout sample. With both requirements combined, it may seem reasonable to subject an endogeneity-corrected model to a holdout prediction task, and this is quite common in the academic marketing literature. One may be inclined to expect that the consistent parameter estimates obtained via instrumental variables (IV) estimation predict better than the biased ordinary least squares (OLS) estimates. This paper shows that this expectation is incorrect. That is, if the holdout sample is similar to the estimation sample so that the regressors are endogenous in both samples, holdout sample validation favors regression estimates that are not corrected for endogeneity (i.e., OLS) over estimates that are corrected for endogeneity (i.e., IV estimation). We also discuss ways in which holdout samples may be used sensibly in the presence of endogeneity. A key takeaway is that if consistent parameter estimates are the primary model objective, the model should be validated with an exogenous (rather than endogenous) holdout sample.

Suggested Citation

  • Peter Ebbes & Dominik Papies & Harald J. van Heerde, 2011. "The Sense and Non-Sense of Holdout Sample Validation in the Presence of Endogeneity," Marketing Science, INFORMS, vol. 30(6), pages 1115-1122, November.
    • Handle: RePEc:inm:ormksc:v:30:y:2011:i:6:p:1115-1122
    DOI: 10.1287/mksc.1110.0666
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mksc.1110.0666
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mksc.1110.0666?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Kleibergen, Frank & Zivot, Eric, 2003. "Bayesian and classical approaches to instrumental variable regression," Journal of Econometrics, Elsevier, vol. 114(1), pages 29-72, May.
    2. Michael Hagerty & V. Srinivasan, 1991. "Comparing the predictive powers of alternative multiple regression models," Psychometrika, Springer;The Psychometric Society, vol. 56(1), pages 77-85, March.
    3. David Besanko & Sachin Gupta & Dipak Jain, 1998. "Logit Demand Estimation Under Competitive Pricing Behavior: An Equilibrium Framework," Management Science, INFORMS, vol. 44(11-Part-1), pages 1533-1547, November.
    4. Joel H. Steckel & Wilfried R. Vanhonacker, 1993. "Cross-Validating Regression Models in Marketing Research," Marketing Science, INFORMS, vol. 12(4), pages 415-427.
    5. Scott A. Neslin, 1990. "A Market Response Model for Coupon Promotions," Marketing Science, INFORMS, vol. 9(2), pages 125-145.
    6. Bart J. Bronnenberg & Vijay Mahajan, 2001. "Unobserved Retailer Behavior in Multimarket Data: Joint Spatial Dependence in Market Shares and Promotion Variables," Marketing Science, INFORMS, vol. 20(3), pages 284-299, October.
    7. Xavier Drèze & Patricia Nisol & Naufel J. Vilcassim, 2004. "Do Promotions Increase Store Expenditures? A Descriptive Study of Household Shopping Behavior," Quantitative Marketing and Economics (QME), Springer, vol. 2(1), pages 59-92, March.
    8. Allenby, Greg M, 1990. "Cross-Validation, the Bayes Theorem, and Small-Sample Bias," Journal of Business & Economic Statistics, American Statistical Association, vol. 8(2), pages 171-178, April.
    9. Steven M. Shugan, 2009. "—Relevancy Is Robust Prediction, Not Alleged Realism," Marketing Science, INFORMS, vol. 28(5), pages 991-998, 09-10.
    10. Albert van Dijk & Harald J. van Heerde & Peter S.H. Leeflang & Dick R. Wittink, 2004. "Similarity-Based Spatial Methods to Estimate Shelf Space Elasticities," Quantitative Marketing and Economics (QME), Springer, vol. 2(3), pages 257-277, September.
    11. Andrews, Rick L. & Currim, Imran S., 2009. "Multi-stage purchase decision models: Accommodating response heterogeneity, common demand shocks, and endogeneity using disaggregate data," International Journal of Research in Marketing, Elsevier, vol. 26(3), pages 197-206.
    12. Pradeep K. Chintagunta, 2001. "Endogeneity and Heterogeneity in a Probit Demand Model: Estimation Using Aggregate Data," Marketing Science, INFORMS, vol. 20(4), pages 442-456, December.
    13. Steven T. Berry, 1994. "Estimating Discrete-Choice Models of Product Differentiation," RAND Journal of Economics, The RAND Corporation, vol. 25(2), pages 242-262, Summer.
    14. J. Miguel Villas-Boas & Russell S. Winer, 1999. "Endogeneity in Brand Choice Models," Management Science, INFORMS, vol. 45(10), pages 1324-1338, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sungho Park & Sachin Gupta, 2012. "Handling Endogenous Regressors by Joint Estimation Using Copulas," Marketing Science, INFORMS, vol. 31(4), pages 567-586, July.
    2. Guhl, Daniel, 2019. "Addressing endogeneity in aggregate logit models with time-varying parameters for optimal retail-pricing," European Journal of Operational Research, Elsevier, vol. 277(2), pages 684-698.
    3. Berman, Ron & Heller, Yuval, 2020. "Naive Analytics Equilibrium," MPRA Paper 103824, University Library of Munich, Germany.
    4. Pradeep Chintagunta & Jean-Pierre Dubé & Khim Yong Goh, 2005. "Beyond the Endogeneity Bias: The Effect of Unmeasured Brand Characteristics on Household-Level Brand Choice Models," Management Science, INFORMS, vol. 51(5), pages 832-849, May.
    5. Thomas Otter & Timothy J. Gilbride & Greg M. Allenby, 2011. "Testing Models of Strategic Behavior Characterized by Conditional Likelihoods," Marketing Science, INFORMS, vol. 30(4), pages 686-701, July.
    6. Avi Goldfarb & Qiang Lu & Sridhar Moorthy, 2009. "Measuring Brand Value in an Equilibrium Framework," Marketing Science, INFORMS, vol. 28(1), pages 69-86, 01-02.
    7. Chan, Tat Y. & Narasimhan, Chakravarthi & Yoon, Yeujun, 2017. "Advertising and price competition in a manufacturer-retailer channel," International Journal of Research in Marketing, Elsevier, vol. 34(3), pages 694-716.
    8. Haaf, C. Grace & Morrow, W. Ross & Azevedo, Inês M.L. & Feit, Elea McDonnell & Michalek, Jeremy J., 2016. "Forecasting light-duty vehicle demand using alternative-specific constants for endogeneity correction versus calibration," Transportation Research Part B: Methodological, Elsevier, vol. 84(C), pages 182-210.
    9. Timothy Richards, 2007. "A nested logit model of strategic promotion," Quantitative Marketing and Economics (QME), Springer, vol. 5(1), pages 63-91, March.
    10. Peter Ebbes, 2007. "A non-technical guide to instrumental variables and regressor-error dependencies (in Russian)," Quantile, Quantile, issue 2, pages 3-20, March.
    11. Oliver J. Rutz & George F. Watson, 2019. "Endogeneity and marketing strategy research: an overview," Journal of the Academy of Marketing Science, Springer, vol. 47(3), pages 479-498, May.
    12. Richards, Timothy J. & Hamilton, Stephen F. & Patterson, Paul M., 2010. "Spatial Competition and Private Labels," Journal of Agricultural and Resource Economics, Western Agricultural Economics Association, vol. 35(2), pages 1-26, August.
    13. Toker Doganoglu & Daniel Klapper, 2006. "Goodwill and dynamic advertising strategies," Quantitative Marketing and Economics (QME), Springer, vol. 4(1), pages 5-29, March.
    14. Yuqian Xu & Mor Armony & Anindya Ghose, 2021. "The Interplay Between Online Reviews and Physician Demand: An Empirical Investigation," Management Science, INFORMS, vol. 67(12), pages 7344-7361, December.
    15. Michaela Draganska & Dipak Jain, 2004. "A Likelihood Approach to Estimating Market Equilibrium Models," Management Science, INFORMS, vol. 50(5), pages 605-616, May.
    16. Tomohito Kamai & Yuichiro Kanazawa, 2016. "Is product with a special feature still rewarding? The case of the Japanese yogurt market," Cogent Economics & Finance, Taylor & Francis Journals, vol. 4(1), pages 1221231-122, December.
    17. David Besanko & Jean-Pierre Dubé & Sachin Gupta, 2003. "Competitive Price Discrimination Strategies in a Vertical Channel Using Aggregate Retail Data," Management Science, INFORMS, vol. 49(9), pages 1121-1138, September.
    18. Yonezawa, Koichi & Richards, Timothy J., 2016. "Competitive Package Size Decisions," Journal of Retailing, Elsevier, vol. 92(4), pages 445-469.
    19. K. Sudhir, 2001. "Competitive Pricing Behavior in the Auto Market: A Structural Analysis," Marketing Science, INFORMS, vol. 20(1), pages 42-60, January.
    20. Sriram, S. & Kadiyali, Vrinda, 2009. "Empirical investigation of channel reactions to brand introductions," International Journal of Research in Marketing, Elsevier, vol. 26(4), pages 345-355.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormksc:v:30:y:2011:i:6:p:1115-1122. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.