IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v53y2007i2p308-322.html
   My bibliography  Save this article

Bias and Variance Approximation in Value Function Estimates

Author

Listed:
  • Shie Mannor

    (Department of Electrical and Computer Engineering, McGill University, Montreal, Quebec, Canada H3A 2A7)

  • Duncan Simester

    (Sloan School of Management, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139)

  • Peng Sun

    (Fuqua School of Business, Duke University, Durham, North Carolina 27708)

  • John N. Tsitsiklis

    (Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139)

Abstract

We consider a finite-state, finite-action, infinite-horizon, discounted reward Markov decision process and study the bias and variance in the value function estimates that result from empirical estimates of the model parameters. We provide closed-form approximations for the bias and variance, which can then be used to derive confidence intervals around the value function estimates. We illustrate and validate our findings using a large database describing the transaction and mailing histories for customers of a mail-order catalog firm.

Suggested Citation

  • Shie Mannor & Duncan Simester & Peng Sun & John N. Tsitsiklis, 2007. "Bias and Variance Approximation in Value Function Estimates," Management Science, INFORMS, vol. 53(2), pages 308-322, February.
    • Handle: RePEc:inm:ormnsc:v:53:y:2007:i:2:p:308-322
    DOI: 10.1287/mnsc.1060.0614
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mnsc.1060.0614
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mnsc.1060.0614?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Rust, John, 1987. "Optimal Replacement of GMC Bus Engines: An Empirical Model of Harold Zurcher," Econometrica, Econometric Society, vol. 55(5), pages 999-1033, September.
    2. Jeffrey I. McGill & Garrett J. van Ryzin, 1999. "Revenue Management: Research Overview and Prospects," Transportation Science, INFORMS, vol. 33(2), pages 233-256, May.
    3. Gabriel R. Bitran & Susana V. Mondschein, 1996. "Mailing Decisions in the Catalog Sales Industry," Management Science, INFORMS, vol. 42(9), pages 1364-1381, September.
    4. Keane, Michael P & Wolpin, Kenneth I, 1994. "The Solution and Estimation of Discrete Choice Dynamic Programming Models by Simulation and Interpolation: Monte Carlo Evidence," The Review of Economics and Statistics, MIT Press, vol. 76(4), pages 648-672, November.
    5. Gregory A. Godfrey & Warren B. Powell, 2002. "An Adaptive Dynamic Programming Algorithm for Dynamic Fleet Management, I: Single Period Travel Times," Transportation Science, INFORMS, vol. 36(1), pages 21-39, February.
    6. Füsun Gönül & Meng Ze Shi, 1998. "Optimal Mailing of Catalogs: A New Methodology Using Estimable Structural Dynamic Programming Models," Management Science, INFORMS, vol. 44(9), pages 1249-1262, September.
    7. Jan Roelf Bult & Tom Wansbeek, 1995. "Optimal Selection for Direct Mail," Marketing Science, INFORMS, vol. 14(4), pages 378-394.
    8. Duncan I. Simester & Peng Sun & John N. Tsitsiklis, 2006. "Dynamic Catalog Mailing Policies," Management Science, INFORMS, vol. 52(5), pages 683-696, May.
    9. Yihong Xia, 2001. "Learning about Predictability: The Effects of Parameter Uncertainty on Dynamic Asset Allocation," Journal of Finance, American Finance Association, vol. 56(1), pages 205-246, February.
    10. Charles E. Clark, 1961. "The Greatest of a Finite Set of Random Variables," Operations Research, INFORMS, vol. 9(2), pages 145-162, April.
    11. Jerzy A. Filar & L. C. M. Kallenberg & Huey-Miin Lee, 1989. "Variance-Penalized Markov Decision Processes," Mathematics of Operations Research, INFORMS, vol. 14(1), pages 147-161, February.
    12. Nicholas Barberis, 2000. "Investing for the Long Run when Returns Are Predictable," Journal of Finance, American Finance Association, vol. 55(1), pages 225-264, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Omid Rafieian, 2023. "Optimizing User Engagement Through Adaptive Ad Sequencing," Marketing Science, INFORMS, vol. 42(5), pages 910-933, September.
    2. William L. Cooper & Bharath Rangarajan, 2012. "Performance Guarantees for Empirical Markov Decision Processes with Applications to Multiperiod Inventory Models," Operations Research, INFORMS, vol. 60(5), pages 1267-1281, October.
    3. Wolfram Wiesemann & Daniel Kuhn & Berç Rustem, 2010. "Robust Markov Decision Processes," Working Papers 034, COMISEF.
    4. David L. Kaufman & Andrew J. Schaefer, 2013. "Robust Modified Policy Iteration," INFORMS Journal on Computing, INFORMS, vol. 25(3), pages 396-410, August.
    5. Varagapriya, V & Singh, Vikas Vikram & Lisser, Abdel, 2024. "Rank-1 transition uncertainties in constrained Markov decision processes," European Journal of Operational Research, Elsevier, vol. 318(1), pages 167-178.
    6. Raghuram Iyengar & Kamel Jedidi & Skander Essegaier & Peter J. Danaher, 2011. "The Impact of Tariff Structure on Customer Retention, Usage, and Profitability of Access Services," Marketing Science, INFORMS, vol. 30(5), pages 820-836, September.
    7. V Varagapriya & Vikas Vikram Singh & Abdel Lisser, 2023. "Joint chance-constrained Markov decision processes," Annals of Operations Research, Springer, vol. 322(2), pages 1013-1035, March.
    8. Huan Xu & Shie Mannor, 2012. "Distributionally Robust Markov Decision Processes," Mathematics of Operations Research, INFORMS, vol. 37(2), pages 288-300, May.
    9. Ricardo Montoya & Oded Netzer & Kamel Jedidi, 2010. "Dynamic Allocation of Pharmaceutical Detailing and Sampling for Long-Term Profitability," Marketing Science, INFORMS, vol. 29(5), pages 909-924, 09-10.
    10. Yuanhui Zhang & Haipeng Wu & Brian T. Denton & James R. Wilson & Jennifer M. Lobo, 2019. "Probabilistic sensitivity analysis on Markov models with uncertain transition probabilities: an application in evaluating treatment decisions for type 2 diabetes," Health Care Management Science, Springer, vol. 22(1), pages 34-52, March.
    11. Bren, Austin & Saghafian, Soroush, 2018. "Data-Driven Percentile Optimization for Multi-Class Queueing Systems with Model Ambiguity: Theory and Application," Working Paper Series rwp18-008, Harvard University, John F. Kennedy School of Government.
    12. Erick Delage & Shie Mannor, 2010. "Percentile Optimization for Markov Decision Processes with Parameter Uncertainty," Operations Research, INFORMS, vol. 58(1), pages 203-213, February.
    13. Maximilian Blesch & Philipp Eisenhauer, 2023. "Robust Decision-Making under Risk and Ambiguity," Rationality and Competition Discussion Paper Series 463, CRC TRR 190 Rationality and Competition.
    14. Shie Mannor & Ofir Mebel & Huan Xu, 2016. "Robust MDPs with k -Rectangular Uncertainty," Mathematics of Operations Research, INFORMS, vol. 41(4), pages 1484-1509, November.
    15. Shiau Hong Lim & Huan Xu & Shie Mannor, 2016. "Reinforcement Learning in Robust Markov Decision Processes," Mathematics of Operations Research, INFORMS, vol. 41(4), pages 1325-1353, November.
    16. Zahra Ghatrani & Archis Ghate, 2024. "Percentile optimization in multi-armed bandit problems," Annals of Operations Research, Springer, vol. 340(2), pages 837-862, September.
    17. Wolfram Wiesemann & Daniel Kuhn & Berç Rustem, 2013. "Robust Markov Decision Processes," Mathematics of Operations Research, INFORMS, vol. 38(1), pages 153-183, February.
    18. Maximilian Blesch & Philipp Eisenhauer, 2021. "Robust Decision-Making Under Risk and Ambiguity," ECONtribute Discussion Papers Series 104, University of Bonn and University of Cologne, Germany.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Romana Khan & Michael Lewis & Vishal Singh, 2009. "Dynamic Customer Management and the Value of One-to-One Marketing," Marketing Science, INFORMS, vol. 28(6), pages 1063-1079, 11-12.
    2. Durango-Cohen, Elizabeth J., 2013. "Modeling contribution behavior in fundraising: Segmentation analysis for a public broadcasting station," European Journal of Operational Research, Elsevier, vol. 227(3), pages 538-551.
    3. Bose, Indranil & Chen, Xi, 2009. "Quantitative models for direct marketing: A review from systems perspective," European Journal of Operational Research, Elsevier, vol. 195(1), pages 1-16, May.
    4. Verhoef, Peter C. & Venkatesan, Rajkumar & McAlister, Leigh & Malthouse, Edward C. & Krafft, Manfred & Ganesan, Shankar, 2010. "CRM in Data-Rich Multichannel Retailing Environments: A Review and Future Research Directions," Journal of Interactive Marketing, Elsevier, vol. 24(2), pages 121-137.
    5. Dimitris Bertsimas & Adam J. Mersereau, 2007. "A Learning Approach for Interactive Marketing to a Customer Segment," Operations Research, INFORMS, vol. 55(6), pages 1120-1135, December.
    6. George, Morris & Kumar, V. & Grewal, Dhruv, 2013. "Maximizing Profits for a Multi-Category Catalog Retailer," Journal of Retailing, Elsevier, vol. 89(4), pages 374-396.
    7. Sarkar, Mainak & De Bruyn, Arnaud, 2021. "LSTM Response Models for Direct Marketing Analytics: Replacing Feature Engineering with Deep Learning," Journal of Interactive Marketing, Elsevier, vol. 53(C), pages 80-95.
    8. Blattberg, Robert C. & Malthouse, Edward C. & Neslin, Scott A., 2009. "Customer Lifetime Value: Empirical Generalizations and Some Conceptual Questions," Journal of Interactive Marketing, Elsevier, vol. 23(2), pages 157-168.
    9. Bas Donkers & Richard Paap & Jedid‐Jah Jonker & Philip Hans Franses, 2006. "Deriving target selection rules from endogenously selected samples," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 21(5), pages 549-562, July.
    10. Mercedes Esteban-Bravo & Jose M. Vidal-Sanz & Gökhan Yildirim, 2014. "Valuing Customer Portfolios with Endogenous Mass and Direct Marketing Interventions Using a Stochastic Dynamic Programming Decomposition," Marketing Science, INFORMS, vol. 33(5), pages 621-640, September.
    11. Duncan I. Simester & Peng Sun & John N. Tsitsiklis, 2006. "Dynamic Catalog Mailing Policies," Management Science, INFORMS, vol. 52(5), pages 683-696, May.
    12. Roland T. Rust & Tuck Siong Chung, 2006. "Marketing Models of Service and Relationships," Marketing Science, INFORMS, vol. 25(6), pages 560-580, 11-12.
    13. Piersma, Nanda & Jonker, Jedid-Jah, 2004. "Determining the optimal direct mailing frequency," European Journal of Operational Research, Elsevier, vol. 158(1), pages 173-182, October.
    14. Drèze, Xavier & Bonfrer, André, 2008. "An empirical investigation of the impact of communication timing on customer equity," Journal of Interactive Marketing, Elsevier, vol. 22(1), pages 36-50.
    15. Durango-Cohen, Elizabeth J. & Torres, Ramón L. & Durango-Cohen, Pablo L., 2013. "Donor Segmentation: When Summary Statistics Don't Tell the Whole Story," Journal of Interactive Marketing, Elsevier, vol. 27(3), pages 172-184.
    16. Alan L. Montgomery, 2001. "Applying Quantitative Marketing Techniques to the Internet," Interfaces, INFORMS, vol. 31(2), pages 90-108, April.
    17. Damgaard, Mette Trier & Gravert, Christina, 2018. "The hidden costs of nudging: Experimental evidence from reminders in fundraising," Journal of Public Economics, Elsevier, vol. 157(C), pages 15-26.
    18. Holtrop, Niels & Wieringa, Jaap E., 2023. "Timing customer reactivation initiatives," International Journal of Research in Marketing, Elsevier, vol. 40(3), pages 570-589.
    19. Roland T. Rust & Peter C. Verhoef, 2005. "Optimizing the Marketing Interventions Mix in Intermediate-Term CRM," Marketing Science, INFORMS, vol. 24(3), pages 477-489, December.
    20. Klein, Robert & Kolb, Johannes, 2015. "Maximizing customer equity subject to capacity constraints," Omega, Elsevier, vol. 55(C), pages 111-125.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:53:y:2007:i:2:p:308-322. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.