IDEAS home Printed from https://ideas.repec.org/a/spr/joptap/v185y2020i3d10.1007_s10957-020-01681-2.html
   My bibliography  Save this article

Reachability and Safety Objectives in Markov Decision Processes on Long but Finite Horizons

Author

Listed:
  • Galit Ashkenazi-Golan

    (Tel-Aviv University)

  • János Flesch

    (Maastricht University)

  • Arkadi Predtetchinski

    (Maastricht University)

  • Eilon Solan

    (Tel-Aviv University)

Abstract

We consider discrete-time Markov decision processes in which the decision maker is interested in long but finite horizons. First we consider reachability objective: the decision maker’s goal is to reach a specific target state with the highest possible probability. A strategy is said to overtake another strategy, if it gives a strictly higher probability of reaching the target state on all sufficiently large but finite horizons. We prove that there exists a pure stationary strategy that is not overtaken by any pure strategy nor by any stationary strategy, under some condition on the transition structure and respectively under genericity. A strategy that is not overtaken by any other strategy, called an overtaking optimal strategy, does not always exist. We provide sufficient conditions for its existence. Next we consider safety objective: the decision maker’s goal is to avoid a specific state with the highest possible probability. We argue that the results proven for reachability objective extend to this model.

Suggested Citation

  • Galit Ashkenazi-Golan & János Flesch & Arkadi Predtetchinski & Eilon Solan, 2020. "Reachability and Safety Objectives in Markov Decision Processes on Long but Finite Horizons," Journal of Optimization Theory and Applications, Springer, vol. 185(3), pages 945-965, June.
    • Handle: RePEc:spr:joptap:v:185:y:2020:i:3:d:10.1007_s10957-020-01681-2
    DOI: 10.1007/s10957-020-01681-2
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10957-020-01681-2
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10957-020-01681-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Alexander J. Zaslavski, 2014. "Turnpike Phenomenon and Infinite Horizon Optimal Control," Springer Optimization and Its Applications, Springer, edition 127, number 978-3-319-08828-0, December.
    2. Méder, Z.Z. & Flesch, J. & Peeters, R.J.A.P., 2012. "Optimal choice for finite and infinite horizons," Research Memorandum 024, Maastricht University, Maastricht Research School of Economics of Technology and Organization (METEOR).
    3. Andrzej S. Nowak & Oscar Vega-Amaya, 1999. "A counterexample on overtaking optimality," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 49(3), pages 435-439, July.
    4. Arie Leizarowitz, 1996. "Overtaking and Almost-Sure Optimality for Infinite Horizon Markov Decision Processes," Mathematics of Operations Research, INFORMS, vol. 21(1), pages 158-181, February.
    5. Emilio De Santis & Fabio Spizzichino, 2016. "Usual and stochastic tail orders between hitting times for two Markov chains," Applied Stochastic Models in Business and Industry, John Wiley & Sons, vol. 32(4), pages 526-538, July.
    6. János Flesch & Arkadi Predtetchinski & Eilon Solan, 2017. "Sporadic Overtaking Optimality in Markov Decision Problems," Dynamic Games and Applications, Springer, vol. 7(2), pages 212-228, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. János Flesch & Arkadi Predtetchinski & Eilon Solan, 2017. "Sporadic Overtaking Optimality in Markov Decision Problems," Dynamic Games and Applications, Springer, vol. 7(2), pages 212-228, June.
    2. Adam Jonsson, 2023. "An axiomatic approach to Markov decision processes," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 97(1), pages 117-133, February.
    3. Francesco Bartaloni, 2021. "Existence of the Optimum in Shallow Lake Type Models with Hysteresis Effect," Journal of Optimization Theory and Applications, Springer, vol. 190(2), pages 358-392, August.
    4. Duersch, Peter & Oechssler, Jörg & Schipper, Burkhard C., 2012. "Unbeatable imitation," Games and Economic Behavior, Elsevier, vol. 76(1), pages 88-96.
    5. Grüne, Lars & Semmler, Willi & Stieler, Marleen, 2015. "Using nonlinear model predictive control for dynamic decision problems in economics," Journal of Economic Dynamics and Control, Elsevier, vol. 60(C), pages 112-133.
    6. Nowak, Andrzej S., 2008. "Equilibrium in a dynamic game of capital accumulation with the overtaking criterion," Economics Letters, Elsevier, vol. 99(2), pages 233-237, May.
    7. Duersch, Peter & Oechssler, Jörg & Schipper, Burkhard C., 2012. "Unbeatable imitation," Games and Economic Behavior, Elsevier, vol. 76(1), pages 88-96.
    8. Rui Fang & Xiaohu Li, 2020. "A stochastic model of cyber attacks with imperfect detection," Communications in Statistics - Theory and Methods, Taylor & Francis Journals, vol. 49(9), pages 2158-2175, May.
    9. Lars Grüne & Christopher M. Kellett & Steven R. Weller, 2017. "On the Relation Between Turnpike Properties for Finite and Infinite Horizon Optimal Control Problems," Journal of Optimization Theory and Applications, Springer, vol. 173(3), pages 727-745, June.
    10. Vivek S. Borkar & Vladimir Gaitsgory, 2019. "Linear Programming Formulation of Long-Run Average Optimal Control Problem," Journal of Optimization Theory and Applications, Springer, vol. 181(1), pages 101-125, April.
    11. Alessandro Arlotto & J. Michael Steele, 2016. "A Central Limit Theorem for Temporally Nonhomogenous Markov Chains with Applications to Dynamic Programming," Mathematics of Operations Research, INFORMS, vol. 41(4), pages 1448-1468, November.
    12. Arie Leizarowitz, 2003. "An Algorithm to Identify and Compute Average Optimal Policies in Multichain Markov Decision Processes," Mathematics of Operations Research, INFORMS, vol. 28(3), pages 553-586, August.
    13. Arie Leizarowitz & Alexander J. Zaslavski, 2007. "Uniqueness and Stability of Optimal Policies of Finite State Markov Decision Processes," Mathematics of Operations Research, INFORMS, vol. 32(1), pages 156-167, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:joptap:v:185:y:2020:i:3:d:10.1007_s10957-020-01681-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.