IDEAS home Printed from https://ideas.repec.org/a/eee/spapps/v35y1990i1p27-45.html
   My bibliography  Save this article

Stochastic approximations for finite-state Markov chains

Author

Listed:
  • Ma, D.-J.
  • Makowski, A.M.
  • Shwartz, A.

Abstract

This paper develops an a.s. convergence theory for a class of projected stochastic approximations driven by finite-state Markov chains. The conditions are mild and are given explicitly in terms of the model data, mainly the Lipschitz continuity of the one-step transition probabilities. The approach used here is a version of the ODE method as proposed by Métivier and Priouret. It combines the Kushner-Clark Lemma with properties of the Poisson equation associated with the underlying family of Markov chains. The class of algorithms studied here was motivated by implementation issues for constrained Markov decision problems, where the policies of interest often depend on quantities not readily available due either to insufficient knowledge of the model parameters or to computational difficulties. This naturally leads to the on-line estimation (or computation) problem investigated here. Several examples from the area of queueing systems are discussed.

Suggested Citation

  • Ma, D.-J. & Makowski, A.M. & Shwartz, A., 1990. "Stochastic approximations for finite-state Markov chains," Stochastic Processes and their Applications, Elsevier, vol. 35(1), pages 27-45, June.
    • Handle: RePEc:eee:spapps:v:35:y:1990:i:1:p:27-45
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/0304-4149(90)90120-H
    Download Restriction: Full text for ScienceDirect subscribers only
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Beggs, Alan, 2022. "Reference points and learning," Journal of Mathematical Economics, Elsevier, vol. 100(C).
    2. Rydén, Tobias, 1997. "On recursive estimation for hidden Markov models," Stochastic Processes and their Applications, Elsevier, vol. 66(1), pages 79-96, February.
    3. Liu, Z. & Almhana, J. & Choulakian, V. & McGorman, R., 2006. "Online EM algorithm for mixture with application to internet traffic modeling," Computational Statistics & Data Analysis, Elsevier, vol. 50(4), pages 1052-1071, February.
    4. Prasenjit Karmakar & Shalabh Bhatnagar, 2018. "Two Time-Scale Stochastic Approximation with Controlled Markov Noise and Off-Policy Temporal-Difference Learning," Mathematics of Operations Research, INFORMS, vol. 43(1), pages 130-151, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:spapps:v:35:y:1990:i:1:p:27-45. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/505572/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.