IDEAS home Printed from https://ideas.repec.org/a/spr/annopr/v190y2011i1p289-30910.1007-s10479-010-0688-z.html
   My bibliography  Save this article

Embedding a state space model into a Markov decision process

Author

Listed:
  • Lars Relund Nielsen
  • Erik Jørgensen
  • Søren Højsgaard

Abstract

In agriculture Markov decision processes (MDPs) with finite state and action space are often used to model sequential decision making over time. For instance, states in the process represent possible levels of traits of the animal and transition probabilities are based on biological models estimated from data collected from the animal or herd. State space models (SSMs) are a general tool for modeling repeated measurements over time where the model parameters can evolve dynamically. In this paper we consider methods for embedding an SSM into an MDP with finite state and action space. Different ways of discretizing an SSM are discussed and methods for reducing the state space of the MDP are presented. An example from dairy production is given. Copyright Springer Science+Business Media, LLC 2011

Suggested Citation

  • Lars Relund Nielsen & Erik Jørgensen & Søren Højsgaard, 2011. "Embedding a state space model into a Markov decision process," Annals of Operations Research, Springer, vol. 190(1), pages 289-309, October.
    • Handle: RePEc:spr:annopr:v:190:y:2011:i:1:p:289-309:10.1007/s10479-010-0688-z
    DOI: 10.1007/s10479-010-0688-z
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s10479-010-0688-z
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s10479-010-0688-z?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. P. Diggle & M. G. Kenward, 1994. "Informative Drop‐Out in Longitudinal Data Analysis," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 43(1), pages 49-73, March.
    2. Lars Relund Nielsen & Daniele Pretolani & Kim Allan Andersen, 2009. "Bicriterion Shortest Paths in Stochastic Time-Dependent Networks," Lecture Notes in Economics and Mathematical Systems, in: Vincent Barichard & Matthias Ehrgott & Xavier Gandibleux & Vincent T'Kindt (ed.), Multiobjective Programming and Goal Programming, pages 57-67, Springer.
    3. Kristensen, Anders R., 1988. "Hierarchic Markov processes and their applications in replacement models," European Journal of Operational Research, Elsevier, vol. 35(2), pages 207-215, May.
    4. Pla, L. M. & Pomar, C. & Pomar, J., 2003. "A Markov decision sow model representing the productive lifespan of herd sows," Agricultural Systems, Elsevier, vol. 76(1), pages 253-272, April.
    5. Kristensen, Anders Ringgaard, 1993. "Bayesian Updating in Hierarchic Markov Processes Applied to the Animal Replacement Problem," European Review of Agricultural Economics, Oxford University Press and the European Agricultural and Applied Economics Publications Foundation, vol. 20(2), pages 223-239.
    6. Anders Kristensen & Erik Jørgensen, 2000. "Multi‐level hierarchic Markov processes as a framework for herd management support," Annals of Operations Research, Springer, vol. 94(1), pages 69-89, January.
    7. Nielsen, Lars Relund & Kristensen, Anders Ringgaard, 2006. "Finding the K best policies in a finite-horizon Markov decision process," European Journal of Operational Research, Elsevier, vol. 175(2), pages 1164-1179, December.
    8. Kennedy, John O. S. & Stott, Alistair W., 1993. "An adaptive decision-making aid for dairy cow replacement," Agricultural Systems, Elsevier, vol. 42(1-2), pages 25-39.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Dowson, Oscar & Philpott, Andy & Mason, Andrew & Downward, Anthony, 2019. "A multi-stage stochastic optimization model of a pastoral dairy farm," European Journal of Operational Research, Elsevier, vol. 274(3), pages 1077-1089.
    2. Borodin, Valeria & Bourtembourg, Jean & Hnaien, Faicel & Labadie, Nacima, 2016. "Handling uncertainty in agricultural supply chain management: A state of the art," European Journal of Operational Research, Elsevier, vol. 254(2), pages 348-359.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Erik Jørgensen & Anders Kristensen & Dennis Nilsson, 2014. "Markov Limid processes for representing and solving renewal problems," Annals of Operations Research, Springer, vol. 219(1), pages 63-84, August.
    2. Yates, C.M. & Rehman, T., 1998. "A linear programming formulation of the Markovian decision process approach to modelling the dairy replacement problem," Agricultural Systems, Elsevier, vol. 58(2), pages 185-201, October.
    3. Pourmoayed, Reza & Nielsen, Lars Relund & Kristensen, Anders Ringgaard, 2016. "A hierarchical Markov decision process modeling feeding and marketing decisions of growing pigs," European Journal of Operational Research, Elsevier, vol. 250(3), pages 925-938.
    4. Nielsen, Lars Relund & Kristensen, Anders Ringgaard, 2006. "Finding the K best policies in a finite-horizon Markov decision process," European Journal of Operational Research, Elsevier, vol. 175(2), pages 1164-1179, December.
    5. Yates, C. M. & Rehman, T. & Chamberlain, A. T., 1996. "Evaluation of the potential effects of embryo transfer on milk production on commercial dairy herds: The development of a markov chain model," Agricultural Systems, Elsevier, vol. 50(1), pages 65-79.
    6. Lan Ge & Anders Kristensen & Monique Mourits & Ruud Huirne, 2014. "A new decision support framework for managing foot-and-mouth disease epidemics," Annals of Operations Research, Springer, vol. 219(1), pages 49-62, August.
    7. Reza Pourmoayed & Lars Relund Nielsen, 2022. "Optimizing pig marketing decisions under price fluctuations," Annals of Operations Research, Springer, vol. 314(2), pages 617-644, July.
    8. Hegrenes, Agnar & Kristensen, Anders Ringgaard & Lien, Gudbrand D., 2003. "Optimal Economic Length Of Leys: A Dynamic Programming Approach," 2003 Annual Meeting, August 16-22, 2003, Durban, South Africa 25848, International Association of Agricultural Economists.
    9. Toft, Nils & Kristensen, Anders R. & Jorgensen, Erik, 2005. "A framework for decision support related to infectious diseases in slaughter pig fattening units," Agricultural Systems, Elsevier, vol. 85(2), pages 120-137, August.
    10. David M. Murray & Jonathan L. Blitstein, 2003. "Methods To Reduce The Impact Of Intraclass Correlation In Group-Randomized Trials," Evaluation Review, , vol. 27(1), pages 79-103, February.
    11. Patrick E. B. FitzGerald, 2002. "Extended Generalized Estimating Equations for Binary Familial Data with Incomplete Families," Biometrics, The International Biometric Society, vol. 58(4), pages 718-726, December.
    12. Pourahmadi, Mohsen & Daniels, Michael J. & Park, Trevor, 2007. "Simultaneous modelling of the Cholesky decomposition of several covariance matrices," Journal of Multivariate Analysis, Elsevier, vol. 98(3), pages 568-587, March.
    13. E. Michael Foster & Grace Y. Fang, 2004. "Alternative Methods for Handling Attrition," Evaluation Review, , vol. 28(5), pages 434-464, October.
    14. Mette Ejrnæs & Anders Holm, 2006. "Comparing Fixed Effects and Covariance Structure Estimators for Panel Data," Sociological Methods & Research, , vol. 35(1), pages 61-83, August.
    15. Miran A. Jaffa & Ayad A. Jaffa, 2019. "A Likelihood-Based Approach with Shared Latent Random Parameters for the Longitudinal Binary and Informative Censoring Processes," Statistics in Biosciences, Springer;International Chinese Statistical Association, vol. 11(3), pages 597-613, December.
    16. Shu Xu & Shelley A. Blozis, 2011. "Sensitivity Analysis of Mixed Models for Incomplete Longitudinal Data," Journal of Educational and Behavioral Statistics, , vol. 36(2), pages 237-256, April.
    17. Demeter, Robert Milan & Bovenhuis, Henk & Kristensen, Anders Ringgaard & Oude Lansink, Alfons G.J.M. & Meuwissen, Miranda P.M. & van Arendonk, Johan A.M., 2011. "Farm Level Economic Implications of Genetic Selection for Improving Milk Fat Composition," 2011 International Congress, August 30-September 2, 2011, Zurich, Switzerland 114444, European Association of Agricultural Economists.
    18. Bian, Yuan & Yi, Grace Y. & He, Wenqing, 2024. "A unified framework of analyzing missing data and variable selection using regularized likelihood," Computational Statistics & Data Analysis, Elsevier, vol. 194(C).
    19. A Ponchet & M Fouladirad & A Grall, 2011. "Maintenance policy on a finite time span for a gradually deteriorating system with imperfect improvements," Journal of Risk and Reliability, , vol. 225(2), pages 105-116, June.
    20. Jennifer Chan & Wai Wan, 2011. "Bayesian approach to analysing longitudinal bivariate binary data with informative dropout," Computational Statistics, Springer, vol. 26(1), pages 121-144, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:annopr:v:190:y:2011:i:1:p:289-309:10.1007/s10479-010-0688-z. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.