Procedures for the Solution of a Finite-Horizon, Partially Observed, Semi-Markov Optimization Problem

This paper presents a finite-time-horizon, partially observed, stochastic optimization problem, where the core process is a controlled, finite-state, discrete-time semi-Markov process. We assume that times of control reset and noise corrupted observations of the core process occur at times of core process transition. The control employed at each time of core process transition is allowed to be functionally dependent on the sample path of the core process only through the history of corrupted observations. Conditions for optimality are stated. We show that the minimum expected cost accrued from a given time t until the terminal time is concave with respect to a sufficient statistic and satisfies a computationally useful functional form. Furthermore, if the set of values that the control vector may assume at each time of control reset is finite, then the expected cost is also piecewise linear with respect to the sufficient statistic. A numerical procedure for the finite-horizon, partially observed, Markov optimization problem is generalized to include the semi-Markov formulation. We give a result that justifies a well known technique for the determination of the optimal cost and an optimal control policy and discuss areas of application.