Finite approximation of the first passage models for discrete-time Markov decision processes with varying discount factors

Abstract

This paper deals with the finite approximation of the first passage models for discrete-time Markov decision processes with varying discount factors. For a given control model $\mathcal {M}$ with denumerable states and compact Borel action sets, and possibly unbounded reward functions, under reasonable conditions we prove that there exists a sequence of control models $\mathcal {M}_{n}$ such that the first passage optimal rewards and policies of $\mathcal {M}_{n}$ converge to those of $\mathcal {M}$, respectively. Based on the convergence theorems, we propose a finite-state and finite-action truncation method for the given control model $\mathcal {M}$, and show that the first passage optimal reward and policies of $\mathcal {M}$ can be approximated by those of the solvable truncated finite control models. Finally, we give the corresponding value and policy iteration algorithms to solve the finite approximation models.