A class of iteration methods is introduced to find the optimal value function υ^* ∈ R ^m in a Markovian decision problem with known optimal stationary policy, represented by a (m, m)-transition-matrix P ^δ and a reward vector γ^δ∈R ^m . Depending on a (m, m)-para-meter-matrix Q, (I - Q) nonsingular, the Q-iteration (for υ^*, when P ^δ and γ ^δ are presented) is explained. Well-known methods are received for special forms of Q. An estimation characterizing the speed of convergence of the Q-iteration is given.