The existing 'residual minimization/direct inversion in the iterative subspace' (DIIS) method for the iterative calculation of low-lying eigenstates of a large matrix is further developed and modified. The DIIS method, which uses the residual minimization criterion, may fail to provide correct low-lying eigenspectra in the case of ill-formed matrices, e.g. the momentum-space representation of Hamiltonian matrices of systems containing transition metal, rare earth, or first-row elements. We suggest the inclusion of another criterion-the vanishing of the overlap integral of an iterative eigenvector with already obtained low-lying eigenvectors in order to prevent the eigenvector from collapsing to lower states. Two numerical examples of the success of our modified DIIS method in contrast to the failure of the conventional DIIS method are presented.