We derive the low energy dynamics of monopoles and dyons in N = 2 supersymmetric Yang-Mills theories with hypermultiplets in arbitrary representations by utilizing a collective coordinate expansion. We consider the most general case that Higgs fields both in the vector multiplet and in the hypermultiplets have nonzero vacuum expectation values. The resulting theory is a supersymmetric quantum mechanics which has been obtained by a nontrivial dimensional reduction of two-dimensional (4,0) supersymmetric sigma models with potentials.