The conventional interpretation of the quantum mechanical probability amplitudes and probabilities is examined in detail with respect to gauge transformations and the Foldy-Wouthuysen transformations. It is shown that the conventional interpretation is gauge dependent and, when applied to the relativistic (Dirac) and the non-relativistic quantum mechanics, cannot be consistent with itself through the Foldy-Wouthuysen transformations. It is also shown that these two difficulties are not present in a gauge invariant formulation that explicitly incorporates Poynting's theorem and the conservation law of energy into the definition of probability amplitudes.