Commonly, the presence of grain boundaries in crystalline solids leads to enhanced atomic transport because atoms can move faster along such two-dimensional lattice defects than through the lattice itself. Recent experiments on gold diffusion into silicon, however, showed for the first time a retardation of the metallic impurity incorporation in polycrystalline material compared to that of monocrystals of the same host. The present paper provides a new model that accounts for the experimental findings with regard to Si:Au. In addition, it may apply to other systems in which a high mobility of the impurity in the lattice goes along with extremely strong segregation effects.