A perturbation scheme is constructed to describe the evolution of stable, localized Rankine-type hydrodynamic vortices under the action of disturbances such as density stratification. It is based on the elimination of singularities in perturbations by using the necessary orthogonality conditions which determine the vortex motion. Along with the discrete-spectrum modes of the linearized problem which can be kept finite by imposing the orthogonality conditions, the continuous-spectrum perturbations play a crucial role. It is shown that in a stratified fluid, a single (monopole) vortex can be destroyed due to the latter modes before it drifts very far, whereas a vortex pair preserves its stability for a longer time. The motion of the latter is studied in two cases: smooth stratification and a density jump. For the motion of a pair under a small angle to the interface, a complete description is given in the framework of our theory, including the effect of reflection of the pair from a region with slightly larger density.