We study the survival of sub-structures (clumps) within larger self-gravitating galaxies or dark matter halos. Building on scaling relations obtained from N-body calculations of violent relaxation, we argue that the tidal field of galaxies and halos can only destroy sub-structures if spherical symmetry is imposed at formation. Unless the larger halo is built up from a few large clumps, we find that clump-clump encounters are unlikely to homogenise the halo on a dynamical timescale. Phase-mixing would proceed faster in the inner parts and allow for the secular evolution of a stellar disc.