We present our effort to measure the proper motions of satellites in the halo of the Milky Way with mainly ground based telescopes as a precursor on what is possible with Gaia. For our first study, we used wide field optical data from the LBT combined with a first epoch of SDSS observations, on the globular cluster Palomar 5 (Pal 5). Since Pal 5 is associated with a tidal stream it is very useful to constrain the shape of the potential of the Milky Way. The motion and other properties of the Pal 5 system constrain the inner halo of the Milky Way to be rather spherical. Further, we combined adaptive optics and HST to get an absolute proper motion of the globular cluster Pyxis. Using the proper motion and the line-of-sight velocity we find that the orbit of Pyxis is rather eccentric with its apocenter at more than 100 kpc and its pericenter at about 30 kpc. The dynamics excludes an association with the ATLAS stream, the Magellanic clouds, and all satellites of the Milky Way at least down to the mass of Leo II. However, the properties of Pyxis, like metallicity and age, point to an origin from a dwarf of at least the mass of Leo II. We therefore propose that Pyxis originated from an unknown relatively massive dwarf galaxy, which is likely today fully disrupted. Assuming that Pyxis is bound to the Milky Way we derive a 68% lower limit on the mass of the Milky Way of 9.5 × 1011 M⊙.