When an electron beam passes through or near a metal structure, it will excite surface plasmons, providing a unique way to access surface plasmon behavior at the nanoscale. An electron beam focused to nanometer dimensions thus functions as a point source that is able to probe the local plasmonic mode structure at deep-subwavelength resolution. In this article, we show how well-controlled coupling between an electron beam and surface plasmons, combined with a far-field detection system, allows characterization and manipulation of plasmons on a variety of plasmonic devices. By mapping the spatial profile of inelastic scattering to resonant modes, the dispersion and losses of surface plasmons are resolved. The technique further allows probing of the confinement of plasmons within cavities and measuring the angular emission profile of nanoantennas. The coupling of electrons to surface plasmons allows the use of the electron beam as a dipole emitter that can be positioned at will. The beam position thereby can select between modes with different symmetries. This effect can be used to exert forces on plasmonic structures on the nanometer length scale with great control.