We discuss the
$\ell $
-adic case of Mazur’s ‘Program B’ over
$\mathbb {Q}$
: the problem of classifying the possible images of
$\ell $
-adic Galois representations attached to elliptic curves E over
$\mathbb {Q}$
, equivalently, classifying the rational points on the corresponding modular curves. The primes
$\ell =2$
and
$\ell \ge 13$
are addressed by prior work, so we focus on the remaining primes
$\ell = 3, 5, 7, 11$
. For each of these
$\ell $
, we compute the directed graph of arithmetically maximal
$\ell $
-power level modular curves
$X_H$
, compute explicit equations for all but three of them and classify the rational points on all of them except
$X_{\mathrm {ns}}^{+}(N)$
, for
$N = 27, 25, 49, 121$
and two-level
$49$
curves of genus
$9$
whose Jacobians have analytic rank
$9$
.
Aside from the
$\ell $
-adic images that are known to arise for infinitely many
${\overline {\mathbb {Q}}}$
-isomorphism classes of elliptic curves
$E/\mathbb {Q}$
, we find only 22 exceptional images that arise for any prime
$\ell $
and any
$E/\mathbb {Q}$
without complex multiplication; these exceptional images are realised by 20 non-CM rational j-invariants. We conjecture that this list of 22 exceptional images is complete and show that any counterexamples must arise from unexpected rational points on
$X_{\mathrm {ns}}^+(\ell )$
with
$\ell \ge 19$
, or one of the six modular curves noted above. This yields a very efficient algorithm to compute the
$\ell $
-adic images of Galois for any elliptic curve over
$\mathbb {Q}$
.
In an appendix with John Voight, we generalise Ribet’s observation that simple abelian varieties attached to newforms on
$\Gamma _1(N)$
are of
$\operatorname {GL}_2$
-type; this extends Kolyvagin’s theorem that analytic rank zero implies algebraic rank zero to isogeny factors of the Jacobian of
$X_H$
.