For a non-CM elliptic curve $E/Q$, its Galois action on all its torsion points can be expressed in terms of a Galois representation. A famous theorem of Serre says that the image of this representation is as "large as possible" up to finite index. We will study what indices are possible assuming that we are willing to exclude a finite number of possible $j$-invariants from consideration.