This article explores the relation between the concept of symmetry and its formalisms. The standard view among philosophers and physicists is that symmetry is completely formalized by mathematical groups. For some mathematicians however, the groupoid is a competing and more general formalism. An analysis of symmetry that justifies this extension has not been adequately spelled out. After a brief explication of how groups, equivalence, and symmetries classes are related, we show that, while it's true in some instances that groups are too restrictive, there are other instances for which the standard extension to groupoids is too unrestrictive. The connection between groups and equivalence classes, when generalized to groupoids, suggests a middle ground between the two.