Symmetry is a salient visual feature in the natural world, yet the perception of symmetry may be influenced by how natural lighting conditions (e.g., shading) fall on the object relative to its symmetry axis. Here, we investigate how symmetry detection may interact with luminance polarity grouping, and whether this modulates neural responses to symmetry, as evidenced by the Sustained Posterior Negativity (SPN) component of Event-Related Potentials (ERPs). Stimuli were dot patterns arranged either symmetrically (reflection, rotation, translation) or quasi-randomly, and by luminance polarity about a grouping axis (i.e., black dots on one side and white dots on the other). We varied the relative angular separation between the symmetry and polarity-grouping axes: 0, 30, 60, 90 deg. Participants performed a two interval-forced-choice (2IFC) task indicating which interval contained the symmetrical pattern. We found that accuracy for the 0 deg polarity-grouped condition was higher compared to the single-polarity condition for rotation and translation (but not reflection symmetry), and higher than all other angular difference (30, 60, 90) conditions for all symmetry types. The SPN was found to be separated topographically into an early and late component, with the early SPN being sensitive to luminance polarity grouping at parietal-occipital electrodes, and the late SPN sensitive to symmetry over central electrodes. The increase in relative angular differences between luminance polarity and symmetry axes highlighted changes between cardinal (0, 90 deg) and other (30, 60 deg) angles. Critically, we found a polarity-grouping effect in the SPN time window for noise only patterns, which was related to symmetry type, suggesting a task/ symmetry pattern influence on SPN processes. We conclude that luminance polarity grouping can facilitate symmetry perception when symmetry is not readily salient, as evidenced by polarity sensitivity of early SPN, yet it can also inhibit neural and behavioral responses when luminance polarity and symmetry axes are not aligned.