Understanding the interaction of group V impurities with intrinsic defects in ZnO is important for developing p-type material. We have studied N-doped ZnO thin films and N-doped bulk ZnO crystals, with positron annihilation spectroscopy, in contrast to earlier studies that have concentrated on N-implanted ZnO crystals. We show that the introduction of N impurities into ZnO, irrespective of whether it is done during the growth of thin films or bulk crystals or through implantation and subsequent thermal treatments, leads to the formation of stable vacancy clusters and negative ion-type defects. Interestingly, the stability of these vacancy clusters is found almost exclusively for N introduction, whereas single Zn vacancy defects or easily removable vacancy clusters are more typically found for ZnO doped with other impurities.