It is shown that the high p-type conductivity in GaN:Mg, grown by metal-organic chemical vapor deposition followed by post-growth annealing, is due to non-equilibrium acceptor concentrations. A series of samples cut from a single GaN:Mg wafer, which initially had undergone rapid thermal annealing (RTA) after growth, has been investigated. The samples were annealed at various temperatures in nitrogen ambient for over 12 hours, and temperature-dependent Hall effect measurements were performed. For samples annealed at temperatures higher than 850 °C, the hole concentrations decrease by at least an order of magnitude, compared with the original sample. This behavior is explained by an Mg acceptor concentration in excess of its equilibrium solubility limit in the original sample; thus, at high enough temperatures, in the absence of hydrogen, Mg acceptors diffuse either to form electrically inactive precipitates or are eliminated. It is worth noting that the acceptor activation energy remains the same for all samples.