Perennial grain crops are expected to sequester soil carbon (C) and improve soil health due to their large and extensive root systems. To examine the rate of initial soil C accumulation in a perennial grain crop, we compared soil under perennial intermediate wheatgrass (IWG) with that under annual winter wheat 4 years after the crops were first planted. In addition, we tested the effect of three nitrogen (N) sources on C pools: Low available N (Low N (Organic N); 90 kg N ha−1 poultry litter), moderately available N (Mid N; 90 kg N ha−1 urea) and high available N (High N; 135 kg N ha−1 urea). We measured aboveground C (grain + straw), and coarse and fine root C to a depth of 1 m. Particulate organic matter (POM-C), fractionated by size, was used to indicate labile and more stabilized soil C pools. At harvest, IWG had 1.9 times more straw C and up to 15 times more root C compared with wheat. There were no differences in the size of the large (6 mm–250 µm) or medium (250–53 µm) POM-C fractions between wheat and IWG (P > 0.05) in surface horizons (0–10 cm). Large POM-C under IWG ranged from 3.6 ± 0.3 to 4.0 ± 0.7 g C kg soil−1 across the three N rates, similar to wheat under which large POM-C ranged from 3.6 ± 1.4 to 4.7 ± 0.7 g C kg soil−1. Averaged across N level, medium POM-C was 11.1 ± 0.8 and 11.3 ± 0.7 g C kg soil−1 for IWG and wheat, respectively. Despite IWG's greater above and belowground biomass (to 70 cm), POM-C fractions in IWG and wheat were similar. Post-hoc power analysis revealed that in order to detect differences in the labile C pool at 0–10 cm with an acceptable power (~80%) a 15% difference would be required between wheat and IWG. This demonstrates that on sandy soils with low cation exchange capacity, perennial IWG will need to be in place for longer than 4 years in order to detect an accumulated soil C difference > 15%.