Femtosecond (fs) broad-band transient absorption (TA) is used to study the intra-band relaxation and depopulation dynamics of electron and hole quantized states in CdSe nanocrystals (NC's) with a range of surface properties. Instead of the drastic reduction in the energy relaxation rate expected due to a 'phonon bottleneck', we observe a fast sub-picosecond 1P-to-1S relaxation, with the rate enhanced in NC's of smaller radius. We use fs IR TA to probe electron and hole intraband transitions, which allows us to distinguish between electron and hole pathways leading to the depopulation of NC quantized states. In contrast to electron relaxation, which is controlled by NC surface passivation, depopulation of hole quantized states is extremely fast (sub-ps-to-ps time scales) in all types samples, independent of NC surface treatment (including NC's overcoated with a ZnS layer). Our results indicate that ultrafast hole dynamics are not due to trapping at surface defects, but rather arise from relaxation into intrinsic NC states.