Considering the nonlocal small-scale effect and surface effect, we perform the size-dependent vibration analysis of carbon nanotube (CNT). The modified governing equations for CNT’s vibration behaviors are derived by using the nonlocal Euler–Bernoulli and Timoshenko beam models, together with the consideration of surface tension and surface elasticity. According to the numerical experiments, both small-scale effect and surface effect make a substantial difference. For flexural vibration, size effect for CNT’s vibration behaviors weakens with the increase of its diameter, but strengthens with the increase of the length–diameter ratio; for shear vibration with constant length–diameter ratio, a nonlinear correlation between size effect and CNT’s diameter exists, suggesting that there is a typical diameter for CNTs, which corresponds to the “strongest” size effect. In addition, the effects of elastic substrate modulus, temperature change, and axial preloading on the vibration behaviors and their size-dependence are analyzed, respectively.