Fundamental relations of surface instability for micro-scale multilayer thin film systems are derived based on the Hill and Hutchinson bifurcation theory and the volume average integral. In the present relations, a size effect is considered through generalizing the plastic hardening modulus to include the strain gradient hardening effects. By using the model, firstly, the parameter-controlled instability region is divided and analyzed, secondly, the surface instability of the multilayer thin films are analyzed. In the surface instability analysis, several kinds of failure band formations are obtained, such as the kink bands and the compressive buckling bands in vertical direction and in inclined direction. Moreover, the failure mechanism of the multilayer thin film system due to surface instability is used to interpret the microstructure features of the surface-nanocrystallization materials.