Boron (B) films and B-rich BNx films with different N contents (4.1–40.3 at.%.) were deposited by dual ion-beam deposition. The films consist of a B-rich phase constructed of icosahedral atomic clusters and a graphitelike boron nitride phase. The films with N content ≤20.3 at.% is dominated by the B-rich phase. Their hardness rises with increasing N content to reach a maximum value of 18.8 GPa. The hardness-to-elastic modulus ratio (H/E) and the critical load of the films also increase, showing stronger wear resistance of the films. These results can be explained if some N–B–N chains are formed at the interstitial sites in the network of the B-rich phase, which cross-link different icosahedral atomic clusters in the B-rich phase and strengthen the rigidity of the structure. For the films with higher N contents, the volume fraction of the graphitelike boron nitride phase becomes higher, and the hardness drops as a consequence. However, the change in the H/E ratio is rather mild. This implies that the wear resistance of the films is not altered and explains why the critical load of the films remains almost unchanged. In addition, the friction coefficient μ of all the films depends on the normal load L in the form of μ = aLy, where a and y are numerical parameters and are insensitive to the change in the N content. Furthermore, compressive stress was found to increase from about 0.12 to 1.7 GPa when the N content increased from 4.1 to 40.3 at.%.