Nanoindentation was performed to evaluate the matrix strength of the ultra-fine grained steel produced by accumulative roll-bonding and subsequently annealed. The nanohardness, associated with the matrix strength, decreases with increasing annealing temperature. Because the matrix strength corresponds to the first term σ0 of the Hall–Petch relation, this result suggests that the σ0 might not be constant for these steels between various grain sizes. Therefore, it is suggested that the change of the macroscopic strength during annealing is dominated by not only the grain coarsening leading to a reduction of grain refinement strengthening, but also the softening of the matrix strength.