O2 (x = 0.05, 0.10, and 0.15) nanoparticles with tetragonal structure have been successfully synthesized by solvothermal method using oxalate precursor route. Powder x-ray diffraction and selected area electron diffraction studies confirmed highly crystalline cassiterite SnO2 structure. The contraction of lattice constants confirmed the incorporation of Co2+ in SnO2 host lattice. Hexagonal nanoparticles with average grain size of 8–13 nm have been formed. With the increasing Co content, the decreasing crystallite size of SnO2 with increasing surface areas from 194 to 219 m2/g was found. The percentage reflectance increases on increasing the cobalt concentration, and a noticeable blue shift appeared. The band gap was found to be 3.85, 3.91, and 4.09 eV, respectively. Co-doped SnO2 showed distinct magnetic behavior with different Co2+ concentration. For x = 0.05 and 0.10, nanoparticles showed paramagnetism with antiferromagnetic interaction, however, on further increasing x = 0.15, the nanoparticles showed canted antiferromagnetic coupling.