Using spectroscopic and small angle X-ray scattering analyses, temporal evolution of ionic and neutral silver clusters inside and on the surface of a glass matrix has been studied during the ion-exchange process. It is found that in the beginning of the ion-exchange process the ionic clusters (Ag$_{N}^{+}$; where N is the number of atoms of the ionic silver cluster) form and start to grow. But, with continuation of the process the generated ionic silver clusters begin fragmentation and resizing which, in turn, results in modification of the interaction with the glass matrix. Our results show that, longer ion-exchange processing time results in re-growing of the ionic silver clusters. Simultaneously, the index of refraction of the ion-exchanged glass first increases, then decreases and increases again. Our findings imply that, there is no simple linear relation between the index of refraction and the ion-exchange duration time. Some of the ionic clusters may transform to neutral ones by absorbing electrons available in the glass matrix. X-ray and AFM analyses confirm that, at the same time, some neutral silver nanoparticles form on the surface of the samples, and their evolution follows the evolution of the forming ionic clusters. Resizing, movement toward the surface and aggregation of the clusters, are the most important consequences of the interaction of the forming clusters with the glass matrix observed in this study.