Spectrometer/Telescope for Imaging X-rays (STIX) is a part of Solar Orbiter (SO) science payload. SO will be launched in October 2018, and after three years of cruise phase, it will reach orbit with perihelion distance of 0.3 a.u. STIX is a Fourier imager equipped with pairs of grids that comprise the flare hard X-ray tomograph. Similar imager types were already used in the past (eq. RHESSI, Yohkoh/HXT), but STIX will incorporate Moiré modulation and a new type of pixelized detectors with CdTe sensor. We developed a method of modeling these detectors' response matrix (DRM) using the Geant4 simulations of X-ray photons interactions with CdTe crystals. Taking into account known detector effects (Fano noise, hole tailing etc.) we modeled the resulting spectra with high accuracy. Comparison of Caliste-SO laboratory measurements of 241Am decay spectrum with our results shows a very good agreement. The modeling based on the Geant4 simulations significantly improves our understanding of detector response to X-ray photons. Developed methodology gives opportunity for detailed simulation of whole instrument response with complicated geometry and secondary radiation from cosmic ray particles taken into account. Moreover, we are developing the Geant4 simulations of aging effects which decrease detector's performance.