Magnesium is incorporated into the growth of Ca1-xMgxF2 to reduce the lattice constant of fluorite (CaF2) which is 0.6% larger than that of Si at room temperature. When grown epitaxially on Si(111) substrates at 300°C, the lattice constant of the alloy became smaller than that of Si by 1.5% when the Mg concentration was around 20%. At higher Mg concentrations, the lattice constant did not decrease any further. This invariability of the lattice constant was caused by a phase separation of the Ca1-xMgxF2 layer into a Mg-rich region and a Mg-deficient region. When the growth temperature was increased, the critical Mg concentration for the phase separation became smaller. When Ca1-xMgxF2 was grown on vicinal Si(111) substrates, the reflection high energy electron diffraction (RHEED) intensity oscillations reflected no change in the composition, suggesting segregation of a Mg-rich phase along the steps. Nevertheless, the oscillations in the intensity of the specular spot for Ca1-xMgxF2 lasted longer than those observed for pure CaF2, suggesting a flatter surface for the alloy. Scanning tunneling microscopy (STM) observations support this model.