The star formation history of galaxies is a complex process usually considered to be stochastic in nature, for which we can only give average descriptions such as the color-density relation. In this work we follow star-forming gas particles in a hydrodynamical N-body simulation back in time in order to study their initial spatial configuration. By keeping record of the time when a gas particle started forming stars we can produce Lagrangian gas-star isochrone surfaces delineating the surfaces of accreting gas that begin producing stars at different times. These surfaces form a complex a network of filaments in Eulerian space from which galaxies accrete cold gas. Lagrangian accretion surfaces are closely packed inside dense regions, intersecting each other, and as a result galaxies inside proto-clusters stop accreting gas early, naturally explaining the color dependence on density. The process described here has a purely gravitational / geometrical origin, arguably operating at a more fundamental level than complex processes such as AGN and supernovae, and providing a conceptual origin for the color-density relation.