The existence of galaxies indicates that the early universe must have been inhomogeneous and might have been highly chaotic. This could have lead to regions of the size of the particle horizon undergoing gravitational collapse to produce black holes with initial masses from 10-5 g upwards. Radiation pressure in the early Universe would cause these black holes to grow by accretion. However, despite previous expectations, this accretion would not be very much unless the initial conditions of the Universe were arranged in a special and a causal manner. Observations indicate that, at the most, only a small fraction of the matter in the early Universe can have undergone gravitational collapse.