Accumulating evidence indicates that some of ultra-luminous X-ray sources (ULXs) are intermediate mass black holes (IMBHs), but the formation process of IMBHs is unknown. One possibility is that they were formed as remnants of population III (Pop III) stars, but it has been thought that the probability of being an ULX is too low for IMBHs distributed in galactic haloes to account for the observed number of ULXs. Here we show that the number of ULXs can be explained by such halo IMBHs passing through a dense molecular cloud, if Pop III star formation is very efficient as recently suggested by the excess of the cosmic near-infrared background radiation that cannot be accounted for by normal galaxy populations. We calculate the luminosity function of X-ray sources in our scenario and find that it is consistent with observed data. Our scenario can explain that ULXs are preferentially found at outskirts of large gas concentrations in star forming regions. A few important physical effects are pointed out and discussed, including gas dynamical friction, radiative efficiency of accretion flow, and radiative feedback to ambient medium. ULXs could last for ${\sim}10^{5-6}$ yr to emit a total energy of ${\sim}10^{53}$ erg, which is sufficient to power the ionized expanding nebulae found by optical observations.