This paper reviews what we know on the Extragalactic Background (EB) and on the nature of its contributing galaxies in the infrared (IR) and sub-millimeter (sub-mm). I first present the EB observationnal constraints, their main cosmological implications, and results from deep IR and sub-mm surveys. Then, using a phenomelogical model of IR galaxy evolution, I show how these data constrain the evolution of IR galaxies, in particular the luminosity density and the luminosity function. And finally, I briefly review the detection of the IR EB fluctuations and their potential implications.