The HeRGÉ (Herschel Radio Galaxy Evolution) project consists of a sample of 70 radio galaxies in the range 1 < z < 5.2. They benefit from continuous coverage from 3 to 870μm with Spitzer, Herschel and sub-mm ground-based instruments (SCUBA, LABOCA). As a calorimeter, IR is an excellent proxy to estimate the contribution of both AGN and starburst, making of radio galaxies perfect candidates to provide new insights into the relationship between AGN and their host galaxies. The IR SED fitting with empirical templates reveals that radio galaxies are luminous and that their black holes and their host galaxies are not growing simultaneously. Extending the SED to optical/near-IR on a subsample of 12 radio galaxies spanning 1 < z < 4 reveal the necessity of three components to reproduce the observations. Making use of the evolutionary code PEGASE.3 and an AGN torus model, we are able to estimate parameters from the AGN torus, the evolved stellar population and the starburst (SB). They reveal that radio galaxies are massive, evolved, forming the bulk of their mass at very high redshift in a short timescale, but experience episodic, strong SB events, often associated with an AGN activity.