The central controversy over whether or not ultraluminous X-ray sources (ULXs) contain a new “intermediate-mass” class of black holes (IMBHs) remains essentially unresolved. Indeed, whilst many recent X-ray spectroscopy results find evidence for a cool (100–200 eV) accretion disc – the expected signature of a ${\sim} 1000$ M$_{\odot}$ IMBH – in ULX spectra, most of the circumstantial evidence (a combination of multiwavelength counterparts, theoretical modelling and the behaviour of accreting black holes in our own Galaxy) argues that the black holes underlying ULXs could be substantially less massive. I will present a new analysis of the deepest XMM-Newton observations of ULXs that directly addresses their underlying nature. This includes the results of a new 100-ks observation of the archetypal ULX Holmberg II X-1. Though a slight soft excess in its X-ray spectrum can be fitted by a cool accretion disc model, a rigorous analysis of the temporal data shows that the black hole cannot be larger than ${\sim}100$ M$_{\odot}$. Interestingly, we find evidence that the putative accretion disc corona is cool and optically thick in this source, unlike most Galactic binaries. We have also undertaken a detailed spectral analysis of the next 12 best ULX datasets in the XMM-Newton archive. Using physically self-consistent spectral modelling we show that whilst all the ULXs show possible cool accretion discs, the majority of these ULXs appear dominated by an optically-thick Comptonising medium. I will argue that this is evidence that most (though not necessarily all) ULXs contain black holes that are at most a few tens of solar masses in size.