This article follows the administrative usage of the term “labour” and its political effects in the period from roughly 1918–1924 in Madras Presidency, India. In this short period, I will argue, fundamental tensions in the ability of the concept to refer coherently to its object came violently to the surface. The prevailing tension in both governmental discourse and in the sphere of political representation concerned the extent to which either caste status or economic class were to be understood as the primary determinant of the meaning of labour. At the nub of this conflict lay the contested status of the descendants of hereditarily unfree labourers who supplied the bulk of the Presidency's labour requirements and were referred to in this period as Adi-Dravidas. Should they be construed as ritually disadvantaged caste subjects who also happened to labour, or as paradigmatic labourers who were also subjected to caste discrimination? Adi-Dravidas provoked both the anxiety of the elite political classes who wished to incorporate them into larger nationalist projects, as well as the reformist zeal of the colonial state, throwing the category “labour” into crisis. By navigating the use to which “labour” was put by caste elites, state officials, and Adi-Dravidas themselves, I will reflect on the coherence of caste and class as analytic concepts for political and social struggles of the kind I am describing.