Organisms engage in various activities that are directed at objects, whether real or imagined. Such activities may be termed “intentional relations.” We present a four-level framework of social understanding that organizes the ways in which social organisms represent the intentional relations of themselves and other agents. We presuppose that the information available to an organism about its own intentional relations (or first person information) is qualitatively different from the information available to that organism about other agents’ intentional relations (or third person information). However, through the integration of these two sources of information, it is possible to generate representations of intentional relations that are uniformly applicable to the activities of both self and other. The four levels of the framework differ in the extent to which such integration occurs and in the degree to which imagination is involved in generating these representations. Most animals exist at the lowest level, at which integration of first and third person sources of information does not occur. Of nonhuman species, only great apes exhibit social understanding at intermediate levels, at which integration of these sources of information provides uniform representations of intentional relations. Only humans attain the highest level, at which it is possible to represent intentional relations with mental objects. We propose that with the development of the imagination, children progress through three stages, equivalent to the later three levels of the framework. The abnormalities in social understanding of autistic individuals are hypothesized to result from a failure to develop integrated representations of intentional relations.