Functional anatomy

Profound differences between biological networks and DNNs can be found at all organizational levels. Single neurons with their complex electro-chemical interactions are unlike the schematic DNN “neurons.” As noted by Ullman (Reference Ullman2019), almost everything known about biological neurons was left out of their DNN counterparts. Connectivity between single elements presents another striking divergence between DNNs and the visual cortex. Typically, DNNs contain many layers with units that are connected to all other units, although there is no such connectivity in the visual cortex. Lateral connections within a cortical area are sparse; in V1 the intrinsic connections are only between same orientation columns (Stettler, Aniruddha, Bennett, & Gilbert, Reference Stettler, Aniruddha, Bennett and Gilbert2002). Feed-forward connections in the ventral pathway, V1 → V2 → V4 → IT, are not one-to-one but many-to-one leading to increase in receptive field size from minutes of arc in V1 to many degrees in the IT cortex (Rolls, Reference Rolls2012). Most feedback from higher to lower areas is less dense than the feed-forward one and does not target single cells (Rolls, Reference Rolls2012). Also, DNNs are usually composed of more than 20 layers including highly specialized ones whereas there are only four areas in the visual hierarchy which are remarkably similar in their basic anatomy. While such profound structural differences are in themselves sufficient to deny that DNNs “are the best models of biological vision,” even a larger gap between DNNs and biological reality stems from ultimate differences in functionality. Note that since both expert ensembles and DNN models share essential characteristics, such as hierarchical feature extraction and integration, and since the latter differ structurally from the visual cortex, showing that the biologically based model is not a good model of the visual cortex, rules out DNNs as well.

Invariance

We recognize an object even though it can vary greatly in appearance. In Figure 1 a triangle with many properties (e.g., size) and many variations within property is displayed. Combining properties and variations leads to an astronomical number of variations, yet all objects are recognized as triangles, are differentiated from each other, and from any of the similarly astronomical possible appearances of, say, a square. It is impossible for a small number of “expert” cells to generate a pattern that is unique to every single triangle yet differs from all squares-generated patterns. The same arguments apply, obviously, to all other object categories. Are we to assume then that there are ensembles dedicated to each of the many thousands object categories out there?

Figure 1. Triangle is recognized as such despite potentially appearing in any one of an astronomical number of variations.

Space and time characteristics of object perception

A collection (~0.6° in total width) of small dots displayed for ~50 ms is easily perceived as a turtle (Fig. 2; Gur, Reference Gur2021). This result is quite informative; the display size means that the dots must be represented by V1 cells which are the only ones with small enough receptive fields. The short exposure means that correctly judging the position and relationship between dots, which is essential for a holistic object perception, cannot result from V1 → V2 → V4 → IT hierarchical convergence since there is simply not enough time (Schmolesky et al., Reference Schmolesky, Youngchang, Hanes, Thompson, Leutgeb, Schall and Leventhal1998). This example is consistent with many studies showing an accurate perception of flashed objects (cf. Crouzet, Kirchner, & Thorpe, Reference Crouzet, Kirchner and Thorpe2010; Greene & Visani, Reference Greene and Visani2015; Gur, Reference Gur2018, Reference Gur2021; Keysers, Xiao, Foldiak, & Perrett, Reference Keysers, Xiao, Foldiak and Perrett2001). Thus, unlike the predictions of the expert ensemble theory, our perception is almost instantaneous, parallel, and detail preserving.

Figure 2. Animal is recognized even when its contour is represented by dots, and is displayed for ~50 ms.

Detailed and holistic

These two characteristics are contradictory to any integrative/computational system where the whole is derived by integrating over its parts. However, this is how we see the world. It is the visual system ability to perceive space simultaneously and in parallel that leads to the holistic yet detailed capacity. The world is perceived almost in a flash-like manner; all the elements’ position and features (size, shape, orientation, etc.) are available for an immediate decision – a building or a face? Such a discrimination between objects with such a retention of details would not have been possible if elements were integrated and serially sent across many synapses to areas downstream from V1.

Finally, any computational system, including those discussed here, can, at best, map output to input. To recognize an object, the system generates a label, “object #2319764,” which is devoid of all the object's constituent elements. This result is quite different from our perceptual experience where an object is perceived together with its minute details. This discrepancy between perceptual reality and even the best possible performance of a computational system clearly means that such a system cannot be a model of the visual system.