We discuss hypotheses that link the measurements we can make with infants to inferences about their developing neural mechanisms. First, we examine evidence from the sensitivity to visual stimulus properties seen in infants’ responses, using both electrophysiological measures (transient and steady-state recordings of visual evoked potentials/visual event-related potentials) and behavioral measures and compare this with the sensitivity of brain processes, known from data on mammalian neurophysiology and human neuroimaging. The evidence for multiple behavioral systems with different patterns of visual sensitivity is discussed. Second, we consider the analogies which can be made between infants’ behavior and that of adults with identified brain damage, and extend these links to hypothesize about the brain basis of visual deficits in infants and children with developmental disorders. Last, we consider how these lines of data might allow us to form “inverse linking hypotheses” about infants’ visual experience.

Class A psychophysical observations are based on the linking hypothesis that perceptually distinguishable stimuli must correspond to different brain events. Class B observations are related to the appearance of stimuli not their discriminability. There is no clear linking hypothesis underlying Class B observations, but they are necessary for studying the effects of context on appearance, including a large class of phenomena known as “illusions.” Class B observations are necessarily measures of observer bias (Fechner’s “constant error”) as opposed to Class A measures of sensitivity (Fechner’s “variable error”). It is therefore important that Class B observations distinguish between response biases, decisional biases, and perceptual biases. This review argues that the commonly used method of single stimuli fails to do this, and that multiple-alternative forced choice (mAFC) methods can do a better job, particularly if combined with a roving pedestal.

When I first came across William James’ dictum that “ … this sense of sameness is the very keel and backbone of our thinking,” I thought he had foreseen the importance of cross-correlation in the brain, and told myself to find out how he had reached this conclusion. When I finally did this a year or two ago, I slowly came to realize that I had completely misunderstood him; from the full quote it is absolutely clear that his dictum cannot be referring to the process by which a cortical simple cell responds selectively to the orientation of features in a visual image, as I had originally supposed. If one translates the original dictum into two more prosaic modern versions, his version would say: “Our minds could not think at all without neural circuits in our brains that compute auto-correlations,” but in my mistaken interpretation the last word would be “cross-correlations.” Others may have made the same mistake, but the difference is profound, and finding what he really meant has been a revelation to me. This essay explains the revelation, describes how to determine experimentally whether the brain does auto- or cross-correlation, and gives the result of preliminary experiments showing clearly that it does both. A revised view of the visual cortex as autocorrelator as well as cross-correlator claims to tell us what complex cells in the visual cortex do, and it assigns a role to its columnar structure that is as important to fulfilling that role as the concept of the receptive field has been to understanding the simple cells’ fulfillment of theirs. The new view has compelling features, broad implications, and suggests a plausible model of how neural circuits in the cortex achieve thought, but it needs further testing.

The notion of a set of neurons that form a “bridge locus” serving as the immediate substrate of visual perception is examined in the light of evidence on the architecture of the visual pathway, of current thinking about perceptual representations, and of the basis of perceptual awareness. The bridge locus is likely to be part of a tangled web of representations, and this complexity raises the question of whether another scheme that relies less on geography might offer a better framework. The bridge locus bears a close relationship to the neural correlate of consciousness (NCC), and like the NCC may be a concept which is no longer precise enough to provide a useful basis for reasoning about the relationship between brain activity and perceptual experience.

Linking propositions have played an important role in refining our understanding of the relationship between neural activity and perception. Over the last 40 years, visual evoked potentials (VEPs) have been used in many different ways to address questions of the relationship between neural activity and perception. This review organizes and discusses this research within the linking proposition framework developed by Davida Teller, and her colleagues. A series of examples from the VEP literature illustrates each of the five classes of linking propositions originally proposed by Davida Teller. The related concept of the bridge locus—the site at which neural activity can be said to first be proscriptive of perception—is discussed and a suggestion is made that the concept be expanded to include an evolution over time and cortical area.

The rapid advances in brain imaging technology over the past 20 years are affording new insights into cortical processing hierarchies in the human brain. These new data provide a complementary front in seeking to understand the links between perceptual and physiological states. Here we review some of the challenges associated with incorporating brain imaging data into such “linking hypotheses,” highlighting some of the considerations needed in brain imaging data acquisition and analysis. We discuss work that has sought to link human brain imaging signals to existing electrophysiological data and opened up new opportunities in studying the neural basis of complex perceptual judgments. We consider a range of approaches when using human functional magnetic resonance imaging to identify brain circuits whose activity changes in a similar manner to perceptual judgments and illustrate these approaches by discussing work that has studied the neural basis of 3D perception and perceptual learning. Finally, we describe approaches that have sought to understand the information content of brain imaging data using machine learning and work that has integrated multimodal data to overcome the limitations associated with individual brain imaging approaches. Together these approaches provide an important route in seeking to understand the links between physiological and psychological states.

Almost 40 years ago, Davida Teller developed the forced-choice preferential looking method for studying infant visual capabilities and used it to study infant color vision. About 10 years ago, she used infant looking preferences to study infant color perception. Here, we examine four data sets in which the infant looking preference was measured using a wide range of saturated colors. Three of those data sets, from papers by Marc Bornstein and by Davida Teller and Anna Franklin and their respective collaborators, were fit successfully using MacLeod and Boynton’s model of the equiluminant plane in color space, in spite of the varied luminances used in those studies. A fourth data set, from a paper by Zemach, Chang, and Teller, was less well fit by that model. Apparently, infants are able to ignore luminance, and pay attention just to the color of stimuli. These results are discussed in the context of Davida Teller’s work on the philosophy of vision science.

The current study tested the development of face and object processing in young children (mean age = 5.24 years), adolescents (mean age = 15.8 years), and adults (mean age = 21.1 years) using stimuli that were equated for low-level visual characteristics (luminance, contrast, and spatial frequency make-up) and methods that equate for difficulty across ages. We also tested sensitivity to luminance and chromatic contrast (i.e., thought to be mediated primarily by the subcortical Magnocellular (M) and Parvocellular (P) pathways, respectively) to determine whether age-related improvements in face or object discrimination were driven by age-related changes in the M and/or P pathways. Results showed a selective age-related improvement in face sensitivity and a relationship between age-related increases in face sensitivity and luminance contrast sensitivity. These results add to the mounting evidence that the M pathway may influence face processing.

Lateral spatial interactions among elements of a scene, which either enhance or degrade visual performance, are ubiquitous in vision. The neural mechanisms underlying lateral spatial interactions are a matter of debate, and various hypotheses have been proposed. Suppressive effects may be due to local inhibitory interactions, whereas facilitatory effects are typically ascribed either to the function of long-range horizontal projections in V1 or to uncertainty reduction. We investigated the development of lateral spatial interactions, facilitation and suppression, and compared their developmental profiles to those of potential underlying mechanisms in the visual system of infant macaques. Animals ranging in age from 10 weeks to 3 years were tested with a lateral masking paradigm. We found that suppressive interactions are present from very early in postnatal life, showing no change over the age range tested. However, facilitation develops slowly over the first year after birth. Our data suggest that the early maturation of suppressive interactions is related to the relatively mature receptive field properties of neurons in early visual cortical areas near birth in infant macaques, whereas the later maturation of facilitation is unlikely to be explained by development of local or long-range connectivity in primary visual cortex. Instead our data favor a late developing feedback or top-down cognitive process to explain the origin of facilitation.

Because targeted early experiential manipulations alter both perception and the response properties of particular cells in the striate cortex, they have been used as evidence for linking hypotheses between the two. However, such hypotheses assume that the effects of the early biased visual input are restricted to just the specific cell population and/or visual areas of interest and that the neural populations that contribute to the visual perception itself do not change. To examine this assumption, we measured the consequences for vision of an extended period of early monocular deprivation (MD) on a kitten (from 19 to 219 days of age) that began well before, and extended beyond, bilateral ablation of visual cortical areas 17 and 18 at 132 days of age. In agreement with previous work, the lesion reduced visual acuity by only a factor of two indicating that the neural sites, other than cortical areas 17 and 18, that support vision in their absence have good spatial resolution. However, these sites appear to be affected profoundly by MD as the effects on vision were just as severe as those observed following MD imposed on normal animals. The pervasive effects of selected early visual deprivation across many cortical areas reported here and elsewhere, together with the potential for perception to be mediated at a different neural site following deprivation than after typical rearing, points to a need for caution in the use of data from early experiential manipulations for formulation of linking hypotheses.

Over the last 35 years or so, there has been substantial progress in revealing and characterizing the many interesting and sometimes mysterious sensory abnormalities that accompany amblyopia. A goal of many of the studies has been to try to make the link between the sensory losses and the underlying neural losses, resulting in several hypotheses about the site, nature, and cause of amblyopia. This article reviews some of these hypotheses, and the assumptions that link the sensory losses to specific physiological alterations in the brain. Despite intensive study, it turns out to be quite difficult to make a simple linking hypothesis, at least at the level of single neurons, and the locus of the sensory loss remains elusive. It is now clear that the simplest notion—that reduced contrast sensitivity of neurons in cortical area V1 explains the reduction in contrast sensitivity—is too simplistic. Considerations of noise, noise correlations, pooling, and the weighting of information also play a critically important role in making perceptual decisions, and our current models of amblyopia do not adequately take these into account. Indeed, although the reduction of contrast sensitivity is generally considered to reflect “early” neural changes, it seems plausible that it reflects changes at many stages of visual processing.

The present paper deals with the classical question how a psychological experience, in this case apparent lightness, is linked by intervening neural processing to physical variables. We address two methodological issues: (a) how does one know the appropriate physical variable (what is the right x?) to look at, and (b) how can behavioral measurements be used to probe the internal transformation that leads to psychological experience. We measured so-called lightness transfer functions (LTFs), that is the functions that describe the mapping between retinal luminance and perceived lightness for naturalistic checkerboard stimuli. The LTFs were measured for different illumination situations: plain view, a cast shadow, and an intervening transparent medium. Observers adjusted the luminance of a comparison patch such that it had the same lightness as each of the test patches. When the data were plotted in luminance–luminance space, we found qualitative differences between mapping functions in different contexts. These differences were greatly diminished when the data were plotted in terms of contrast. On contrast–contrast coordinates, the data were compatible with a single linear generative model. This result is an indication that, for the naturalistic scenes used here, lightness perception depends mostly on local contrast. We further discuss that, in addition to the mean adjustments, one may find it useful to consider also the variability of an observer’s adjustments in order to infer the true luminance-to-lightness mapping function.

The dissociation of a figure from its background is an essential feat of visual perception, as it allows us to detect, recognize, and interact with shapes and objects in our environment. In order to understand how the human brain gives rise to the perception of figures, we here review experiments that explore the links between activity in visual cortex and performance of perceptual tasks related to figure perception. We organize our review according to a proposed model that attempts to contextualize figure processing within the more general framework of object processing in the brain. Overall, the current literature provides us with individual linking hypotheses as to cortical regions that are necessary for particular tasks related to figure perception. Attempts to reach a more complete understanding of how the brain instantiates figure and object perception, however, will have to consider the temporal interaction between the many regions involved, the details of which may vary widely across different tasks.

Many complex behaviors rely on guidance from sensations. To perform these behaviors, the motor system must decode information relevant to the task from the sensory system. However, identifying the neurons responsible for encoding the appropriate sensory information remains a difficult problem for neurophysiologists. A key step toward identifying candidate systems is finding neurons or groups of neurons capable of representing the stimuli adequately to support behavior. A traditional approach involves quantitatively measuring the performance of single neurons and comparing this to the performance of the animal. One of the strongest pieces of evidence in support of a neuronal population being involved in a behavioral task comes from the signals being sufficient to support behavior. Numerous experiments using perceptual decision tasks show that visual cortical neurons in many areas have this property. However, most visually guided behaviors are not categorical but continuous and dynamic. In this article, we review the concept of sufficiency and the tools used to measure neural and behavioral performance. We show how concepts from information theory can be used to measure the ongoing performance of both neurons and animal behavior. Finally, we apply these tools to dorsal medial superior temporal (MSTd) neurons and demonstrate that these neurons can represent stimuli important to navigation to a distant goal. We find that MSTd neurons represent ongoing steering error in a virtual-reality steering task. Although most individual neurons were insufficient to support the behavior, some very nearly matched the animal’s estimation performance. These results are consistent with many results from perceptual experiments and in line with the predictions of Mountcastle’s “lower envelope principle.”

In the 1990s, seminal work from Newsome and colleagues made it possible to study the neuronal mechanisms of simple perceptual decisions. The key strength of this work was the clear and direct link between neuronal activity and choice processes. Since then, a great deal of research has extended these initial discoveries to more complex forms of decision making, with the goal of bringing the same strength of linkage between neural and psychological processes. Here, we discuss the progress of two such research programs, namely our own, that are aimed at understanding memory-guided decisions and reward-guided decisions. These problems differ in the relevant brain areas, in the progress that has been achieved, and in the extent of broader understanding achieved so far. However, they are unified by the use of theoretical insights about how to link neuronal activity to decisions.