Skip to main content Accessibility help
  • Print publication year: 2010
  • Online publication date: July 2011

1 - Neural networks for perceptual processing: from simulation tools to theories

from Part I - General themes



This paper has two main aims. First, to give an introduction to some of the construction techniques – the ‘nuts-and-bolts’ as it were – of neural networks deployed by the authors in this book. Our intention is to emphasise conceptual principles and their associated terminology, and to do this wherever possible without recourse to detailed mathematical descriptions. However, the term ‘neural network’ has taken on a multitude of meanings over the last couple of decades, depending on its methodological and scientific context. A second aim, therefore, given that the application of the techniques described in this book may appear rather diverse, is to supply some meta-theoretical landmarks to help understand the significance of the ensuing results.

In general terms, neural networks are tools for building models of systems that are characterised by data sets which are often (but not always) derived by sampling a system input-output behaviour. While a neural network model is of some utility if it mimics the behaviour of the target system, it is far more useful if key mechanisms underlying the model functionality can be unearthed, and identified with those of the underlying system. That is, the modeller can ‘break into’ the model, viewed initially as an input-output ‘black box’, and find internal representations, variable relationships, and structures which may correspond with the underlying target system. This target system may be entirely non-biological (e.g. stock market prices), or be of biological origin, but have nothing to do with brains (e.g. ecologically driven patterns of population dynamics).

Adelson, E. H. & Movshon, J. A. 1982. Phenomenal coherence of moving visual patterns. Nature 300, 523–525.
Andersen, R. A., Essick, G. K. & Siegel, R. M. 1985. Encoding of spatial location by posterior parietal neurons. Science 230, 456–458.
Barto, A. G. 1985. Learning by statistical cooperation of sele-interested neuron-like computing elements. Hum Neurobiol 4, 229–256.
Barto, A. G. & Anandan, P. 1985. Pattern-recognizing stochastic learning automata. IEEE Syst Man CyberneticsSMC-15, 360–375.
Barto, A. G., Sutton, R. S. & Anderson, C. W. 1983. Neuronlike elements that can solve difficult learning control problems. IEEE Trans Systems Man Cybernetics 13, 835–846.
Bishop, C. M. 1996. Neural Networks for Pattern Recognition. Oxford University Press.
Cheng, B. & Titterington, D. M. 1994. Neural networks: a review from a statistical perspective. Statist Sci 9, 2–54.
Cohen, J. D., Dunbar, K. & McClelland, J. L. 1990. On the control of automatic processes – a parallel distributed-processing account of the Stroop effect. Psychol Rev 97, 332–361.
Crick, F. 1989. The recent excitement about neural networks. Nature 337, 129–132.
Dayan, P. 2002. Levels of analysis in neural modeling. In Encyclopedia of Cognitive Science (ed. L. Nadel). Nature Publishing Group; John Wiley & Sons Ltd.
Dror, I. E. & Gallogly, D. P. 1999. Computational analyses in cognitive neuroscience: in defense of biological implausibility. Psychon Bull Rev 6, 173–182.
Elman, J. L. 1990. Finding structure in time. Cogn Sci 14, 179–211.
Fitzsimonds, R. M., Song, H. J. & Poo, M. M. 1997. Propagation of activity-dependent synaptic depression in simple neural networks. Nature 388, 439–448.
Green, C. D. 2001. Scientific models, connectionist networks, and cognitive science. Theory Psychol 11, 97–117.
Gurney, K. 1997. An Introduction to Neural Networks. UCL Press (Taylor and Francis group).
Gurney, K., Prescott, T. J., Wickens, J. R. & Redgrave, P. 2004. Computational models of the basal ganglia: from robots to membranes. Trends Neurosci 27, 453–459.
Gurney, K. N. & Wright, M. J. 1992. A self-organising neural network model of image velocity encoding. Biol Cybernetics 68, 173–181.
Haykin, S. 1999. Neural Networks: A Comprehensive Foundation. Prentice Hall.
Hebb, D. 1949. The Organization of Behaviour. John Wiley.
Hinton, G. E. & Shallice, T. 1991. Lesioning an attractor network – investigations of acquired dyslexia. Psychol Rev 98, 74–95.
Hodgkin, A. L. & Huxley, A. F. 1952. A quantitative description of membrane current and its application to conduction and excitation in nerve. J Physiol 117, 500–544.
Hopfield, J. J. 1982. Neural networks and physical systems with emergent collective computational abilities. Proc Natl Acad Sci USA 79, 2554–2558.
Hubel, D. H. & Wiesel, T. N. 1968. Receptive fields and functional architecture of monkey striate cortex. J Physiol 195, 215–243.
Jacobs, R. A., Jordan, M. I. & Barto, A. G. 1991. Task decomposition through competition in a modular connectionist architecture. Cogn Sci 15, 219–250.
Jordan, M. I. 1986. Serial Order: a Parallel Distributed Approach. University of California, Institute for Cognitive Science.
Koch, C. 1999. The Biophysics of Computation: Information Processing in Single Neurons. Oxford University Press.
Koch, C., Mo, C. & Softky, W. 2002. Single-cell models. In The Handbook of Brain Theory and Neural Networks (ed. Arbib, M.). MIT Press.
Kohonen, T. 1984. Self-organization and Associative Memory. Springer Verlag.
Lippmann, R. 1987. An introduction to computing with neural nets. ASSP Magazine, IEEE 4, 4–22.
MacLeod, C. M. 1991. Half a century of research on the Stroop effect – an integrative review. Psychol Bull 109, 163–203.
Makhoul, J., El-Jaroudi, A. & Schwartz, R. 1989. Formation of disconnected decision regions with a single hidden layer. In International Joint Conference on Neural Networks, Vol. 1, pp. 455–460.
Marr, D. 1982. Vision: A Computational Investigation into Human Representation and Processing of Visual Information. W.H. Freeeman and Co.
Marr, D. & Poggio, T. 1976. From Understanding Computation to Understanding Neural Circuitry. MIT AI Laboratory.
McClelland, J. L., McNaughton, B. L. & O' Reilly, R. C. 1995. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychol Rev 102, 419–457.
McClelland, J. L. & Rumelhart, D. E. 1985. Distributed memory and the representation of general and specific information. J Exp Psychol Gen 114, 159–188.
McCloskey, M. 1991. Networks and theories – the place of connectionism in cognitive science. Psychol Sci 2, 387–395.
McCulloch, W. S. & Pitts, W. 1943. A logical calculus of the ideas immanent in nervous activity. Bull Math Biophysics 7, 115–133.
Minsky, M. & Papert, S. 1969. Perceptrons. MIT Press.
Movshon, J. A., Adelson, E. H., Gizzi, M. S. & Newsom, W. T. 1985. The analysis of moving visual patterns. In Pattern Recognition Mechanisms (ed. Chagas, C., Gattas, R. & Gross, C. G.), pp. 117–151. Springer-Verlag.
Parker, D. B. 1982. Learning-logic. Office of Technology Licensing, Stanford University.
Price, D. & Willshaw, D. 2000. Mechanisms of Cortical Development. Oxford University Press.
Rieke, F., Warland, D., Ruyter van Steveninck, R. & Bialek, W. 1996. Spikes. MIT Press.
Rosenblatt, F. 1958. The perceptron: a probabilistic model for information storage and organization in the brain. Psychol Rev 65, 386–408.
Rosin, P. L. & Fierens, F. 1995. Improving neural net generalisation. In Proceedings of IGARSS'95. Firenze, Italy.
Rumelhart, D. E., McClelland, J. L. & ,The PDP Research Group. 1986. Parallel Distributed Processing: Explorations in the Microstructure of Cognition. MIT Press.
Rumelhart, D. E. & Todd, P. 1993. Learning and connectionist representations. In Attention and Performance XIV (ed. Meyer, D. & Kornblum, S.), pp. 3–31. MIT Press.
Shang, Y. & Wah, B. W. 1996. Global optimization for neural network training. IEEE Computer 29, 45–54.
Shepherd, G. M. & Koch, C. 1998. Introduction to synaptic circuits. In The Synaptic Organization of the Brain (ed. G. M. Shepherd), pp. 1–36. Oxford University Press.
Smolensky, P. 1988. On the proper treatment of connectionism. Behav Brain Sci 11, 1–23.
Stone, M. 1974. Cross-validatory choice and assesment of statistical predictions. J R Statist Soc B36, 111–133.
Stroop, J. R. 1935. Studies of interference in serial verbal reactions. J Exp Psychol 18, 643–662.
Sutton, R. S. 1988. Learning to predict by the method of temporal differences. Machine Learn 3, 9–44.
Sutton, R. S. & Barto, A. G. 1998. Reinforcement: An Introduction. MIT Press.
Swindale, N. V. 1980. A model for the formation of ocular dominance stripes. Proc R Soc B 208, 243–264.
Touretzky, D. S. 1995. Connectionist and symbolic representations. In The Handbook of Brain Theory and Neural Networks, 1st Edn. (ed. Arbib, M.), pp. 243–247. MIT Press.
Tveter, D. R. 1996. Backpropagator's review.
Vallortigara, G. & Bressan, P. 1991. Occlusion and the perception of coherent motion. Vision Res 31, 1967–1978.
der Malsburg, C. 1973. Self-organization of orientation sensitive cells in the striate cortex. Kybernetik 14, 85–100.
Wallach, H. 1976. On perceived identity 1: The direction of motion of straight lines. In On Perception (ed. Wallach, H.). Quadrangle.
Werbos, P. 1974. Beyond Regression: New Tools for Prediction and Analysis in the Behavioural Sciences. Harvard University.
Widrow, B. & Hoff Jr, M. E. 1960. Adaptive switching circuits. In IRE WESCON Convention Record, pp. 96–104.
Widrow, B. & Stearns, S. D. 1985. Adaptive Signal Processing. Prentice Hall.
Wieland, A. & Leighton, R. 1987. Geometric analysis of neural network capabilities. In 1st IEEE International Conference on Neural Networks, Vol. III, pp. 385–392. San Diego, California.
Williams, R. J. 1987. Reinforcement Learning Connectionist Systems. Northeastern University, Boston.
Willshaw, D. J. & der Malsburg, C. 1976. How patterned neural connections can be set up by self-organization. Proc R Soc B 194, 431–445.
Zipser, D. 1992. Identification models of the nervous system. Neuroscience 47, 853–862.
Zipser, D. & Andersen, R. A. 1988. A back-propagation programmed network that simulates response properties of a subset of posterior parietal neurons. Nature 331, 679–684.