Skip to main content Accessibility help
×
Home
  • Cited by 1
  • Print publication year: 2011
  • Online publication date: November 2011

4 - Optimised agent-based modelling of action selection

from Part I - Rational and optimal decision making

Summary

Summary

The problem of action selection has two components: what is selected? How is it selected? To understand what is selected, it is necessary to recognise that animals do not choose among behaviours per se; rather, behaviour reflects observed interactions among brains, bodies, and environments (embeddedness). To understand what guides selection, it is useful to take a normative, functional perspective that evaluates behaviour in terms of a fitness metric. This perspective can be especially useful for understanding apparently irrational action selection. Bringing together these issues therefore requires integrating function and mechanism in models of action selection. This chapter describes ‘optimised agent-based modelling’, a methodology that integrates functional and mechanistic perspectives in the context of embedded agent–environment interactions. Using this methodology, I demonstrate that successful action selection can arise from the joint activity of parallel, loosely coupled sensorimotor processes, and I show how an instance of apparently suboptimal decision making (the matching law) can be accounted for by adaptation to competitive foraging environments.

Introduction

Life is all about action. Bodies and brains have been shaped by natural selection above all for the ability to produce the right action at the right time. This basic fact leads to two observations. First, the neural substrates underpinning action selection must encapsulate mechanisms for perception as well as those supporting motor movements (Friston, 2009), and their operations must be understood in terms of interactions among brains, bodies, and environments. In other words, action selection mechanisms are embodied and embedded. Second, despite the generality of action selection mechanisms, it is unlikely that they can deliver optimal behaviour in all possible situations. Action selection models therefore need to integrate functional and mechanistic perspectives (McNamara and Houston, 2009), especially when observed behaviour departs from what appears to be optimal or ‘rational’ (Houston et al., this volume). The goal of this chapter is to describe and illustrate a methodology – optimised agent-based modelling (oABM; Seth, 2007) – that accommodates both of these observations, and to contrast this methodology with standard techniques in ‘optimal foraging theory’ (OFT; Stephens and Krebs, 1986). The central idea is that the oABM approach provides a unified framework for modelling natural action selection, ‘rational’ and otherwise.

Related content

Powered by UNSILO
References
Baum, W. M 1974 On two types of deviation from the matching law: Bias and undermatchingJ. Exp. Anal. Behav 22 231
Bitterman, M. E 1965 Phyletic differences in learningAm. Psychol 20 396
Blumberg, B 1994 Action selection in Hamsterdam: lessons from ethologyFrom Animals to Animats 3: Proceedings of the Third International Conference on the Simulation of Adaptive BehaviorCliff, DHusbands, PMeyer, J. AWilson, SCambridge, MAMIT Press107
Braitenberg, V 1984 Vehicles: Experiments in Synthetic PsychologyCambridge, MAMIT Press
Brooks, R. A 1986 A robust layered control system for a mobile robotIEEE J. Robotic. Autom 2 14
Bryson, J. J 2000 Hierarchy and sequence versus full parallelism in reactive action selection architecturesFrom Animals to Animats 6: Proceedings of the Sixth International Conference on the Simulation of Adaptive BehaviorMeyer, J. ABerthoz, AFloreano, DRoitblat, HWilson, SCambridge, MAMIT Press,147
Charnov, E 1976 Optimal foraging: the marginal value theoremTheor. Popul. Biol 9 129
Clark, A 1997 Being There. Putting Brain, Body, and World Together AgainCambridge, MAMIT Press
Davison, MMcCarthy, D 1988 The Matching LawHillsdale, NJErlbaum
Dawkins, R 1976 Hierarchical organisation: a candidate principle for ethologyGrowing Points in EthologyBateson, PHinde, RCambridgeCambridge University Press7
Dayan, P 2002 Motivated reinforcement learningAdvances in Neural Information Processing SystemsDietterich, T. GBecker, SGhahramani, ZCambridge, MAMIT Press, pp. 11–18
DeAngelis, D. LGross, L. J 1992 Individual-Based Models and Approaches in Ecology: Populations, Communities and EcosystemsLondonChapman and Hall
Di Paolo, ENoble, JBullock, S 2000 Simulation models as opaque thought experimentsArtificial Life VII: The Seventh International Conference on the Simulation and Synthesis of Living SystemsBedau, M. AMcCaskill, J. SPackard, N. HRasmussen, SPortland, ORMIT Press,497
Erev, IBarron, G 2005 On adaptation, maximization, and reinforcement learning among cognitive strategiesPsychol. Rev 112 912
Fagen, R 1987 A generalized habitat matching lawEvol. Ecol 1 5
Fretwell, S 1972 Populations in Seasonal EnvironmentsPrinceton, NJPrinceton University Press
Friedman, DMassaro, D. W 1998 Understanding variability in binary and continuous choicePsycho. B. Rev 5 370
Friston, K 2009 The free-energy principle: a rough guide to the brainTrends Cogn. Sci 13 293
Friston, K. JDaunizeau, JKiebel, S. J 2009 Reinforcement learning or active inference?PLoS One 4 e6421
Gaissmaier, WSchooler, L. J 2008 The smart potential behind probability matchingCognition 109 416
Glimcher, P. WRustichini, A 2004 Neuroeconomics: the consilience of brain and decisionScience 306 447
Gluck, M. ABower, G. H 1988 From conditioning to category learning: an adaptive network modelJ. Exp. Psychol. Gen 117 227
Goldstone, R. LAshpole, B. C 2004 Human foraging behavior in a virtual environmentPsychon. B. Rev 11 508
Goss-Custard, J 1977 Optimal foraging and size selection of worms by redshank in the fieldAnim. Behav 25 10
Grimm, V 1999 Ten years of individual-based modelling in ecology: what have we learnt, and what could we learn in the future?Ecol. Model 115 129
Grimm, VRailsback, S 2005 Individual-based Modeling and EcologyPrinceton, NJPrinceton University Press
Grimm, VRevilla, EBerger, U 2005 Pattern-oriented modeling of agent-based complex systems: lessons from ecologyScience 310 987
Hallam, JMalcolm, C 1994 Behaviour: perception, action and intelligence: the view from situated roboticsPhil. Trans. R. Soc. Lond. A 349 29
Harley, C. B 1981 Learning the evolutionarily stable strategyJ. Theor. Biol 89 611
Hendriks-Jansen, H 1996 Catching Ourselves in the Act: Situated Activity, Interactive Emergence, and Human ThoughtCambridge, MAMIT Press
Herrnstein, R. J 1961 Relative and absolute strength of response as a function of frequency of reinforcementJ. Exp. Anal. Behav 4 267
Herrnstein, R. J 1970 On the law of effectJ. Exp. Anal. Behav 13 243
Herrnstein, R. J 1997 The Matching Law: Papers in Psychology and EconomicsCambridge, MAHarvard University Press
Herrnstein, R. JVaughan, W 1980 Melioration and behavioral allocationLimits to Action: The Allocation of Individual BehaviorStaddon, J. ENew YorkAcademic Press143
Hinson, J. MStaddon, J. E 1983 Hill-climbing by pigeonsJ. Exp. Anal. Behav 39 25
Houston, A 1986 The matching law applies to wagtails’ foraging in the wildJ. Exp. Anal. Behav 45 15
Houston, AMcNamara, J 1984 Imperfectly optimal animalsBehav. Ecol. Sociobiol 15 61
Houston, AMcNamara, J 1988 A framework for the functional analysis of behaviourBehav. Brain Sci 11 117
Houston, AMcNamara, J 1999 Models of Adaptive BehaviorCambridgeCambridge University Press
Houston, ASumida, B. H 1987 Learning rules, matching and frequency dependenceJ. Theor. Biol 126 289
Huston, MDeAngelis, D. LPost, W 1988 New computer models unify ecological theoryBioScience 38 682
Iwasa, YHigashi, MYamamura, N 1981 Prey distribution as a factor determining the choice of optimal strategyAmer. Nat 117 710
Judson, O 1994 The rise of the individual-based model in ecologyTrends Ecol. Evol 9 9
Kable, J. WGlimcher, P. W 2009 The neurobiology of decision: consensus and controversyNeuron 63 733
Kahneman, DTversky, A 2000 Choices, Values, and FramesCambridgeCambridge University Press
Koehler, D. JJames, G 2009 Probability matching in choice under uncertainty: intuition versus deliberationCognition 113 123
Krebs, JKacelnik, A 1991 Decision makingBehavioural Ecology: An Evolutionary ApproachKrebs, JDavies, NOxfordBlackwell Scientific Publishers105
Loewenstein, YPrelec, DSeung, H. S 2009 Operant matching as a Nash equilibrium of an intertemporal gameNeural Comput 21 2755
Loewenstein, YSeung, H. S 2006 Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activityProc. Natl. Acad. Sci. USA 103 15224
Lorenz, K 1937 The nature of instinct: the conception of instinctive behaviorInstinctive Behavior: The Development of a Modern ConceptSchiller, CLashley, KNew YorkInternational University Press129
Maes, P 1990 A bottom-up mechanism for behavior selection in an artificial creatureFrom Animals to AnimatsArcady Meyer, JWilson, S. WCambridge, MAMIT Press169
McNamara, JHouston, A 1980 The application of statistical decision theory to animal behaviourJ. Theor. Biol 85 673
McNamara, J. MHouston, A. I 2009 Integrating function and mechanismTrends Ecol. Evol 24 670
Mitchell, M 1997 An Introduction to Genetic AlgorithmsCambridge, MAMIT Press
Myers, J. L 1976 Probability learning and sequence learningHandbook of Learning and Cognitive Processes: Approaches to Human Learning and MotivationEstes, W. KHillsdale, NJErlbaum171
Niv, YJoel, DMeilijson, IRuppin, E 2001 Evolution of reinforcement learning in uncertain environments: a simple explanation for complex foraging behaviorAdapt. Behav 10 5
Pascual, M 2005 Computational ecology: from the complex to the simple and backPLoS Comput Biol 1 101
Pfeifer, R 1996 Building ‘fungus eaters’: design principles of autonomous agentsFrom Animals to Animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive BehaviorMaes, PMataric, MMeyer, J. APollack, JWilson, WCambridge, MAMIT Press3
Prescott, T. JRedgrave, PGurney, K 1999 Layered control architectures in robots and vertebratesAdapt. Behav 7 99
Redgrave, PPrescott, T. JGurney, K 1999 The basal ganglia: a vertebrate solution to the selection problemNeuroscience 89 1009
Rosenblatt, KPayton, D 1989 A fine-grained alternative to the subsumption architecture for mobile robot controlProceedings of the IEEE/INNS International Joint Conference on Neural NetworksWashingtonIEEE Press317
Sakai, YFukai, T 2008 The actor–critic learning is behind the matching law: matching versus optimal behaviorsNeural Comput 20 227
Seth, A. K 1998 Evolving action selection and selective attention without actions, attention, or selectionProceedings of the Fifth International Conference on the Simulation of Adaptive BehaviorPfeifer, RBlumberg, BMeyer, J. AWilson, SCambridge, MAMIT Press139
Seth, A. K 1999 Evolving behavioral choice: an investigation of Herrnstein's matching lawProceedings of the Fifth European Conference on Artificial LifeFloreano, DNicoud, J. DMondada, FBerlinSpringer-Verlag225
Seth, A. K 2000
Seth, A. K 2000 Unorthodox optimal foraging theoryFrom Animals to Animats 6: Proceedings of the Sixth International Conference on the Simulation of Adaptive BehaviorMeyer, J. ABerthoz, AFloreano, DRoitblat, HWilson, SCambridge, MAMIT Press478
Seth, A. K 2001 Modeling group foraging: individual suboptimality, interference, and a kind of matchingAdapt. Behav 9 67
Seth, A. K 2001 Spatially explicit models of forager interferenceProceedings of the Sixth European Conference on Artificial LifeKelemen, JSosik, PBerlinSpringer-Verlag151
Seth, A. K 2002 Agent-based modelling and the environmental complexity thesisFrom Animals to Animats 7: Proceedings of the Seventh International Conference on the Simulation of Adaptive BehaviorHallam, BFloreano, DHallam, JHeyes, GMeyer, J. ACambridge, MAMIT Press13
Seth, A. K 2002 Competitive foraging, decision making, and the ecological rationality of the matching lawFrom Animals to Animats 7: Proceedings of the Seventh International Conference on the Simulation of Adaptive BehaviorHallam, BFloreano, DHallam, JHeyes, GMeyer, J. ACambridge, MAMIT Press359
Seth, A. K 2007 The ecology of action selection: insights from artificial lifePhil. Trans. R. Soc. Lond. B Biol. Sci 362 1545
Shanks, D. RTunney, R. JMcCarthy, J. D 2002 A re-examination of probability matching and rational choiceJ. Behav. Decis. Making 15 233
Shimp, C. P 1966 Probabalistically reinforced choice behavior in pigeonsJ. Exp. Anal. Behav 9 443
Silberberg, AThomas, J. RBerendzen, N 1991 Human choice on concurrent variable-interval variable-ratio schedulesJ. Exp. Anal. Behav 56 575
Stephens, DKrebs, J 1986 Foraging TheoryPrinceton, NJPrinceton University Press
Sutherland, W 1983 Aggregation and the ‘ideal free’ distributionJ. Anim. Ecol 52 821
Sutton, RBarto, A 1998 Reinforcement LearningCambridge, MAMIT Press
Thorndike, E. L 1911 Animal IntelligenceNew YorkMacmillan
Thuisjman, FPeleg, BAmitai, MShmida, A 1995 Automata, matching, and foraging behavior of beesJ. Theor. Biol 175 305
Tinbergen, N 1950 The hierarchical organisation of nervous mechanisms underlying instinctive behaviorSym. Soc. Exp. Biol 4 305
Tinbergen, N 1963 On the aims and methods of ethologyZeitschr. Tierpsychol 20 410
Todd, P. MGigerenzer, G 2000 Precis of simple heuristics that make us smartBehav. Brain Sci 23 727
Tyrrell, T 1993 The use of hierarchies for action selectionAdapt. Behav 1 387
Vulkan, N 2000 An economist's perspective on probability matchingJ. Econ Surv 14 101
Wagner, G. PAltenberg, L. A 1996 Complex adaptations and the evolution of evolvabilityEvolution 50 967
Weber, T 1998 News from the realm of the ideal free distributionTrends Ecol. Evol 13 89
Werner, G 1994 Using second-order neural connections for motivation of behavioral choiceFrom Animals to Animats 3: Proceedings of the Third International Conference on the Simulation of Adaptive BehaviorCliff, DHusbands, PMeyer, J. AWilson, SCambridge, MAMIT Press154
West, RStanovich, K 2003 Is probability matching smart? Associations between probabilistic choices and cognitive abilityMem. Cognition 31 243
Wheeler, Mde Bourcier, P 1995 How not to murder your neighbor: using synthetic behavioral ecology to study aggressive signallingAdapt. Behav 3 235
Yu, A. JDayan, P 2005 Uncertainty, neuromodulation, and attentionNeuron 46 681