{"title": "Intrinsically Motivated Reinforcement Learning", "book": "Advances in Neural Information Processing Systems", "page_first": 1281, "page_last": 1288, "abstract": null, "full_text": "Intrinsically Motivated Reinforcement Learning\n\n\n\n         Satinder Singh                 Andrew G. Barto              Nuttapong Chentanez\n     Computer Science & Eng.       Dept. of Computer Science        Computer Science & Eng.\n      University of Michigan       University of Massachusetts       University of Michigan\n        baveja@umich.edu               barto@cs.umass.edu             nchentan@umich.edu\n\n\n\n                                            Abstract\n\n            Psychologists call behavior intrinsically motivated when it is engaged in\n            for its own sake rather than as a step toward solving a specific problem\n            of clear practical value. But what we learn during intrinsically motivated\n            behavior is essential for our development as competent autonomous en-\n            tities able to efficiently solve a wide range of practical problems as they\n            arise. In this paper we present initial results from a computational study\n            of intrinsically motivated reinforcement learning aimed at allowing arti-\n            ficial agents to construct and extend hierarchies of reusable skills that are\n            needed for competent autonomy.\n\n\n1      Introduction\n\nPsychologists distinguish between extrinsic motivation, which means being moved to do\nsomething because of some specific rewarding outcome, and intrinsic motivation, which\nrefers to being moved to do something because it is inherently enjoyable. Intrinsic motiva-\ntion leads organisms to engage in exploration, play, and other behavior driven by curiosity\nin the absence of explicit reward. These activities favor the development of broad com-\npetence rather than being directed to more externally-directed goals (e.g., ref. [14]). In\ncontrast, machine learning algorithms are typically applied to single problems and so do\nnot cope flexibly with new problems as they arise over extended periods of time.\n\nAlthough the acquisition of competence may not be driven by specific problems, this com-\npetence is routinely enlisted to solve many different specific problems over the agent's\nlifetime. The skills making up general competence act as the \"building blocks\" out of\nwhich an agent can form solutions to new problems as they arise. Instead of facing each\nnew challenge by trying to create a solution out of low-level primitives, it can focus on\ncombining and adjusting its higher-level skills. In animals, this greatly increases the effi-\nciency of learning to solve new problems, and our main objective is to achieve a similar\nefficiency in our machine learning algorithms and architectures.\n\nThis paper presents an elaboration of the reinforcement learning (RL) framework [11] that\nencompasses the autonomous development of skill hierarchies through intrinsically mo-\ntivated reinforcement learning. We illustrate its ability to allow an agent to learn broad\ncompetence in a simple \"playroom\" environment. In a related paper [1], we provide more\nextensive background for this approach, whereas here the focus is more on algorithmic\ndetails.\n\nLack of space prevents us from providing a comprehensive background to the many ideas\nto which our approach is connected. Many researchers have argued for this kind of devel-\n\n\f\nopmental approach in which an agent undergoes an extended developmental period dur-\ning which collections of reusable skills are autonomously learned that will be useful for\na wide range of later challenges (e.g., [4, 13]). The previous machine learning research\nmost closely related is that of Schmidhuber (e.g., [8]) on confidence-based curiosity and\nthe ideas of exploration and shaping bonuses [6, 10], although our definition of intrinsic re-\nward differs from these. The most direct inspiration behind the experiment reported in this\npaper, comes from neuroscience. The neuromodulator dopamine has long been associated\nwith reward learning [9]. Recent studies [2, 3] have focused on the idea that dopamine not\nonly plays a critical role in the extrinsic motivational control of behaviors aimed at harvest-\ning explicit rewards, but also in the intrinsic motivational control of behaviors associated\nwith novelty and exploration. For instance, salient, novel sensory stimuli inspire the same\nsort of phasic activity of dopamine cells as unpredicted rewards. However, this activation\nextinguishes more or less quickly as the stimuli become familiar. This may underlie the fact\nthat novelty itself has rewarding characteristics [7]. These connections are key components\nof our approach to intrinsically motivated RL.\n\n2         Reinforcement Learning of Skills\n\nAccording to the \"standard\" view of RL (e.g., [11]) the agent-environment interaction is\nenvisioned as the interaction between a controller (the agent) and the controlled system (the\nenvironment), with a specialized reward signal coming from a \"critic\" in the environment\nthat evaluates (usually with a scalar reward value) the agent's behavior (Fig. 1A). The agent\nlearns to improve its skill in controlling the environment in the sense of learning how to\nincrease the total amount of reward it receives over time from the critic.\n\n\n\n                                                                         External Environment\n                        Environment                          Actions                              Sensations\n\n\n                            Critic                                       Internal Environment\n\n\n                                                                                Critic\n                         Rewards\n          Actions                           States                            Rewards\n\n                                                            Decisions                               States\n\n\n\n\n                                                                                Agent\n                            Agent\n\n                                                                                                 \"Organism\"\n     A                                                 B\n\n     Figure 1: Agent-Environment Interaction in RL. A: The usual view. B: An elaboration.\n\nSutton and Barto [11] point out that one should not identify this RL agent with an entire\nanimal or robot. An an animal's reward signals are determined by processes within its brain\nthat monitor not only external state but also the animal's internal state. The critic is in an\nanimal's head. Fig. 1B makes this more explicit by \"factoring\" the environment of Fig. 1A\ninto an external environment and an internal environment, the latter of which contains the\ncritic which determines primary reward. This scheme still includes cases in which reward\nis essentially an external stimulus (e.g., a pat on the head or a word of praise). These are\nsimply stimuli transduced by the internal environment so as to generate the appropriate\nlevel of primary reward.\n\nThe usual practice in applying RL algorithms is to formulate the problem one wants the\nagent to learn how to solve (e.g., win at backgammon) and define a reward function spe-\ncially tailored for this problem (e.g., reward = 1 on a win, reward = 0 on a loss). Sometimes\nconsiderable ingenuity is required to craft an appropriate reward function. The point of\ndeparture for our approach is to note that the internal environment contains, among other\nthings, the organism's motivational system, which needs to be a sophisticated system that\n\n\f\nshould not have to be redesigned for different problems. Handcrafting a different special-\npurpose motivational system (as in the usual RL practice) should be largely unnecessary.\n\nSkills--Autonomous mental development should result in a collection of reusable skills.\nBut what do we mean by a skill? Our approach to skills builds on the theory of options\n[12]. Briefly, an option is something like a subroutine. It consists of 1) an option policy that\ndirects the agent's behavior for a subset of the environment states, 2) an initiation set con-\nsisting of all the states in which the option can be initiated, and 3) a termination condition,\nwhich specifies the conditions under which the option terminates. It is important to note\nthat an option is not a sequence of actions; it is a closed-loop control rule, meaning that\nit is responsive to on-going state changes. Furthermore, because options can invoke other\noptions as actions, hierarchical skills and algorithms for learning them naturally emerge\nfrom the conception of skills as options. Theoretically, when options are added to the set\nof admissible agent actions, the usual Markov decision process (MDP) formulation of RL\nextends to semi-Markov decision processes (SMDPs), with the one-step actions now be-\ncoming the \"primitive actions.\" All of the theory and algorithms applicable to SMDPs can\nbe appropriated for decision making and learning with options [12].\n\nTwo components of the the options framework are especially important for our approach:\n1. Option Models: An option model is a probabilistic description of the effects of executing\nan option. As a function of an environment state where the option is initiated, it gives the\nprobability with which the option will terminate at any other state, and it gives the total\namount of reward expected over the option's execution. Option models can be learned\nfrom experience (usually only approximately) using standard methods. Option models\nallow stochastic planning methods to be extended to handle planning at higher levels of\nabstraction.\n2. Intra-option Learning Methods: These methods allow the policies of many options to\nbe updated simultaneously during an agent's interaction with the environment. If an option\ncould have produced a primitive action in a given state, its policy can be updated on the\nbasis of the observed consequences even though it was not directing the agent's behavior\nat the time.\n\nIn most of the work with options, the set of options must be provided by the system designer.\nWhile an option's policy can be improved through learning, each option has to be prede-\nfined by providing its initiation set, termination condition, and the reward function that\nevaluates its performance. Many researchers have recognized the desirability of automati-\ncally creating options, and several approaches have recently been proposed (e.g., [5]). For\nthe most part, these methods extract options from the learning system's attempts to solve a\nparticular problem, whereas our approach creates options outside of the context of solving\nany particular problem.\n\nDeveloping Hierarchical Collections of Skills--Children accumulate skills while they\nengage in intrinsically motivated behavior, e.g., while at play. When they notice that some-\nthing they can do reliably results in an interesting consequence, they remember this in a\nform that will allow them to bring this consequence about if they wish to do so at a future\ntime when they think it might contribute to a specific goal. Moreover, they improve the\nefficiency with which they bring about this interesting consequence with repetition, before\nthey become bored and move on to something else. We claim that the concepts of an option\nand an option model are exactly appropriate to model this type of behavior. Indeed, one of\nour main contributions is a (preliminary) demonstration of this claim.\n\n\n3    Intrinsically Motivated RL\n\nOur main departure from the usual application of RL is that our agent maintains a knowl-\nedge base of skills that it learns using intrinsic rewards. In most other regards, our ex-\ntended RL framework is based on putting together learning and planning algorithms for\n\n\f\n  Loop forever\n      Current state st, current primitive action at, current option ot,\n      extrinsic reward ret, intrinsic reward rit\n\n      Obtain next state st+1\n\n      //-- Deal with special case if next state is salient\n      If st+1 is a salient event e\n          If option for e, oe, does not exist in O (skill-KB)\n                Create option oe in skill-KB;\n                Add st to Ioe // initialize initiation set\n                Set oe (st+1) = 1 // set termination probability\n          //-- set intrinsic reward value\n          ri         =  [1 - P oe (s\n              t+1                                     t+1|st)] //  is a constant multiplier\n      else\n          ri         = 0\n              t+1\n\n\n      //-- Update all option models\n      For each option o = oe in skill-KB (O)\n          If st+1  Io, then add st to Io // grow initiation set\n          If at is greedy action for o in state st\n                //-- update option transition probability model\n                                       \n                P o(x|st)  [(1 - o(st+1)P o(x|st+1) + o(st+1)st+1x]\n                //-- update option reward model\n                             \n                Ro(st)  [re + (1 - o(s\n                                            t                          t+1))Ro(st+1)]\n\n\n      //-- Q-learning update of behavior action-value function\n                        \n      QB(st, at)  [re + ri +  max\n                                  t              t               aAO QB (st+1, a)]\n\n\n      //-- SMDP-planning update of behavior action-value function\n      For each option o in skill-KB\n                             \n          QB(st, o)  [Ro(st) +                                        P o(x|s\n                                                                xS                t) maxaAO QB (x, a)]\n\n\n      //-- Update option action-value functions\n      For each option o  O such that st  Io\n                             \n          Qo(st, at)  [re +  (o(s\n                                            t                   t+1)  terminal value for option o)\n                                                                +(1 - o(st+1))  maxaAO Qo(st+1, a)]\n          For each option o  O such that st  Io and o = o\n                                       \n                Qo(st, o )  Ro (st) +                                     P o (x|s\n                                                                   xS                t)[o(x)  terminal val for option o\n                                                                           +((1 - o(x))  maxaAO Qo(x, a))]\n\n      Choose at+1 using -greedy policy w.r.to QB // -- Choose next action\n      //-- Determine next extrinsic reward\n      Set ret+1 to the extrinsic reward for transition st, at  st+1\n\n      Set st  st+1; at  at+1; re  re                                             ri\n                                                           t           t+1; rit        t+1\n\n\n\nFigure 2: Learning Algorithm. Extrinsic reward is denoted re while intrinsic reward is denoted ri.\n                                  \nEquations of the form x  [y] are short for x  (1-)x+[y]. The behavior action value function\nQB is updated using a combination of Q-learning and SMDP planning. Throughout  is a discount\nfactor and  is the step-size. The option action value functions Qo are updated using intra-option\nQ-learning. Note that the intrinsic reward is only used in updating QB and not any of the Qo.\n\n\f\noptions [12].\nBehavior The agent behaves in its environment according to an -greedy policy with re-\nspect to an action-value function QB that is learned using a mix of Q-learning and SMDP\nplanning as described in Fig. 2. Initially only the primitive actions are available to the agent.\nOver time, skills represented internally as options and their models also become available\nto the agent as action choices. Thus, QB maps states s and actions a (both primitive and\noptions) to the expected long-term utility of taking that action a in state s.\nSalient Events In our current implementation we assume that the agent has intrinsic or\nhardwired notions of interesting or \"salient\" events in its environment. For example, in\nthe playroom environment we present shortly, the agent finds changes in light and sound\nintensity to be salient. These are intended to be independent of any specific task and likely\nto be applicable to many environments.\nReward In addition to the usual extrinsic rewards there are occasional intrinsic rewards\ngenerated by the agent's critic (see Fig. 1B). In this implementation, the agent's intrinsic\nreward is generated in a way suggested by the novelty response of dopamine neurons. The\nintrinsic reward for each salient event is proportional to the error in the prediction of the\nsalient event according to the learned option model for that event (see Fig. 2 for detail).\nSkill-KB The agent maintains a knowledge base of skills that it has learned in its environ-\nment. Initially this may be empty. The first time a salient event occurs, say light turned\non, structures to learn an option that achieves that salient event (turn-light-on option) are\ncreated in the skill-KB. In addition, structures to learn an option model are also created.\nSo for option o, Qo maps states s and actions a (again, both primitive and options) to the\nlong-term utility of taking action a in state s. The option for a salient event terminates with\nprobability one in any state that achieves that event and never terminates in any other state.\nThe initiation set, Io, for an option o is incrementally expanded to includes states that lead\nto states in the current initiation set.\nLearning The details of the learning algorithm are presented in Fig. 2.\n\n\n4    Playroom Domain: Empirical Results\n\nWe implemented intrinsically motivated RL (of Fig. 2) in a simple artificial \"playroom\"\ndomain shown in Fig. 3A. In the playroom are a number of objects: a light switch, a ball,\na bell, two movable blocks that are also buttons for turning music on and off, as well as a\ntoy monkey that can make sounds. The agent has an eye, a hand, and a visual marker (seen\nas a cross hair in the figure). The agent's sensors tell it what objects (if any) are under the\neye, hand and marker. At any time step, the agent has the following actions available to\nit: 1) move eye to hand, 2) move eye to marker, 3) move eye one step north, south, east or\nwest, 4) move eye to random object, 5) move hand to eye, and 6) move marker to eye. In\naddition, if both the eye and and hand are on some object, then natural operations suggested\nby the object become available, e.g., if both the hand and the eye are on the light switch,\nthen the action of flicking the light switch becomes available, and if both the hand and eye\nare on the ball, then the action of kicking the ball becomes available (which when pushed,\nmoves in a straight line to the marker).\n\nThe objects in the playroom all have potentially interesting characteristics. The bell rings\nonce and moves to a random adjacent square if the ball is kicked into it. The light switch\ncontrols the lighting in the room. The colors of any of the blocks in the room are only\nvisible if the light is on, otherwise they appear similarly gray. The blue block if pressed\nturns music on, while the red block if pressed turns music off. Either block can be pushed\nand as a result moves to a random adjacent square. The toy monkey makes frightened\nsounds if simultaneously the room is dark and the music is on and the bell is rung. These\nobjects were designed to have varying degrees of difficulty to engage. For example, to get\nthe monkey to cry out requires the agent to do the following sequence of actions: 1) get its\neye to the light switch, 2) move hand to eye, 3) push the light switch to turn the light on, 4)\nfind the blue block with its eye, 5) move the hand to the eye, 6) press the blue block to turn\n\n\f\nmusic on, 7) find the light switch with its eye, 8) move hand to eye, 9) press light switch\nto turn light off, 10) find the bell with its eye, 11) move the marker to the eye, 12) find the\nball with its eye, 13) move its hand to the ball, and 14) kick the ball to make the bell ring.\nNotice that if the agent has already learned how to turn the light on and off, how to turn\nmusic on, and how to make the bell ring, then those learned skills would be of obvious use\nin simplifying this process of engaging the toy monkey.\n\nA                      B                                                                                           C\n                                                                        Performance of Learned Options                                                                   Effect of Intrinsically Motivated Learning\n                                                                                                                                                                10000\n                                                                     120\n\n                                                                     100                                                                                         8000\n\n                                                                     80      Sound On                                                                                                Extrinsic Reward Only \n                                                                                                                                                                 6000\n\n                                                                     60      Light On \n                                                                                    Music On                                                                     4000\n                                                                     40                   Toy Monkey On                                                                              Intrinsic & Extrinsic Rewards \n\n                                                                     20                                                                                          2000\n\n                                                                      0  \n                            Average # of Actions to Salient Event       0    0.5     1        1.5    2      2.5                                                     0\n                                                                                                                   Number of steps between extrinsic rewards        0         100      200    300    400    500    600\n                                                                                                            7\n                                                                              Number of Actions x 10                                                                           Number of extrinsic rewards\n\n\nFigure 3: A. Playroom domain. B. Speed of learning of various skills. C. The effect of intrinsically\nmotivated learning when extrinsic reward is present. See text for details\n\nFor this simple example, changes in light and sound intensity are considered salient by the\nplayroom agent. Because the initial action value function, QB, is uninformative, the agent\nstarts by exploring its environment randomly. Each first encounter with a salient event\ninitiates the learning of an option and an option model for that salient event. For example,\nthe first time the agent happens to turn the light on, it initiates the data structures necessary\nfor learning and storing the light-on option. As the agent moves around the environment, all\nthe options (initiated so far) and their models are simultaneously updated using intra-option\nlearning.\n\nAs shown in Fig. 2, the intrinsic reward is used to update QB. As a result, when the agent\nencounters an unpredicted salient event a few times, its updated action value function drives\nit to repeatedly attempt to achieve that salient event. There are two interesting side effects\nof this: 1) as the agent tries to repeatedly achieve the salient event, learning improves both\nits policy for doing so and its option-model that predicts the salient event, and 2) as its\noption policy and option model improve, the intrinsic reward diminishes and the agent gets\n\"bored\" with the associated salient event and moves on. Of course, the option policy and\nmodel become accurate in states the agent encounters frequently. Occasionally, the agent\nencounters the salient event in a state (set of sensor readings) that it has not encountered\nbefore, and it generates intrinsic reward again (it is \"surprised\").\n\nA summary of results is presented in Fig. 4. Each panel of the figure is for a distinct salient\nevent. The graph in each panel shows both the time steps at which the event occurs as\nwell as the intrinsic reward associated by the agent to each occurrence. Each occurrence is\ndenoted by a vertical bar whose height denotes the amount of associated intrinsic reward.\nNote that as one goes from top to bottom in this figure, the salient events become harder to\nachieve and, in fact, become more hierarchical. Indeed, the lowest one for turning on the\nmonkey noise (Non) needs light on, music on, light off, sound on in sequence. A number\nof interesting results can be observed in this figure. First note that the salient events that\nare simpler to achieve occur earlier in time. For example, Lon (light turning on) and Loff\n(light turning off) are the simplest salient events, and the agent makes these happen quite\nearly. The agent tries them a large number of times before getting bored and moving on to\nother salient events. The reward obtained for each of these events diminishes after repeated\nexposure to the event. Thus, automatically, the skill of achieving the simpler events are\nlearned before those for the more complex events.\n\n\f\nFigure 4: Results from the playroom domain. Each panel depicts the occurrences of salient events\nas well as the associated intrinsic rewards. See text for details.\n\nOf course, the events keep happening despite their diminished capacity to reward because\nthey are needed to achieve the more complex events. Consequently, the agent continues to\nturn the light on and off even after it has learned this skill because this is a step along the\nway toward turning on the music, as well as along the way toward turning on the monkey\nnoise. Finally note that the more complex skills are learned relatively quickly once the\nrequired sub-skills are in place, as one can see by the few rewards the agent receives for\nthem. The agent is able to bootstrap and build upon the options it has already learned for the\nsimpler events. We confirmed the hierarchical nature of the learned options by inspecting\nthe greedy policies for the more complex options like Non and Noff. The fact that all the\noptions are successfully learned is also seen in Fig. 3B in which we show how long it takes\nto bring about the events at different points in the agent's experience (there is an upper\ncutoff of 120 steps). This figure also shows that the simpler skills are learned earlier than\nthe more complex ones.\n\nAn agent having a collection of skills learned through intrinsic reward can learn a wide\nvariety of extrinsically rewarded tasks more easily than an agent lacking these skills. To\nillustrate, we looked at a playroom task in which extrinsic reward was available only if\nthe agent succeeded in making the monkey cry out. This requires the 14 steps described\nabove. This is difficult for an agent to learn if only the extrinsic reward is available, but\nmuch easier if the agent can use intrinsic reward to learn a collection of skills, some of\nwhich are relevant to the overall task. Fig. 3C compares the performance of two agents in\nthis task. Each starts out with no knowledge of task, but one employs the intrinsic reward\nmechanism we have discussed above. The extrinsic reward is always available, but only\nwhen the monkey cries out. The figure, which shows the average of 100 repetitions of the\nexperiment, clearly shows the advantage of learning with intrinsic reward.\n\n\nDiscussion      One of the key aspects of the Playroom example was that intrinsic reward\nwas generated only by unexpected salient events. But this is only one of the simplest\n\n\f\npossibilities and has many limitations. It cannot account for what makes many forms of\nexploration and manipulation \"interesting.\" In the future, we intend to implement compu-\ntational analogs of other forms of intrinsic motivation as suggested in the psychological,\nstatistical, and neuroscience literatures.\n\nDespite the \"toy\" nature of this domain, these results are among the most sophisticated\nwe have seen involving intrinsically motivated learning. Moreover, they were achieved\nquite directly by combining a collection of existing RL algorithms for learning options and\noption-models with a simple notion of intrinsic reward. The idea of intrinsic motivation for\nartificial agents is certainly not new, but we hope to have shown that the elaboration of the\nformal RL framework in the direction we have pursued, together with the use of recently-\ndeveloped hierarchical RL algorithms, provides a fruitful basis for developing competently\nautonomous agents.\n\nAcknowledgement Satinder Singh and Nuttapong Chentanez were funded by NSF grant CCF\n0432027 and by a grant from DARPA's IPTO program. Andrew Barto was funded by NSF grant\nCCF 0432143 and by a grant from DARPA's IPTO program.\n\n\nReferences\n\n [1] A. G. Barto, S. Singh, and N. Chentanez. Intrinsically motivated learning of hierarchical collec-\n     tions of skills. In Proceedings of the 3rd International Conference on Developmental Learning\n     (ICDL '04), LaJolla CA, 2004.\n\n [2] P. Dayan and B. W. Balleine. Reward, motivation and reinforcement learning. Neuron, 36:285\n     298, 2002.\n\n [3] S. Kakade and P. Dayan. Dopamine: Generalization and bonuses. Neural Networks, 15:549\n     559, 2002.\n\n [4] F. Kaplan and P.-Y. Oudeyer. Motivational principles for visual know-how development. In\n     C. G. Prince, L. Berthouze, H. Kozima, D. Bullock, G. Stojanov, and C. Balkenius, editors,\n     Proceedings of the Third International Workshop on Epigenetic Robotics : Modeling Cognitive\n     Development in Robotic Systems, pages 7380, Edinburgh, Scotland, 2003. Lund University\n     Cognitive Studies.\n\n [5] A. McGovern. Autonomous Discovery of Temporal Abstractions from Interaction with An En-\n     vironment. PhD thesis, University of Massachusetts, 2002.\n\n [6] A. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and\n     application to reward shaping. In Proceedings of the Sixteenth ICML. Morgan Kaufmann, 1999.\n\n [7] P. Reed, C. Mitchell, and T. Nokes. Intrinsic reinforcing properties of putatively neutral stimuli\n     in an instrumental two-lever discrimination task. Animal Learning and Behavior, 24:3845,\n     1996.\n\n [8] J. Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural\n     controllers. In From Animals to Animats: Proceedings of the First International Conference on\n     Simulation of Adaptive Behavior, pages 222227, Cambridge, MA, 1991. MIT Press.\n\n [9] W. Schultz. Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80:1\n     27, 1998.\n\n[10] R. S. Sutton. Integrated modeling and control based on reinforcement learning and dynamic\n     programming. In Proceedings of NIPS, pages 471478, San Mateo, CA, 1991.\n\n[11] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge,\n     MA, 1998.\n\n[12] R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for tempo-\n     ral abstraction in reinforcement learning. Artificial Intelligence, 112:181211, 1999.\n\n[13] J. Wang, J. McClelland, A. Pentland, O. Sporns, I. Stockman, M. Sur, and E. Thelen. Au-\n     tonomous mental develoopment by robots and animals. Science, 291:599600, 2001.\n\n[14] R. W. White. Motivation reconsidered: The concept of competence. Psychological Review,\n     66:297333, 1959.\n\n\f\n", "award": [], "sourceid": 2552, "authors": [{"given_name": "Nuttapong", "family_name": "Chentanez", "institution": null}, {"given_name": "Andrew", "family_name": "Barto", "institution": null}, {"given_name": "Satinder", "family_name": "Singh", "institution": null}]}