{"title": "Control What You Can: Intrinsically Motivated Task-Planning Agent", "book": "Advances in Neural Information Processing Systems", "page_first": 12541, "page_last": 12552, "abstract": "We present a novel intrinsically motivated agent that learns how to control the\nenvironment in a sample efficient manner, that is with as few environment interactions as possible, by optimizing learning progress. It learns what can be controlled, how to allocate time and attention as well as the relations between objects using surprise-based motivation. The effectiveness of our method is demonstrated in a synthetic and robotic manipulation environment yielding considerably improved performance and smaller sample complexity compared to an intrinsically motivated, non-hierarchical and state-of-the-art hierarchical baseline. In a nutshell, our work combines several task-level planning agent structures (backtracking search on task-graph, probabilistic road-maps, allocation of search efforts) with intrinsic motivation to achieve learning from scratch.", "full_text": "Control What You Can\n\nIntrinsically Motivated Task-Planning Agent\n\nSebastian Blaes\n\nMarin Vlastelica Pogan\u02c7ci\u00b4c\n\nJia-Jie Zhu\n\nGeorg Martius\n\nAutonomous Learning Group\n\nMax Planck Institute for Intelligent Systems\n\n{sebastian.blaes,marin.vlastelica,jzhu,georg.martius}@tue.mpg.de\n\nT\u00fcbingen, Germany\n\nAbstract\n\nWe present a novel intrinsically motivated agent that learns how to control the\nenvironment in a sample ef\ufb01cient manner, that is with as few environment interac-\ntions as possible, by optimizing learning progress. It learns what can be controlled,\nhow to allocate time and attention as well as the relations between objects using\nsurprise-based motivation. The effectiveness of our method is demonstrated in a\nsynthetic and robotic manipulation environment yielding considerably improved\nperformance and smaller sample complexity compared to an intrinsically moti-\nvated, non-hierarchical and state-of-the-art hierarchical baseline. In a nutshell, our\nwork combines several task-level planning agent structures (backtracking search\non task-graph, probabilistic road-maps, allocation of search efforts) with intrinsic\nmotivation to achieve learning from scratch.\n\n1\n\nIntroduction\n\nThis paper studies the question of how to make an autonomous agent learn to gain maximal control\nof its environment under little external reward. To answer this question, we turn to the true learning\nexperts: children. Children are remarkably fast in learning new skills; How do they do this? In\nrecent years psychologists and psychiatrists acquired a much better understanding of the underlying\nmechanisms that facilitate these abilities. Babies seemingly do it by conducting experiments and\nanalyzing the statistics in their observations to form intuitive theories about the world [12]. Thereby,\nhuman learning seems to be supported by hard-wired abilities, e.g.\nidentifying human faces or\nthe well tuned relationship between teacher (adult) and student (infant) [11] and by self-motivated\nplaying, often with any objects within their reach. The purpose may not be immediately clear to us.\nBut to play is to manipulate, to gain control. In the spirit of this cognitive developmental process,\nwe speci\ufb01cally design an agent that is 1) intrinsically motivated to gain control of the environment\n2) capable of learning its own curriculum and to reason about object relations.\nAs a motivational example, consider an environment with a heavy object that cannot be moved without\nusing a tool such as a forklift, as depicted in Fig. 1(6). To move the heavy object, the agent needs\nto learn \ufb01rst how to control itself and then how to use the tool. In the beginning, we do not assume\nthe agent has knowledge of the tool, object, or physics. Everything needs to be learned from scratch\nwhich is highly challenging for current RL algorithms. Without external rewards, an agent may be\ndriven by intrinsic motivation (IM) to gain control of its own internal representation of the world,\nwhich includes itself and objects in the environment. It often faces a decision of what to attempt to\nlearn with limited time and attention: If there are several objects that can be manipulated, which one\nshould be dealt with \ufb01rst? In our approach the scheduling is solved by an automatic curriculum that\naims at improving learning progress. The learning progress may have its unique advantage over other\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fFigure 1: Overview of CWYC method. All components except (1) are learned. Details can be found\nin the main text. Videos and code are available at https://s-bl.github.io/cwyc/\n\nquantities such as prediction error (curiosity): It renders unsolvable tasks uninteresting as soon as\nprogress stalls.\nInstead of an end-to-end architecture, we adopt a core reasoning structure about tasks and sub-goals.\nInspired by task-level planning methods from the robotics and AI planning communities, we model\nthe agent using a planning architecture in the form of chained sub-tasks. In practice, this is modeled\nas a task-graph as in Fig. 1(4). In order to manipulate the state of the tool, the agent and the tool\nneed to be in a speci\ufb01c relation. Our agent learns such relationships by an attention mechanism\nbootstrapped by surprise detection.\nOur main contributions are:\n\n1. We propose to use maximizing controllability and surprise as intrinsic motivation for\nsolving challenging control problems. The computational effectiveness of this cognitive\ndevelopment-inspired approach is empirically demonstrated.\n\n2. We propose to adopt several task-level planning ideas (backtracking search on task-\ngraph/goal regression, probabilistic road-maps, allocation of search efforts) for designing\nIM agents to achieve task completion and skill acquisition from scratch.\n\nTo our knowledge, no prior IM study has adopted similar controllability and task-planning insights.\nThe contributions are validated through 1) a synthetic environment, with exhaustive analysis and\nablation, that cannot be solved by state-of-the-art methods even with oracle rewards; 2) a robotic\nmanipulation environment where tool supported manipulation is necessary.\n\n2 Related Work\n\nIn this section, we give a survey on the recent computational approaches to intrinsic motivation\n(IM). This is by no means comprehensive due to the large body of literature on this topic. Generally\nspeaking, there are a few types of IM in literature: learning progress (competence, empowerment),\ncuriosity (surprise, prediction error), self-play (adversarial generation), auxiliary tasks, maximizing\ninformation theoretical quantities, etc. To help readers clearly understand the relation between our\nwork and the literature, we provide the following table.\n\nCWYC Ours\nh-DQN [21]\nIMGEP [10]\nCURIOUS [8]\nSAC-X [29]\nRelational RL [38]\nICM [26]\nGoal GAN [9]\nAsymmetric self-play [34]\n\nIntrinsic motivation\nlearning progress + surprise\nreaching subgoals\nlearning progress\nlearning progress\nauxiliary task\n-\nprediction error\nadversarial goal\nself-play\n\nComputational methods\ntask-level planning, relational attention\nHRL, DQN\nmemory-based\nDDPG, HER, E-UVFA\nHRL, (DDPG-like) PI\nrelation net, IMPALA\nA3C, ICM\nGAN, TRPO\nAlice/Bob, TRPO, REINFORCE\n\n2\n\ncurrent subtaskpreceding subtask (1) Task-spaces(2) Task selector(3) Subtask plannerStart(6) Rollout in environmentsuccesstrailsprogresspred_errttrails(5) Subgoal generator(4) Dependency graph(7) History and forward modelsHistoryprogresssuccess rateprediction error 12K12K......33312KjS312SSgoalgoal\ufb01nal goal12K...3...1:2:3:K:state(8) Intrinsic Motivationsurpriseone trailIntrinsic MotivationTask Planning and RL\fLearning progress describes the rate of change of an agent gaining competence in certain skills. It is\na heuristic for measuring interests inspired by observing human children. This is the focus of many\nrecent studies [2, 8, 10, 18, 25, 31, 32]. Our work can be thought of as an instantiation of that as\nwell using maximizing controllability and a task-planning structure. Empowerment [20] proposes a\nquantity that measures the control of the agent over its future sensory input. Curiosity, as a form of\nIM, is usually modeled as the prediction error of the agent\u2019s world model. For example, in challenging\nvideo game domains it can lead to remarkable success [26] or to learn options [7]. Self-play as IM,\nwhere two agents engage in an adversarial game, was demonstrated to increase learning speed [34].\nThis is also related to the idea of using GANs for goal generation as in [9].\nRecently, auxiliary prediction tasks were used to aid representation learning[17]. In comparison,\nour goal is not to train the feature representation but to study the developmental process. Similarly,\ninformed auxiliary tasks as a form of IM was considered in [29]. Many RL tasks can be formulated as\naiming to maximize certain information theoretical quantities [14, 23, 36, 37]. In contrast, we focus\non IM inspired by human children. In [21] a list of given sub-goals is scheduled whereas our attention\nmodel/goal generation is learned. Our work is closely related to [38], multi-head self-attention is used\nto learn non-local relations between entities which are then fed as input into an actor-critic network.\nIn this work, learning these relations is separated from learning the policy and done by a low capacity\nnetwork. More details are given in Sec. 3. In addition, we learn an automatic curriculum.\nTask-level planning has been extensively studied in robotics and AI planning communities in form of\ngeometric planning methods (e.g., RRT [22], PRM [19]) and optimization-based planning [28, 33].\nThere is a parallel between the sparse reward problem and optimization-based planning: the lack of\ngradient information if the robot is not in contact with the object of interest. Notably, our use of the\nsurprise signals is reminiscent to the event-triggered control design [3, 16] in the control community\nand was also proposed in cognitive sciences [6].\n\n3 Method\n\nWe call our method Control What You Can (CWYC). The goal of this work is to make an agent\nlearn to control itself and objects in its environment, or more generically, to control the components\nof its internal representation. We assume the observable state-space S \u2208 Rn is partitioned into\ngroups of potentially controllable components (coordinates) referred to as goal spaces, similar to\n[1]. Manipulating these components is formulated as self-imposed tasks. These can be the agent\u2019s\nposition (task: move agent to location (x, y)), object positions (task: move object to position (x, y)),\netc. The component\u2019s semantics and whether it is controllable are unknown to the agent. The\nperception problem, that is constructing goal spaces from high-dimensional image data or other\nsensor modalities, is an orthogonal line of research. Readers interested in representation learning are\nreferred to, e. g. [5, 27]. Formally, the coordinates in the state s, corresponding to each goal-reaching\ntask \u03c4 \u2208 K = {1, 2, . . . , K}, are indexed by m\u03c4 \u2282 {1, 2, . . . , n} and denoted by sm\u03c4 . The goal in\neach task is denoted as g\u03c4 \u2208 R|m\u03c4|. For instance, if task 1 has its goal-space along the coordinates 3\nand 4 (e.g. agent\u2019s location) then m1 = (3, 4) and sm1 = (s3, s4) are corresponding state values. It\nis assumed that goal spaces are non-overlapping and encompass only simultaneously controllable\ncomponents.\nDuring the learning/development phase, the agent can decide which task (e. g. itself or object) it\nattempts to control. Intuitively, it should be bene\ufb01cial for the learning algorithm to concentrate on\ntasks where the agent can make progress in and inferring potential task dependencies. In order to\nexpress the capability of controlling a certain object, we consider goal-reaching tasks with randomly\nselected goals. When the agent can reach any goals allowed by the environment, it has achieved\ncontrol of the component (e. g. moving a box to any desired location). In many challenging scenarios,\ne. g. object manipulation, there are \u201cfunnel states\u201d that must be discovered. For instance, in a tool-use\ntask the funnel states are where the agent picks up the tool and where the tool touches another\nobject that needs to be manipulated. Our architecture combines relational learning embedded in an\noverall intrinsically motivated learning framework based on a learned probabilistic graph that chains\nlow-level goal-directed RL controllers.\nOur approach contains several components as illustrated in Fig. 1. Their detailed interplay is as\nfollows: The tasks (1) control groups of components (coordinates) of the state. A task selector\n(bandit)(2) is used to select a self-imposed task \u03c4 (\ufb01nal task) maximizing expected learning progress.\n\n3\n\n\f(a) tool-use/object manipulation\n\n(b) robotic object manipulation\n\nFigure 2: Environments used to test CWYC. (a) basic tool-use/object manipulation environment;\n(b) robotic object manipulation environment. The hook needs to be used to move the box.\n\nGiven a \ufb01nal task, the task planner (3) computes a viable sub-task sequence (bold) from a learned\ntask graph (4). The sub-goal generators (5) (relational attention networks) create for every time step\na goal gi(t) in the current sub-task i. The goal-conditioned low-level policies for each task control\nthe agent a(t) \u223c \u03c0i(s(t), gi(t)) in the environment (6). Let us comprise (3,4,5, \u03c0i\u2200i) into the acting\npolicy a(t) \u223c \u03a0(s(t), \u03c4, g\u03c4 ) (internally using the current sub-task policy and goal etc). After one\nrollout different quantities, measuring the training progress, are computed and stored in the per task\nhistory buffer (7). An intrinsic motivation module (8) computes the rewards and target signals for (2),\n(3), and (5) based on learning progress and prediction errors. All components are trained concurrently\nand without external supervision. Prior knowledge enters only in the form of specifying the goal\nspaces (groups of coordinates of the state space). The environment allows the agent to select which\ntask to do next and generates a random arrangement with a random goal.\n\n3.1\n\nIntrinsic Motivation\n\n\u2206 rollout is the time derivative of the success rate, quantifying whether the\n\nIn general, our agent is motivated to learn as fast as possible, i. e. maximizing instantaneous learning\nprogress, and to be as successful as possible, i.e. maximizing success rate, in each task. When\nperforming a particular task \u03c4, with the goal g\u03c4 the agent computes the reward for the low-level\ncontroller as the negative distance to the goal as ri(t) = \u2212(cid:107)smi(t) \u2212 gi(cid:107)2 and declares success as:\nsucci = maxt(cid:74)(cid:107)smi(t) \u2212 gi(cid:107)2 \u2264 \u03b4i(cid:75) where \u03b4i is a precision threshold and(cid:74)\u00b7(cid:75) is the Iverson bracket.\nThe maximum is taken over all time steps in the current rollout (trial). We choose the euclidean\ndistance between the task relevant sub-state to the goal state as distance metric because it is general\nin that it does not impose a particular structure on the goal spaces and therefore can be easily applied\nto any goal space. We calculate the following key measures to quantify intrinsic motivations:\nSuccess rate (controlability) sri = E\u03b7(succi), where \u03b7 is the state distribution induced by \u03a0(\u00b7, i,\u00b7).\nIn practice sri is estimated as a running mean of the last attempts of task i.\nLearning progress \u03c1i = \u2206 sri\nagent gets better at task i compared to earlier attempts.\nInitially, any success signals might be so sparse that learning becomes slow because of uninformed\nexploration. Hence, we employ surprise as a proxy that guides the agent\u2019s attention to tasks and\nstates that might be interesting.\nPrediction error ei(t) in goal space i of a forward model f : [S \u00d7 A] \u2192 S trained using squared\nloss e(t) = (cid:107)(f (s(t), a(t)) + s(t)) \u2212 s(t + 1)(cid:107)2 and ei = emi denotes the error in the goal space i.\nSurprising events surprisei(t) \u2208 {0, 1} is 1 if the prediction error emi(t) in task i exceeds a\ncon\ufb01dence interval (computed over the history), 0 otherwise, see also [13].\nTo understand why surprising events can be informative, let us consider again our example: Assume\nthe agent just knows how to move itself. It will move around and will not be able to manipulate other\nparts of its state-space, i. e. it can neither move the heavy box nor the tool. Whenever it accidentally\nhits the tool, the tool moves and creates a surprise signal in the coordinates of the tool task. Thus, it\nis likely that this particular situation is a good starting point for solving the tool task and make further\nexplorations.\n\n4\n\nagentlocomotiongoaltoolheavyobj. goalheavy obj.50% obj.random\f3.2 Task-Planning Architecture\nThe task selector T , Fig. 1(2) models the learning progress when attempting to solve a task. It is\nimplemented as a multi-armed bandit. While no learning progress is available, the surprise signal is\nused as a proxy. Thus, the internal reward signal for the bandit for a rollout attempting task i is\n\ni = |\u03c1i| + \u03b2T max\nrT\n\nt\n\n(surprisei(t))\n\n(1)\nwith \u03b2T (cid:28) 1. The multi-armed bandit is used to choose the (\ufb01nal) task for a rollout using a stochastic\npolicy. More details can be found in Sec. A.1. In our setup, the corresponding goal within this task is\ndetermined by the environment (in a random fashion).\nBecause dif\ufb01cult tasks require sub-tasks to be performed in certain orders, a task planner determines\nthe sequence of sub-tasks. The task planner models how quick (sub)-task i can be solved when\nperforming sub-task j directly before it. As before, we use surprising events as a proxy signal for\npotential future success. The values of each task transition is captured by Bi,j, where i \u2208 [1, . . . , K]\nand j \u2208 [S, 1, , . . . , K] with S representing the \u201cstart\u201d:\n\ni,j(cid:80)\n\nQB\nk QB\ni,k\n\nBi,j =\n\nwith QB\n\ni,j =\n\n(cid:28)\n\n1 \u2212 Ti,j\n\nT max + \u03b2B max\n\nt\n\n(surprisei(t))\n\n(2)\n\n(cid:29)\n\nwhere (cid:104)\u00b7(cid:105) denotes a running average and Ti,j is the runtime for solving task i by doing task j before\n(maximum number time steps T max if not successful). Similarly to Eq. 1, this quantity is initially\ndominated by the surprise signals and later by the actual success values.\nThe matrix B represents the adjacency matrix of the task graph, see Fig. 1(4). It is used to construct a\nsequence of sub-tasks by starting from the \ufb01nal task \u03c4 and determining the previous sub-task with an\n\u0001-greedy policy using B\u03c4,\u00b7. Then this is repeated for the next (prerequisite) sub-task, until S (start) is\nsampled (no loops are allowed), see also Fig. 1(3) and (4).\nEach (sub)-task is itself a goal-reaching problem. In order to decide which sub-goals need to be\nchosen we employ an attention network for each task transition, i. e. Gi,j for the transition from task\nj to task i. As before, the aim of the goal proposal network Gi,j is to maximize the success rate of\nsolving task i when using the proposed goal in task j before. In the example, in order to pick up\nthe tool, the goal of the preceding locomotion task should be the location of the tool. An attention\nnetwork that can learn relations between observations is required. We use an architecture that models\nlocal pairwise distance relationships. It associates a value/attention to each point in the goal-space of\nthe preceding task as a function of the state s: Gi,j : S \u2192 R: (omitting index i,j)\n\nGi,j(s) = e\u2212\u03b3(cid:80)n\n\nk=1\n\n(cid:80)n\nl=k+1 (cid:107)w1\n\nklsk+w2\n\nklsl+w3\n\nkl(cid:107)2\n\nwhere w1, w2, w3, and \u03b3 are trainable parameters. The network is trained by regression, minimizing\nthe loss\n\nLG\ni,j(w1\n\nkl, w2\n\nkl, w3\n\nkl, \u03b3) =\n\nmin\nkl,w3\n\nkl,\u03b3\n\nw1\n\nkl,w2\n\n(cid:107)Gi,j(sk) \u2212 rG\n\nwhere k enumerates training samples and with the following target signal rG\ni,j(st) = min(1, succi \u00b7 \u0393i,j(st) + surprisei(t))\nrG\n\ni,j(sk)(cid:107)2\ni,j(st) \u2208 [0, 1]:\n\nn(cid:88)\n\nk=1\n\n(3)\n\n(4)\n\n(5)\n\nk,l = \u2212w2\n\nfor all st that occurred during task j where \u0393i,j(s) is 1 if the agent decides to switch from task j\nto task i in state s and zero otherwise. To get an intuition about the parametrization, consider a\nparticular pair of coordinates (k, l), say agent\u2019s and tool\u2019s x-coordinate. The model can express\nk,l (cid:54)= 0 that both have to be at distance zero for rG to be 1. However, with w3 the\nwith w1\nsystem can also model offsets, global reference points and other relationships. Further details on the\narchitecture and training can be found in Suppl. A.5. We observe that the goal proposal network can\nlearn a relationship after a few examples (in the order of 10), possibly due to the restricted model\nclass. The goal proposal network can be thought of as a relational network [30], albeit is easier to\ntrain. Sampling a goal from the network is done by computing the maximum analytically as detailed\nin Suppl. A.3. The low-level control in each task has its own policy \u03c0i learned by soft actor critic\n(SAC) [15] or DDPG+HER [1].\n\n5\n\n\fCWYC w oracle\n\n(a)\n\nCWYC\n\nHIRO ----- ICM-S\n\n(b)\n\n----- ICM-E\n\nSAC\n\n(c)\n\n(e)\n\n(d)\n\n(f)\n\nreachability analysis of the heavy object\n\nt = 0.5 \u00d7 107\n\nt = 1.4 \u00d7 107\n\nFigure 3: Competence of the agents in controlling all aspects of the synthetic environment. Overall\nperformance (a) (maximal 70%). Individual task competence in (b-e). HIRO and SAC can only learn\nthe locomotion task. All performance plots (as well in remaining \ufb01gures) show median and shaded\n25% / 75% pecentiles averaged over 10 random seeds. In (c-e) the green curve is below the blue\ncurve. (f) shows the gain in reachability: probability of reaching the point with the heavy box from\n20 random starting states. The reachability is initially zero.\n\n4 Experimental Results\n\nThrough experiments in two different environments, we wish to investigate empirically: does the\nCWYC agent learn ef\ufb01ciently to gain control over the environment? What about challenging tasks\nthat require a sequence of sub-tasks and uncontrollable objects? How is the behavior of CWYC\ndifferent from that of other (H)RL agents? To give readers a sense of the computational property of\nCWYC, we use an implementation1 of HIRO [24] as a HRL baseline which is suitable for continuous\ncontrol tasks. However, it solves each task independently as it does not support the multi-task setting.\nAs IM baselines, we implement two versions of ICM [26] such that they work with continuous action\nspaces, using SAC as RL agent. ICM-S uses the surprise signal as additional reward signal while\nICM-E uses the raw prediction error directly. In addition, we show the baselines of using only the\nlow-level controllers (SAC [15] or DDPG+HER [1]) for each individual task independently and spend\nresources on all tasks with equal probability.\nWe also add CWYC with a hand-crafted oracle task planner (B) and oracle sub-goal generator (G)\ndenoted as CWYC w oracle, see Suppl. D.1. The code as well as the environment implementations is\npublicly available. The pseudocode is provided in Suppl. B.\nSynthetic environment. The synthetic object manipulation arena, as shown in Fig. 2(a), consists\nof a point mass agent with two degrees of freedom and several objects surrounded by a wall. It is\nimplemented in the MuJoCo physics simulator [35] and has continuous state and action spaces. To\nmake the tasks dif\ufb01cult, we consider the case with 4 different objects: 1. the tool, that can be picked\nup easily; 2. the heavy object that needs the tool to be moved; 3. an unreliable object denoted as 50%\nobject that does not respond to control during 50% of the rollouts; and 4. a random object that moves\naround randomly and cannot be manipulated by the agent, see Fig. 2(a). The detail of the physics in\nthis environment can be found in Suppl. C.1.\nFigure 3 shows the performance of the CWYC-agent compared to the hierarchical baseline (HIRO),\nIM baselines (ICM-(S/E), non-hierarchical baseline (SAC) and the hand-crafted upper baseline\n(oracle). The main measure is competence, i. e. the overall success-rate ( 1\ni=1 sri) of controlling\nK\nthe internal state, i. e. reaching a random goal in each task-space. In this setting an average maximum\nof 70% success rate can be achieved due to the completely unsolvable \u201crandom object\u201d and the \u201c50%\n\n(cid:80)K\n\n1https://github.com/n0c1urne/hrl.git\n\n6\n\n1001010505101001010505100.00.20.40.60.81.0\fCWYC on HER\n\nDDPG+HER\n\nHIRO ----- ICM-S\n\n----- ICM-E\n\nSAC\n\nFigure 4: Success rates of reaching, tool using and object manipulation in the robotic environment.\nCWYC as well as DDPG+HER learn all three tasks perfectly. CWYC has improved sample complex-\nity. SAC, HIRO and ICM-(S/E) learn the reaching task only slowly and the other two tasks not at\nall.\n\nobject\u201d that can be moved only in half of the rollouts and is unmovable in all other rollouts. The results\nshow that our method is able to quickly gain control over the environment, also illustrated by the\nreachability which grows with time and reaches almost full coverage for the heavy object. After 107\nsteps2, the agent can control what is controllable. The SAC, ICM-(S/E) and HIRO baselines attempt to\nsolve each task independently and spend resources equally between tasks. All four baselines succeed\nin the locomotion task only. They do not learn to pick up any of the other objects and transport them\nto a desired location. As a remark, the arena is relatively large such that random encounters are not\nlikely. Providing oracle reward signals makes the baselines (HIRO/SAC) learn to control the tool\neventually, but still signi\ufb01cantly slower than CWYC, see Fig. 8(b), and the heavy object remains\nuncontrollable see Suppl. D.2.\nRobotic manipulation. The robotic manipulation environment consists of a robotic arm with a\ngripper (3 + 1 DOF) in front of a table with a hook and a box (at random locations), see Fig. 2(b). The\nbox cannot be reached by the gripper directly. Instead, the robot has to use the hook to manipulate the\nbox. The observed state space is 40 dimensional. The environment is based on the OpenAI Gym [4]\nrobotics environment. The goal-spaces/tasks are de\ufb01ned as (1) reaching a target position with the\ngripper, (2) manipulating the hook, and (3) manipulation the box. Further details can be found in\nSuppl. C.2. Compared to the synthetic environment, object relations are much less obvious in this\nenvironment. Especially the ones involving the hook because of its non-trivial shape. This makes\nlearning object relation much harder. For instance, while trying to grasp the hook, the gripper might\ntouch the hook at wrong positions thus failing at manipulation. However, the objects are relatively\nclose to each other leading to more frequent random manipulations. The results are shown in Fig. 4.\nAsymptotically, both CWYC and the HER baseline manage to solve all three tasks almost perfectly.\nThe other baselines cannot solve it. Regarding the time required to learn the tasks, our method shows\na clear advantage over the HER baseline, solving the 2nd and 3rd task faster.\n\n(a) trajectory\n\n(b) prediction error\n\n(c) relation learning\n\nFigure 5: Surprise and relational funnel state learning in the synthetic experiment. (a) the agent\nbumps into the tool (the red line indicates the agent\u2019s trajectory before the encounter, the blue line\nindicates the joint trajectory of the agent and the tool (after encounter)); (b) surprise (prediction error\nabove 5\u03c3 con\ufb01dence level) marks the funnel state where tool\u2019s and agent\u2019s position coincide. (c)\navg. distance (cid:104)d(g, g(cid:63))(cid:105) of the generated goals g (locomotion targets) from the tool location g(cid:63) in\ndependence of the number of surprising events.\n\n2an unsuccessful rollout takes 1600 steps, so 107 steps are in the order of 104 trials\n\n7\n\n\f(a) resource allocation\n\n(b) task planner B\n\n(c) learned task graph\n\nFigure 6: Resource allocation and task planning structure in the synthetic environment. (a) resource\nallocation (Fig. 1(2)): relative time spend on each task in order to maximize learning progress. (b)\nTask planner (Fig. 1(3)): the probabilities Bi,j of selecting task j (column) before task i (row).\nSelf-loops (gray) are not permitted. Every sub-task sequence begins in the start state. (c) Learned\ntask graph (Fig. 1(4)) derived from (b). The arrows point to the preceding task, which corresponds to\nthe planning direction. The red states show an example plan for moving the heavy box.\n\n5 Analysis and Ablation: Why and How CWYC Works\nHow does the agent gain control of the environment? We start by investigating how the surprising\nevents help to identify the funnel states/relationships \u2013 a critical part of our architecture. When the\nagent is, for instance, involuntarily bumping into a tool, the latter will suddenly move \u2013 causing a\nlarge prediction error in the forward models of the tool goal-space, see Fig. 5(a,b). Only a few of such\nsurprising observations are needed to make the sub-goal generators, Fig. 1(5), effective, see Fig. 5(c).\nA more detailed analysis follows below. For further details on the training process, see Suppl. A.5.\nResource allocation is managed by the task selector, Fig. 1(2), based on maximizing learning\nprogress and surprise, see Eq. 1. As shown in Fig. 6(a), starting from a uniform tasks selection, the\nagent quickly spends most of its time on learning locomotion, because it is the task where the agent\nmakes the most progress in, cf. Fig. 3. After locomotion has been learned well enough, the agent starts\nto concentrate on new tasks that require the locomotion skill (moving tool and the \u201c50% object\u201d).\nAfterwards, the heavy object becomes controllable due to the competence in the tool manipulation\ntask (at about 3.5 \u00b7 106 steps). The agent automatically shifts its attention to the object manipulation\ntask.\nThe task selector produces the expected result that simple tasks are solved \ufb01rst and stop getting\nattention as soon as they cannot be improved more than other tasks. This is in contrast to approaches\nthat are solely based on curiosity/prediction error. When all tasks are controllable (progress plateaus)\nthe 50% object attracts most of the agent\u2019s attention due to randomness in the success rate. As a\nremark, the learned resource allocation of the oracle agent is similar to that of CWYC.\nNext, we study how the agent understands the task structure. The funnel states , discovered\nabove, need to be visited frequently in order to collect data on the indirectly controllable parts of the\nenvironment (e. g. tool and heavy box). The natural dependencies between tasks is learned by the\n\nlocomotion \u2192 tool\n\ntool \u2192 heavy object\n\nl\na\ni\nt\ni\nn\ni\n\nFigure 7: Subgoal proposal networks and the learned object relationships. Intuitively, a nonzero value\n(red) indicates that this relationship is used. For clarity, each panel shows min(|w1|,|w2|) (see Eq. 3)\nof the relevant goal proposal networks. Example situations with the corresponding generated goal are\npresented on the right. The locomotion and tool task goals, i.e red and purple \ufb02ags, are generated by\nthe subgoal generators, Fig. 1(5), the object goal, i.e yellow \ufb02ag, is sampled randomly.\n\n8\n\nrandomobj50%objtoollocostartheavyobj\f(a)\n\n(b)\n\n(c)\n\n(d)\n\nFigure 8: (a) Performance comparison of ablated versions of CWYC: uniform task selection (\u2020) and\nwithout surprise signal ((cid:5)). (b) baselines with oracle reward on tool task. (c-d) number of positive\ntraining samples for goal network and quality of sampled goals (see Fig. 5) CWYC(cid:5) vs. CWYC.\n\ntask planner B, Fig. 1(3). Initially the dependencies between the sub-tasks are unknown such that\nBi,j = 0 resulting in a 1/K+1 probability of selecting a certain preceding sub-task (or \u201cstart\u201d). After\nlearning, the CWYC agent has found which tasks need to be executed in which order , see Fig. 6(b-c).\nWhen executing a plan, sub-goals have to be generated. This is where the relational funnel states\nlearned by the sub-goal generators (Fig. 1(5)) come in. The sub-goal generators Gi,j learn initially\nfrom surprising events and attempt to learn the relation among the components of the observation\nvector. For instance, every time the tool is moved, the agent\u2019s location is close to that of the tool.\nFigure 7 displays the learned relationships for the sub-goal generation for the locomotion \u2192 tool\ntransition and for the tool \u2192 heavy object transition. A non-zero value indicates that the corresponding\ncomponents are involved in the relationship. The full parametrization is visualized and explained in\nSuppl. A.5. The system identi\ufb01es that for successfully solving the tool task the coordinates with the\nagent and the tool have to coincide. Likewise, for moving the heavy box, the agent, tool, and heavy\nbox have to be at the same location. The goal proposal network updates the current goal every 5 steps\nby computing the goal with the maximal value, see Suppl. A.5 for more details.\nWe ablate different components of our architecture to demonstrate their impact on the performance.\nWe remove the surprise detection, indicated as CWYC(cid:5). A version without resource allocation\n(uniform task sampling) is denoted as CWYC\u2020. Figure 8(a) shows the performance for the different\nablation studies and reveals that the surprise signal is a critical part of the machinery. If removed,\nit reduces the performance to the SAC baseline, i. e. only solves the locomotion task. Figure 8(c,d)\nprovide insight why this is happening. Without the surprise signal, the goal proposal network does not\nget enough positive training data to learn from; hence, constantly samples random goals prohibiting\nsuccessful switches which would create additional training data. Logically, the resource allocation\nspeeds up learning such that the hard tasks are mastered faster, cf. CWYC\u2020 and CWYC.\n\n6 Conclusion\nWe present the control what you can (CWYC) method that makes an autonomous agent learn to\ncontrol the components of its environment effectively. We adopt a task-planning agent architecture\nwhile all components are learned from scratch. Driven by learning progress, the IM agent learns\nan automatic curriculum which allows it to not invest resources in uncontrollable objects, nor try\nunproportionally often to improve its performance on not fully solvable tasks. This key feature\ndifferentiates CWYC from approaches solely based on curiosity.\n\nAcknowledgement\nJia-Jie Zhu is supported by funding from the European Union\u2019s Horizon 2020 research and innovation\nprogramme under the Marie Sk\u0142odowska-Curie grant agreement No 798321. The authors thank the\nInternational Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Sebas-\ntian Blaes and Marin Vlastelica Pogan\u02c7ci\u00b4c. We acknowledge the support from the German Federal\nMinistry of Education and Research (BMBF) through the T\u00fcbingen AI Center (FKZ: 01IS18039B).\n\n9\n\n\fReferences\n[1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder,\nBob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience\nreplay. In Advances in Neural Information Processing Systems (NIPS\u201917), pages 5048\u20135058,\n2017.\n\n[2] Adrien Baranes and Pierre-Yves Oudeyer. Active Learning of Inverse Models with Intrinsically\nMotivated Goal Exploration in Robots. Robotics and Autonomous Systems, 61(1):69\u201373, January\n2013.\n\n[3] Dominik Baumann, Sebastian Trimpe, Jia-Jie Zhu, and Georg Martius. Deep reinforcement\nlearning for event-triggered control. In IEEE Conference on Decision and Control, CDC\u201918,\npages 943\u2013950, 2018.\n\n[4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang,\n\nand Wojciech Zaremba. OpenAI Gym. arXiv:1606.01540, 2016.\n\n[5] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt\nBotvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa-\ntion. arXiv:1901.11390, 2019.\n\n[6] Martin V. Butz. Which structures are out there. In Thomas K. Metzinger and Wanja Wiese,\neditors, Philosophy and Predictive Processing, chapter 8. MIND Group, Frankfurt am Main,\nGermany, 2017.\n\n[7] Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh. Intrinsically motivated rein-\nforcement learning. In Advances in neural information processing systems (NIPS\u201905), pages\n1281\u20131288, 2005.\n\n[8] C\u00e9dric Colas, Pierre-Yves Oudeyer, Olivier Sigaud, Pierre Fournier, and Mohamed Chetouani.\nCURIOUS: intrinsically motivated modular multi-goal reinforcement learning. In International\nConference on Machine Learning (ICML\u201919), pages 1331\u20131340, 2019.\n\n[9] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for\nreinforcement learning agents. In International Conference on Machine Learning (IMCL\u201918),\npages 1514\u20131523, 2018.\n\n[10] S\u00e9bastien Forestier, Yoan Mollard, and Pierre-Yves Oudeyer. Intrinsically motivated goal\n\nexploration processes with automatic curriculum learning. arXiv:1708.02190, 2017.\n\n[11] Alison Gopnik. The gardener and the carpenter: What the new science of child development\n\ntells us about the relationship between parents and children. Macmillan, 2016.\n\n[12] Alison Gopnik, Clark Glymour, David Sobel, Laura Schulz, Tamar Kushnir, and David Danks.\nA theory of causal learning in children: Causal maps and bayes nets. Psychological review,\n111:3\u201332, 02 2004.\n\n[13] Christian Gumbsch, Sebastian Otte, and Martin V Butz. A computational model for the\ndynamical learning of event taxonomies. In Proceedings of the 39th Annual Meeting of the\nCognitive Science Society, pages 452\u2013457, 2017.\n\n[14] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning\nwith deep energy-based policies. In International Conference on Machine Learning (ICML\u201917),\npages 1352\u20131361, 2017.\n\n[15] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-\npolicy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy\nand Andreas Krause, editors, Proceedings of the 35th International Conference on Machine\nLearning, volume 80 of PMLR, pages 1861\u20131870. PMLR, 10\u201315 Jul 2018.\n\n[16] WPMH Heemels, Karl Henrik Johansson, and Paulo Tabuada. An introduction to event-triggered\nand self-triggered control. In Decision and Control (CDC), 2012 IEEE 51st Annual Conference\non, pages 3270\u20133285. IEEE, 2012.\n\n10\n\n\f[17] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo,\nDavid Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary\ntasks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France,\nApril 24-26, 2017, Conference Track Proceedings, 2017.\n\n[18] F. Kaplan and P.-Y. Oudeyer. Maximizing learning progress: An internal reward system for\n\ndevelopment. Embodied Arti\ufb01cial Intelligence, pages 259\u2013270, 2004.\n\n[19] Lydia E Kavraki, Petr Svestka, J-C Latombe, and Mark H Overmars. Probabilistic roadmaps\nfor path planning in high-dimensional con\ufb01guration spaces. IEEE transactions on Robotics and\nAutomation, 12(4):566\u2013580, 1996.\n\n[20] Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. Empowerment: A universal\nagent-centric measure of control. In Evolutionary Computation, 2005. The 2005 IEEE Congress\non, volume 1, pages 128\u2013135. IEEE, 2005.\n\n[21] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical\ndeep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In\nAdvances in Neural Information Processing Systems (NIPS\u201916), pages 3675\u20133683, 2016.\n\n[22] Steven M LaValle. Rapidly-exploring random trees: A new tool for path planning. Technical\n\nreport, Computer Science Dept, Iowa State University, October 1998. TR 98-11.\n\n[23] D. Y. Little and F. T. Sommer. Learning and exploration in action-perception loops. Frontiers\n\nin Neural Circuits, 7(37), 2013.\n\n[24] O\ufb01r Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-ef\ufb01cient hierarchical\nreinforcement learning. In Advances in Neural Information Processing Systems (NIPS\u201917),\npages 3303\u20133313, 2018.\n\n[25] P.-Y. Oudeyer, F. Kaplan, and V. V. Hafner. Intrinsic motivation systems for autonomous mental\n\ndevelopment. IEEE Trans. on Evo. Computation, 11(2):265\u2013286, 2007.\n\n[26] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration\nby self-supervised prediction. In International Conference on Machine Learning (ICML\u201917),\nvolume 2017, 2017.\n\n[27] Alexandre P\u00e9r\u00e9, S\u00e9bastien Forestier, Olivier Sigaud, and Pierre-Yves Oudeyer. Unsupervised\nlearning of goal spaces for intrinsically motivated goal exploration. In International Conference\non Learning Representations, ICLR\u201918, Conference Track, 2018.\n\n[28] Michael Posa, Cecilia Cantu, and Russ Tedrake. A direct method for trajectory optimization\nof rigid bodies through contact. The International Journal of Robotics Research, 33(1):69\u201381,\n2014.\n\n[29] Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom\nvan de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing\nsolving sparse reward tasks from scratch. In International Conference on Machine Learning\n(ICML\u201918), volume 80, pages 4344\u20134353, 2018.\n\n[30] Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter\nBattaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In\nAdvances in neural information processing systems (NIPS\u201917), pages 4967\u20134976, 2017.\n\n[31] J. Schmidhuber. Developmental robotics, optimal arti\ufb01cial curiosity, creativity, music, and the\n\n\ufb01ne arts. Connection Science, 18(2):173\u2013187, 2006.\n\n[32] J\u00fcrgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building\nneural controllers. In International Conference on Simulation of Adaptive Behavior: From\nanimals to animats (SAB\u201991), pages 222\u2013227, 1991.\n\n[33] Bruno Siciliano and Oussama Khatib. Springer handbook of robotics. Springer, 2016.\n\n11\n\n\f[34] Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob\nFergus. Intrinsic motivation and automatic curricula via asymmetric self-play. In International\nConference on Learning Representations, ICLR 2018, Conference Track, 2018.\n\n[35] E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control.\nIn IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS\u201912), pages\n5026\u20135033, Oct 2012.\n\n[36] S. Yi, F. Gomez, and J. Schmidhuber. Planning to be surprised: Optimal Bayesian exploration\nin dynamic environments. In Proc. Fourth Conference on Arti\ufb01cial General Intelligence (AGI),\nGoogle, Mountain View, CA, 2011.\n\n[37] Keyan Zahedi, Georg Martius, and Nihat Ay. Linear combination of one-step predictive\ninformation with an external reward in an episodic policy gradient setting: a critical analysis.\nFrontiers in Psychology, 4(801), 2013.\n\n[38] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl\nTuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Relational deep reinforcement\nlearning. arXiv:1806.01830, 2018.\n\n12\n\n\f", "award": [], "sourceid": 6819, "authors": [{"given_name": "Sebastian", "family_name": "Blaes", "institution": "Max-Planck Institute for Intelligent Systems, Tuebingen, Germany"}, {"given_name": "Marin", "family_name": "Vlastelica Pogan\u010di\u0107", "institution": "Max Planck Institute for Intelligent Systems"}, {"given_name": "Jiajie", "family_name": "Zhu", "institution": "Max Planck Institute for Intelligent Systems"}, {"given_name": "Georg", "family_name": "Martius", "institution": "MPI for Intelligent Systems"}]}