{"title": "Visual Memory for Robust Path Following", "book": "Advances in Neural Information Processing Systems", "page_first": 765, "page_last": 774, "abstract": "Humans routinely retrace a path in a novel environment both forwards and backwards despite uncertainty in their motion. In this paper, we present an approach for doing so. Given a demonstration of a path, a first network generates an abstraction of the path. Equipped with this abstraction, a second network then observes the world and decides how to act in order to retrace the path under noisy actuation and a changing environment. The two networks are optimized end-to-end at training time. We evaluate the method in two realistic simulators, performing path following both forwards and backwards. Our experiments show that our approach outperforms both a classical approach to solving this task as well as a number of other baselines.", "full_text": "Visual Memory for Robust Path Following\n\nAshish Kumar\u2217 Saurabh Gupta\u2217\n\nDavid Fouhey\n\nashish_kumar@berkeley.edu, {sgupta, dfouhey, svlevine, malik}@eecs.berkeley.edu\n\nUniversity of California, Berkeley\n\nSergey Levine\n\nJitendra Malik\n\nAbstract\n\nHumans routinely retrace paths in a novel environment both forwards and back-\nwards despite uncertainty in their motion. This paper presents an approach for\ndoing so. Given a demonstration of a path, a \ufb01rst network generates a path ab-\nstraction. Equipped with this abstraction, a second network observes the world\nand decides how to act to retrace the path under noisy actuation and a changing\nenvironment. The two networks are optimized end-to-end at training time. We eval-\nuate the method in two realistic simulators, performing path following and homing\nunder actuation noise and environmental changes. Our experiments show that our\napproach outperforms classical approaches and other learning based baselines.\n\n1\n\nIntroduction\n\nConsider the \ufb01rst morning of a conference in a city you have never been to. Rushing to the \ufb01rst\ntalk, you might follow your phone\u2019s directions through a series of twists and turns to reach the\nvenue. When you return later in the day, you can retrace your steps to your hotel relatively robustly,\nremembering to take a left turn at the bistro and keep straight past the coffee shop. The next day,\nyou may probably only look at your phone to check your email. At \ufb01rst glance, this seems like a\ntrivial ability. Humans routinely do this, for instance when a friend shows you the bathroom in their\napartment or when you go to a room in a new building. On second glance though, it is an amazing\nability since one never retraces one\u2019s steps exactly and the visual experience is constantly changing in\nfairly dramatic ways: people move their cars, shops open and close, and seasons change. This paper\naims to replicate this ability to retrace paths (including reversals) in new environments with imperfect\nability to replicate one\u2019s actions (actuation) as well as a changing world.\nHow might we solve this problem? One classical approach, common in robotics, would be to build a\nfull 3D model of the world via SLAM, from building facades to the side-mirrors of cars, during the\n\ufb01rst pass; after this is done, path following reduces to localizing in the model and selecting the best\naction. For the task of navigation, this is simultaneously too much work \u2013 precisely reconstructing\nthe facade is less important than recognizing it as \u201cthe bistro at which I turn left\u201d \u2013 as well as too\nlittle work \u2013 the parts of the reconstruction that provide stable localization and the parts that do not\nare mixed together with no way to disentangle them.\nGiven these dif\ufb01culties of classical approaches, a large number of learning-based approaches have\nsprung up to solve this and other related navigation tasks (e.g. [28, 27, 18, 35, 43, 30]). In these\nworks, an agent directly predicts actions from image observations in an end-to-end fashion. However,\nunlike our setup of a single demonstration in a new environment, many of these setups require the\nagent to have a great deal of experience with the test environment via reward based supervision\n[27, 28, 43] or exhaustive human demonstrations [35]. Moreover, unlike most work on navigation in\nnew environments [18], our setup poses the additional challenge of noisy actuation as well as a world\nthat can change between the initial demonstration and path execution.\n\n\u2217Equal contribution. Project website with videos: https://ashishkumar1993.github.io/rpf/.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fFigure 1: Problem Setup: Given images and associated actions from a traversed path (A), we want to retrace\nthe path both forward and backward under actuation noise (i.e. uncertainty in movement) in a changing world (e.g.\nthe chairs have been moved between the demonstration and execution) (B). This paper presents an end-to-end\nneural architecture for this task. We use the demonstration path to generate a path abstraction that is used by a\nlearned controller to reliably convey the agent to the desired target location at execution time.\n\nOur approach, which we describe in Section 3, consists of a module that learns to convert a series of\nobservations of a path to an abstract representation, and a learned controller that implicitly localizes\nthe agent along this abstracted path using the current observation and outputs actions that bring the\nagent to the desired goal location. We see a number of advantages to training the whole approach\nend-to-end on data in comparison to classical approaches. First, the learned model can use statistical\nregularities to make its performance more robust: for example, it can learn to count doors in a\ntexture-less hallway rather than localize at each point and when it returns along a path, it can learn\nto look on the right for a table that was previously on the left. Second, by virtue of being learned\nentirely end-to-end, rather than being learned and designed piecemeal, the approach can automatically\nlearn the features that are necessary for the task at hand. As a concrete demonstration, we evaluate\na homing task in which the agent retraces a path in reverse; the network learns to produce features\nnecessary for solving this task without explicitly designing any wide-baseline features or proxy tasks.\nWe evaluate our approach on multiple datasets in Section 4 in a series of experiments that aim to\nprobe to what extent we can learn to retrace a path under noisy actuation and in a changing world. We\ncompare to a variety of classical and learned alternate approaches, and outperform them. In particular,\nour experimental results show the value of end-to-end learning the entire path-following process.\n\n2 Related Work\n\nIn this work, we study the problem of retracing a path under noisy actuation and a changing world.\nThis touches on classical work in robotics and recent works learning-based efforts.\nOne classic approach to solving the path retracing problem (also referred to as Visual Teach and\nRepeat [15]) is to chain together core robotics primitives of mapping, localizing, and planning: on\nthe initial demonstration, the agent builds a 3D map; during subsequent navigation attempts, the\nagent localizes itself in the map and generates actions accordingly. Each problem has been studied\nextensively in robotics, typically focusing on geometric solutions, i.e. using metric maps and locations.\nFor example, mapping has been often studied as the classic simultaneous localization and mapping\n(SLAM) problem [12\u201314, 19] in which one builds a 3D map of the world. Localization is often\nframed as recovering camera pose with respect to a global or local 3D map [9, 10] or by performing\nvisual odometry [29, 42] to obtain a metric displacement from a start position. Finally, planning is\noften done assuming direct access to a noiseless map and often a noiseless agent location [6, 22, 25].\nThe distinction from this line of work is that our approach is learned end-to-end. This means that\nintermediate representations are automatically learned, rich non-geometric information can be incor-\nporated, and modules are optimized jointly for the end-goal. Obviously, researchers are aware of the\nlimitations of purely geometric and pipelined approaches, and have developed extensions: explicitly\nincorporating semantics into SLAM [4], adding margins in planning to account for uncertainty [3],\nand using learning for sub-sets of the SLAM problem [5, 8, 21, 23, 32, 36, 40]. The strength of our\nproposed method is that these strategies are learned automatically and are speci\ufb01ed by what is needed\nempirically to navigate an environment as opposed to human intuition.\nWe are not the \ufb01rst to recognize the potential for end-to-end task-driven navigation. In recent\nyears there has been a \ufb02urry of work in this area. A lot of this work focuses on the shorter time-\n\n2\n\nA. Demonstration.B. Execution under noisy actions and changes in the world.\u2026ES\u2026ChallengesActuation NoiseChanges in WorldfES\fFigure 2: Proposed Approach: As input, our model takes a sequence of images (processed by \u03c6) and actions\nat these images a(p)j. It abstracts this sequence into a sequence of memories. A second, recurrent, network\n\u03c0 uses this sequence of memories to emit actions that retrace the path. At each time-step, \u03c0 reads in: (a) the\nsequence, softly attending to the relevant part at past time-step via \u03b7; (b) the current observation O from the\nworld (also processed by \u03c6); as well as its recurrent hidden state h. As output, \u03c0 updates the attention location \u03b7\nand its hidden state, and emits action \u02c6a that the agent executes in the world.\n\nscale task of collision avoidance, e.g. how do I move through the door without bumping into it\n[11, 16, 17, 20, 33, 34]. In contrast, our work focuses on the longer time-scale problem of path\nfollowing e.g. how do I get back to my of\ufb01ce from the coffee machine. At this time-scale, some\nworks have framed the problem of navigation as learning to reach different goals in a \ufb01xed training\nenvironment [27, 28], designing policies that can directly act in new test environments [18, 24, 30, 41],\nself-supervised learning for reaching goals in a well-traversed environment [35] or by following a\ndemonstration [31]. Our work is more similar to this last line of work [31, 35]. It is distinguished,\nhowever, by its lack of explicit localization as well as its navigation under actuation noise and\nchanging environments. This noise is a crucial distinction because noiseless execution of actions and\nan unchanged environment means that memorization alone is suf\ufb01cient. We also study the task of\nhoming, where no direct demonstration images are available.\n\n3 Robust Path Following\n\nProblem Setup. Consider an agent that we are trying to train to operate in a new environment E\nthat it has never been in before. Let us assume that the agent\u2019s state at time t is represented by st,\nand that the agent has some primitive set of stochastic actions A that it can execute. Executing an\naction a \u2208 A, takes the agent to state st+1 via a stochastic transition function f i.e. st+1 is sampled\nrandomly from the distribution f (st, a, E). We assume that the agent is equipped with a \ufb01rst person\nRGB camera to obtain visual observations of the environment I = \u03c1(E, s), where the function \u03c1\nrenders an environment E from agent\u2019s current location speci\ufb01ed by s. We do not tackle the problem\nof higher-level planning and abstract away low-level motor control to focus on the problem of robust\npath following. Our approach can be used with classical (or even learned) approaches for path\nplanning and low-level motor control.\nSuppose we move this agent around in this new environment along a path p. Given such a traversal,\nwe want the agent to be able to reliably re-trace the path p or to follow a related path such as the\nreversed path (denoted \u02dcp). We want to be able to do this in situations where the agent has noisy\nactuation and sensing, and the environment changes (to E(cid:48)) between the demonstration of p and\nwhen the agent is tested to autonomously traverse p (or \u02dcp). Note that, the only sensory information\nthat is available to the agent as it is trying to traverse the path p (or \u02dcp) in the environment E (or\nE(cid:48)) is via \u03c1. It does not have access to the ground truth state of the system st, but only RGB image\nobservations of the environment from that state.\nOur goal is to learn a policy \u03a0 that will achieve this. As input, \u03a0 assumes access to the: path p;\nactions a(p) taken while executing the path; visual observations along the path I = {I1 . . . IJ : Ij =\n\u03c1(E, pj)}; as well as the current visual observation O. As output, \u03a0 predicts actions from A that\nsuccessfully and ef\ufb01ciently convey the agent to the destination pJ, i.e. \u02c6a = \u03a0(p, I, O). This action \u02c6a\nis executed in the stochastic environment; the agent obtains the new visual observation at the next\nstate; and this process is repeated for a \ufb01xed number of time steps.\nWe now describe the policy function \u03a0. Intuitively, our policy function \u03a0 uses the observation O to\nimplicitly localize the agent with respect to its memory of the path. Given this localization, the policy\n\n3\n\nESMemoryLegend:ObserveAct\u03d5\u03c8Network\u03d5h\u03bco\u0302aFeature\u03b7\u2026World\u03c0\u03d5\freads out relevant information (such as relative pose and actions) and uses this in context of O to take\nan action that conveys the agent to the desired target location. The entire policy is implemented using\nneural network modules that are differentiable and learned end-to-end using training data for the task\nof ef\ufb01ciently going to the desired goal location.\nWe \ufb01rst describe the basic architecture for \u03a0 where we want to retrace the same path p that was\ndemonstrated and then describe the extension to retrace a related path \u02dcp. We call our proposed policy\n\u03a0 as Robust Path Following policy and denote it by RPF.\n\n3.1 Learned Controller for Robust Path Following\n\nThe policy \u03a0(p, I, O) is realized as follows. We \ufb01rst use p and I to compute a path description M (p)\nthat captures the local information needed to follow the path. M (p) is a sequence of tuples consisting\nof features of the reference image, associated reference position and the associated reference action\nfor each step j in the trajectory, or\n\nM (p) = {(a(p)j, \u03c6(Ij)) : j \u2208 [1 . . . J]}.\n\n(1)\nWe use this path description with a learned controller that takes as input the current image observation\nO to output actions to follow the path under noisy actuation. We represent the policy \u03c0 with a\nrecurrent neural network that iterates over the path description M (p) as the agent moves through the\nenvironment. This iteration is implemented using attention that traverses over the path description.\nAt each step, the path signature is read into \u00b5t with differentiable soft attention centered at \u03b7t:\n\n(cid:88)\n\n\u00b5t =\n\n\u03c8 (M (p)j) e\u2212|\u03b7t\u2212j|.\n\n(2)\n\nThe recurrent function \u03c0 with state ht is implemented as:\n\nj\n\nht+1, b, \u02c6a = \u03c0(ht, \u00b5t, \u03c6(O)) and \u03b7t+1 = \u03b7t + \u03c3(b).\n\n(3)\nAs input, it takes the internal state, ht, attended path signature \u00b5t and featurized image observation\n\u03c6(O). In return, it gives a new state ht+1, pointer increment b, and action \u02c6a that the agent should\nexecute. This pointer increment is passed through an increment function \u03c3, and added to \u03b7t to yield\nthe new pointer \u03b7t+1. Given the agent moves by one step per action in expectation, we use 1 + tanh\nas the increment function \u03c3. We set \u03b71 = 1 and h1 = 0.\nNote that we factor out the controller from the path description. This factorization of the environment\nand goal speci\ufb01c information into a path description that is separate from the policy lets us learn a\nsingle policy \u03c0 that can do different things in different environments with different path descriptions\nwithout requiring any re-training or adaptation. The policy can then also be thought of as a robust\nparameterized goal-oriented closed-loop controller.\n\n3.2 Feature Synthesis for Following Related Paths\n\nSo far our approach can only repeat the paths that we have already taken, but our approach can be\nextended to follow paths \u02dcp that are related to but not the same as the path p that was demonstrated.\nWe do this by synthesizing features for the path \u02dcp using whatever information is available for the\ndemonstrated path p. We synthesize features \u02c6\u03c6(\u02dcpj) for location \u02dcpj using observed features \u03c6(Ii)\nfrom location pi as follows:\n\n\u03c9i,j = \u2126 ((\u03c6(Ii), \u03b4(pi, \u02dcpj))) and \u02c6\u03c6(\u02dcpj) = \u03a3g (\u03c91,j, \u03c92,j, . . . , \u03c9N,j) .\n\n(4)\nHere, the function \u03b4 computes the relative pose of image Ii with respect to the desired synthesis\nlocation \u02dcpj. \u03c6 computes the representation for image Ii through a CNN followed by two fully\nconnected layers. \u2126 fuses the relative pose with the representation for the image to obtain the\ncontribution \u03c9i,j of image Ii towards representation at location \u02dcpj. These contributions \u03c9i,j from\ndifferent images are accumulated through a weighted addition by function \u03a3g to obtain the synthesized\nfeature \u02c6\u03c6(\u02dcpj) at location \u02dcpj. The path description M (\u02dcp) can then be obtained as a collection of\ntuples (a(\u02dcp)j, \u02c6\u03c6(\u02dcpj)).\nImplementation Details. We now describe the particular architecture that we use throughout the\npaper. \u03c6 is a 5 layer Convolutional Network with [32, 64, 128, 256, 512] \ufb01lters respectively. Each\nconv layer is followed by a max-pooling. \u03c8 and \u2126 are fully connected networks consisting of two\nlayers; \u03c0 is implemented using GRUs. We train the entire network from scratch in an end-to-end\nmanner using Adam optimizer for 120000 iterations, where each episode is 40 steps long.\n\n4\n\n\f4 Experiments\n\nThis paper studies the task of retracing a route in a new environment (either forwards or backwards)\nunder noisy actuation and a changing world, given a demonstration. Our experiments evaluate: a) to\nwhat extent can we solve this task, b) what is the role of visual memories, and c) how our proposed\nsolution compares to classical geometry-based and other learning-based solutions. Crucially, we\nfurther study how our policies perform on settings outside of what they were explicitly trained on.\n\n4.1 Experimental setup\n\nSimulators. We use two simulation environments that permit rendering from arbitrary viewpoints\nand allow separation of held-out environments for testing. The \ufb01rst simulator is based on real world\nscans from Stanford Building Parser Dataset [2] (SBPD) and the Matterport 3D Dataset [7](MP3D).\nThese scans have been used to study navigation tasks in [18], and we adapt their publicly available\nsimulation code. We use splits that ensure that the testing environment comes from an entirely\ndifferent building than the training or validation environments. The second simulation environment is\nbased on SUNCG [39]. SUNCG consists of synthetic indoor environments with manually created\nroom and furniture layouts that have corresponding textured meshes [39]. Because these environments\nare graphics codes, SUNCG permits evaluation of the effect of environmental changes. In particular,\nobjects can be removed without inducing artifacts that a network will pick up on. Once again, splits\nensure no overlap between training and testing environments.\nAgent Actions. We assume that the agent has 4 macro-actions, stay in place, rotate left or right by \u03b8\n(= 30\u25e6), and move forward x units (= 40cm).\nNoise Model. Our work studies path retracing both with actuation noise (i.e. the outcome of actions\nis stochastic) and a changing world (the world changes between the demonstration and autonomous\noperation of the agent). Actuation Noise: In both environments, when the agent outputs the rotation\nactions it actually rotates by \u223cNtrunc(\u03b8, 57.3\u25e6, 0.2|\u03b8|). Here, Ntrunc(\u00b5, \u03c3, \u03b4) refers to a normal\ndistribution with mean \u00b5 and standard deviation \u03c3 that is truncated to (\u00b5 \u2212 \u03b4, \u00b5 + \u03b4). When the\nagent executes a move forward action it rotates by \u223cNtrunc(0, 57.3\u25e6, 0.2|\u03b8|) and then translates by\n\u223cNtrunc(x, 5cm, 0.2x). The factor of 0.2 controls the amount of noise in actuation and we vary\nit in our experiments. World Changes: World changes are studied in the SUNCG environment.\nDemonstrations are provided in an environment with objects removed uniformly with a probability of\nr (= .5). The task is to get to the desired target location in presence of even fewer objects (r is .1, or\n.3) or additional objects (r is .7, or .9).\nTasks. Given a path p from p0 to pT , we consider the two tasks of trajectory following i.e. going\nfrom p0 to pT as well as homing i.e. going from pT to p0. We evaluate these tasks under a variety\nof noise conditions and environments.\nEvaluation Criteria. We characterize the success of the agent by measuring how close it gets to\nthe goal location. We analyze each approach over 500 trials in a novel environment (not used for\ntraining). We report three metrics: a) Success Rate (reaching within 2 steps or 10% of the initial\ndistance to goal, whichever is larger), b) SPL (Success weighted by normalized inverse Path Length\nas described in [1]), and c) Median Normalized Distance to goal at end of episode.\nModel Training. We use imitation learning to train our policies. Although the agent never has access\nto its true location as it traverses the environment, the true location is available in the simulator. This\nis used to compute a set of \u2018good actions\u2019 that will convey the agent to the desired target location.\nThis set of good actions are ones that lead to a larger reduction in the distance to goal when compared\nto forward action. We optimize the policy to minimize the negative log of the sum of probabilities of\nall good actions at each time step.\n\n4.2 Baselines\n\nWe compare with a number of baselines that represent either classical approaches or test the impor-\ntance of various components of our system.\nOpen Loop. We repeat the reference actions (or their reverse for homing). Under perfect actuation\nthis would achieve perfect performance. With actuation noise, this baseline is a measure of the\nhardness of the task and tests to what extent learning to act under actuation noise is necessary.\n\n5\n\n\fTable 1: Performance over 500 trials on the test set (SBPD area4) in base settings for the Following and\nHoming tasks. We report Success Rate, SPL and Median Normalized Distance. See text for details.\n\nOpen Loop Visual Servoing\n\n3D Recons. + Localize\n\nRPF (no visual mem.)\n\nRPF\n\nFollowing\nSuccess Rate \u2191\nSPL \u2191\nMedian Normalized Distance \u2193\n\nHoming\nSuccess Rate \u2191\nSPL \u2191\nMedian Normalized Distance \u2193\n\n0.216\n0.191\n0.257\n\n0.216\n0.191\n0.257\n\n0.318\n0.293\n0.204\n\n\u2013\n\u2013\n\u2013\n\n0.826\n0.766\n0.086\n\n0.000\n0.000\n0.853\n\n0.782\n0.648\n0.068\n\n0.780\n0.644\n0.062\n\n0.866\n0.726\n0.056\n\n0.866\n0.740\n0.056\n\nVisual Servoing. For each action, we compute the L2 distance between SIFT feature matches of\ntarget reference image (initially set to \ufb01rst reference image) and image expected after executing that\naction (we obtain this image by virtually executing this step in the simulator). The policy actually\nexecutes the action that has the lowest distance. To decide when to increment the target image, we\ncheck the ground truth proximity to the next reference image. We stop when the agent reaches the\nlast target image.\n3D Reconstruction and Localization. We use the publicly available COLMAP package [37, 38] that\nimplements a variety of geometric mapping and localization algorithms. Note that these geometry-\nbased methods require high-resolution images at high frame rates. Thus, we sample high-resolution\nimages (1024 \u00d7 1024 vs. 224 \u00d7 224 for our policies) at 5\u00d7 the frame rate (145 vs. 30 images for our\npolicy for a trajectory of length 30) along the reference trajectory along with ground truth poses. This\nensures that reconstruction via SIFT key-point matching [26] and bundle adjustment always succeeds.\nGiven this reconstruction, the agent localizes itself by registering SIFT key-points on the current\nimage with the 3D reconstruction derived from the reference images. It then estimates free space by\nmarking a small region around each point on the reference trajectory as free. Given this inferred free\nspace and localization, it executes the action that most ef\ufb01ciently conveys it to the goal location. If\nthe localization fails, it rotates left at each step until localization succeeds (or the episode ends).\nRPF with No Visual Memory. We also compare to a policy without any visual memory. We do this\nby only using the action and pose as part of the memory M (p) in Eq. 1 i.e. removing \u03c6(Ij). We\nretain the rest of the architecture of RPF. Note that this is a competitive comparison as the policy can\nlearn to replay actions meaningfully in context of current images from the environment.\nAll learned policies are trained in base settings: the noise level is at 20%, reference trajectories are of\nlength 30, the policy is executed for 40 time steps, and these reference trajectories are sampled to\nbe far from obstacles. Figure 3 and Table 1 present our experimental results. We report the success\nrate, SPL, and median normalized distance and compare against the baselines described above. In\naddition, we also show a bootstrapped 95% con\ufb01dence interval for each plot.\n\n4.3 Results\n\nExperiments on Matterport Data. We \ufb01rst study the following and homing tasks in static environ-\nments. As the environment does not need to change, we do these experiments on realistic Matterport\ndata. In particular, we train policies on 4 \ufb02oors from SBPD and 6 buildings from MP3D. All policies\nare tested on area4 from SBPD which is from an altogether different building than the 4 \ufb02oors used\nfor training. We found adding visually diverse data from the MP3D dataset was crucial for good\nperformance as our models are trained entirely from scratch.\nFollowing Task. Table 1 (top) presents results for the trajectory following task. Open loop replay\nonly succeeds 22% of the time. This is because actuation noise compounds over time and causes\nboth drift and collisions. Visual feedback through servoing improves success rate to 32%, though it is\nlimited as it only localizes against a single image at a time. This is addressed by \u20183D Reconstruction\n+ Localization\u2019 which uses all reference images for localization (by reconstructing the environment),\nand achieves a success rate of 83%. RPF achieves a higher success rate of 87%. RPF without visual\nmemories succeeds 78% of the time. This is a striking result. Just the sequence of actions, when\nreplayed intelligently through a learned controller, leads to compelling path tracing performance.\nThat is, sequences of actions can robustly be interpreted in context of current image observations\nwithout any explicit 3D reconstruction or localization. Of course, additionally using visual memories\nfrom the demonstrated path improves performance.\n\n6\n\n\fFigure 3: Generalization Performance: Rows show different metrics, and columns show different test settings.\nVertical dotted lines mark \u2018base settings\u2019 that the policies were trained on. We show generalization of RPF to\n(far left) actuation noise levels, (center left) obstacle distance, and (center) path length for path following, and\n(center right) actuation noise for homing. The far right plot shows performance under environmental changes.\n\nHoming Task. Table 1 (bottom) shows results for the homing task on Matterport data. We set up\nthe homing task by sampling the same trajectories but simply providing images from a 180\u25e6 rotated\ncamera at each reference location. Thus open loop replay and RPF without visual memories perform\nabout the same as before. Note that this scenario is impossible for visual servoing as there are no\ndirect images to servo to. While, in principle 3D Reconstruction and Localization can tackle this\nscenario, it performs poorly (0% success, 0.85 median normalized distance) as SIFT features don\u2019t\nmatch well across large baselines. In comparison, RPF that learns to speculate features still performs\nequally well at 87%. Note that this is better than RPF without visual memories, demonstrating that\nour feature prediction technique is able to extract meaningful signal from related images.\nTesting on Out-of Train Settings. We have shown so far that our trained policies outperform\nappropriate baselines for the task when tested on novel environments. However, we are still training\nand testing on the same settings, such as noise level, trajectory length and distribution of trajectories.\nWe next test how well our learned policies work when we test them on settings they have not been\ntrained on. We do this by testing the policies trained in the \u2018Base Setting\u2019, in different settings in\nnovel environments. We explore three novel settings: a) Actuation Noise: we vary the noise level of\nthe environment, b) Harder Trajectories: we pick trajectories that are closer to obstacles and require\ncareful maneuverability, and c) Path Length: we increase the length of the trajectories to follow.\nWe do not retrain our policies for these settings, and simply execute the policy learned in the \u2018Base\nSetting\u2019 on these additional settings.\nFigure 3 (left 3 columns) presents the results. Rows plots the different metrics (success rate, SPL,\nand median normalized distance), and columns plot different test conditions. Plots in the \ufb01rst column\nshow performance as a function of actuation noise. Policies trained at 20% noise are tested under\nnoise varying between 0% and 50%. Open loop performs perfectly with no noise, but its performance\nrapidly degrades as noise increases. In comparison, RPF performance degrades gracefully. Plots in\nthe second column show performance as we move to harder trajectories that are sampled to be closer\nto obstacles (as we move from the left to the right on the plot). RPF degrades much more gracefully\nthan the 3D reconstruction based method. The third column shows performance as a function of\ntrajectory length. RPF policies generalize reasonably well to even 3\u00d7 longer trajectories than were\nseen during training. Column 4 shows similar plots of performance variation as a function of the\nactuation noise for the homing task. Finally, Figure 4 shows multiple rollouts from our RPF policy\nand contrasts them with purely open loop rollouts.\n\n7\n\n0.00.20.40.60.81.0Success Rate(Higher is better)Open LoopVisual Servoing3D Recons. + LocalizationRPF (no visual memory)RPF0.00.20.40.60.81.0SPL(Higher is better)0.00.10.20.30.40.5Actuation Noise 0.00.20.40.60.81.0MedianNormalized Distance(Lower is better)45678Closeness to ObstaclesGeneralization to Novel Settings for Path Following(SBPD + MP3D)3040506070Trajectory Length0.00.10.20.30.40.5Actuation NoiseHoming(SBPD + MP3D)2060100140180% Objects at Execution TimeEnvironmentalChanges (SUNCG)\fFollowing (Matterport)\n\nHoming (Matterport)\n\nChanging Env (SUNCG)\n\nFigure 4: Trajectory Visualizations: We visualize sample success and failure trajectory in top view for the\nthree different scenarios. We show multiple roll-outs by resampling multiple times from the noise model. Note\nthat these top views are shown here only for visualization, the policy does not receive them and operates purely\nbased on the \ufb01rst person views. RPF trajectories are in red, and open loop ones are in blue. Following and\nHoming: RPF trajectories more tightly follow the reference trajectory, while open loop roll-outs spread out and\ncollide with nearby obstacles. Overall, RPF gets to the goal more reliably, but fails when it drifts too far from the\nreference trajectory. Execution in Changing Envirnonment in SUNCG: Our approach is able to go around\nobstacles, but fails if a very large deviation is required.\nExperiments on SUNCG. We next report experiments on SUNCG for the base setting. We trained\non 48 houses from the House3D training set and report performance on 12 entirely disjoint houses\nfrom the test set. We plot the metrics in Figure 3 (last column). The base setting (shown with the\ndotted vertical line) here corresponds to when there are 100% objects at execution time i.e. the\nenvironment is the same between when the reference trajectory was provided, and when the agent has\nto execute the trajectory. As RPF learns features, it can better adapt to change in visual imagery in\nthe synthetic dataset as compared to feature-based geometric methods. Once again, RPF was trained\nin base settings.\nRobustness to Environmental Changes. Figure 3 (last column) also plots performance as the\nenvironment changes. Points to the left of the dotted line correspond to the setting where objects are\nremoved from the environment, while points to the right correspond to when objects are added into\nthe environment. In both these regimes, RPF continues to perform well. In contrast, performance for\nthe 3D reconstruction and localization method degrades sharply (see median normalized distance plot)\nas the environment at execution time deviates from one at demonstration time. This is expected and\nknown of geometry based methods that do not cope well with changes in the environment. Moreover,\nthe agent doesn\u2019t just need to be robust to visual changes between reference images and the current\nobservations, but it must also exhibit local going-around-behavior as the reference trajectory may go\nthrough the newly added obstacles. While RPF is able to do so when there is a minor detour, it fails\nwhen a much larger detour is required as shown in Figure 4 (right).\nAblations and Comparisons to Other Learning Methods. Figure 5 reports success rate for the\npath retracing task for other learning methods: a) Trained Nearest Neighbors: we use similarity\nbetween current observation with reference images to vote for the action, and b) GRU: a stan-\ndard recurrent model without the proposed sequential memory. We also compare to ablations of\nRPF: a) RPF without Recurrence: \u03c0 is implemented with a feed-forward network as opposed to\na recurrent function, and b) RPF with a Constant Increment instead of the learned increment\nfunction. RPF does bet-\nter than these other learn-\ning methods, and our choice\nof modeling \u03c0 as a recur-\nrent function, and learning\nhow to increment the pointer\nare important. We also in-\nvestigated other choices for\nimplementing the increment\nfunction \u03c3. Increment func-\ntions with a small spread\naround 1 worked better than\nother choices.\nDiscussion. In this paper, we studied the task of following paths under noisy actuation and changing\nenvironments. We operationalized insights from classical robotics with learning-based models and\ndesigned neural architectures that implicitly tackle localization to output actions to convey the robot\ndirectly to the target location. We made thorough comparisons with geometry-based classical methods\nand reasonable learning baselines, and demonstrated the effectiveness of our proposed approach.\n\nFigure 5: Other Comparisons: We report success rate for the path following\ntask on SBPD + MP3D for other learning based methods. See text for details.\n\n8\n\n0.00.10.20.30.40.5Actuation Noise0.00.20.40.60.81.0Success Rate(Higher is better)Nearest NeighborGRURPF (no recurrence)RPF (const. incr.)RPF45678Closeness to Obstacles3040506070Trajectory length\fReferences\n[1] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen\nKoltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied\nnavigation agents. arXiv preprint arXiv:1807.06757, 2018. 5\n\n[2] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese.\n\n3D semantic parsing of large-scale indoor spaces. In CVPR, 2016. 5\n\n[3] Brian Axelrod, Leslie Pack Kaelbling, and Tom\u00e1s Lozano-P\u00e9rez. Provably safe robot navigation with\n\nobstacle uncertainty. In RSS, 2017. 2\n\n[4] Sean L Bowman, Nikolay Atanasov, Kostas Daniilidis, and George J Pappas. Probabilistic data association\n\nfor semantic slam. In ICRA, 2017. 2\n\n[5] Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, and Jan Kautz. MapNet: Geometry-aware\n\nlearning of maps for camera localization. In CVPR, 2018. 2\n\n[6] John Canny. The complexity of robot motion planning. MIT press, 1988. 2\n\n[7] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran\nSong, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from RGB-D data in indoor environments.\nIn 3DV, 2017. 5\n\n[8] Devendra Singh Chaplot, Emilio Parisotto, and Ruslan Salakhutdinov. Active neural localization. In ICLR,\n\n2018. 2\n\n[9] Lee Clement, Jonathan Kelly, and Timothy D Barfoot. Robust monocular visual teach and repeat aided by\n\nlocal ground planarity and color-constant imagery. JFR, 2017. 2\n\n[10] Mark Cummins and Paul Newman. FAB-MAP: Probabilistic localization and mapping in the space of\n\nappearance. IJRR, 2008. 2\n\n[11] Shreyansh Daftry, J Andrew Bagnell, and Martial Hebert. Learning transferable policies for monocular\n\nreactive mav control. In ISER, 2016. 3\n\n[12] Andrew J Davison and David W Murray. Mobile robot localisation using active vision. In ECCV, 1998. 2\n\n[13] Jakob Engel, Thomas Sch\u00f6ps, and Daniel Cremers. LSD-SLAM: Large-scale direct monocular SLAM. In\n\nECCV, 2014.\n\n[14] J. Fuentes-Pacheco, J. Ruiz-Ascencio, and J. M. Rend\u00f3n-Mancha. Visual simultaneous localization and\n\nmapping: a survey. Arti\ufb01cial Intelligence Review, 2015. 2\n\n[15] Paul Furgale and Timothy D Barfoot. Visual teach and repeat for long-range rover autonomy. JFR, 2010. 2\n\n[16] Dhiraj Gandhi, Lerrel Pinto, and Abhinav Gupta. Learning to \ufb02y by crashing. In IROS, 2017. 3\n\n[17] Alessandro Giusti, J\u00e9r\u00f4me Guzzi, Dan C Cire\u00b8san, Fang-Lin He, Juan P Rodr\u00edguez, Flavio Fontana, Matthias\nFaessler, Christian Forster, J\u00fcrgen Schmidhuber, Gianni Di Caro, et al. A machine learning approach to\nvisual perception of forest trails for mobile robots. RAL, 2016. 3\n\n[18] Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping\n\nand planning for visual navigation. In CVPR, 2017. 1, 3, 5\n\n[19] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli,\nJamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and Andrew Fitzgibbon. KinectFusion:\nreal-time 3D reconstruction and interaction using a moving depth camera. UIST, 2011. 2\n\n[20] Gregory Kahn, Adam Villa\ufb02or, Bosen Ding, Pieter Abbeel, and Sergey Levine. Self-supervised deep\nreinforcement learning with generalized computation graphs for robot navigation. arXiv preprint\narXiv:1709.10489, 2017. 3\n\n[21] Abhishek Kar, Christian H\u00e4ne, and Jitendra Malik. Learning a multi-view stereo machine. In NIPS, 2017.\n\n2\n\n[22] Lydia E Kavraki, Petr Svestka, J-C Latombe, and Mark H Overmars. Probabilistic roadmaps for path\n\nplanning in high-dimensional con\ufb01guration spaces. RA, 1996. 2\n\n9\n\n\f[23] Alex Kendall, Matthew Grimes, and Roberto Cipolla. Posenet: A convolutional network for real-time\n\n6-dof camera relocalization. In ICCV, 2015. 2\n\n[24] Arbaaz Khan, Clark Zhang, Nikolay Atanasov, Konstantinos Karydis, Vijay Kumar, and Daniel D Lee.\n\nMemory augmented control networks. In ICLR, 2018. 3\n\n[25] Steven M Lavalle and James J Kuffner Jr. Rapidly-exploring random trees: Progress and prospects. In\n\nAlgorithmic and Computational Robotics: New Directions, 2000. 2\n\n[26] David G Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004. 6\n\n[27] Piotr Mirowski, Matthew Koichi Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson,\nDenis Teplyashin, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, and Raia Hadsell. Learning\nto navigate in cities without a map. arXiv preprint arXiv:1804.00168, 2018. 1, 3\n\n[28] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil,\nRoss Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments.\nIn ICLR, 2017. 1, 3\n\n[29] David Nist\u00e9r, Oleg Naroditsky, and James Bergen. Visual odometry. In CVPR, 2004. 2\n\n[30] Emilio Parisotto and Ruslan Salakhutdinov. Neural Map: Structured memory for deep reinforcement\n\nlearning. In ICLR, 2018. 1, 3\n\n[31] Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan\nShelhamer, Jitendra Malik, Alexei A. Efros, and Trevor Darrell. Zero-shot visual imitation. In ICLR, 2018.\n3\n\n[32] Sudeep Pillai and John J Leonard. Towards visual ego-motion learning in robots. arXiv preprint\n\narXiv:1705.10279, 2017. 2\n\n[33] Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In NIPS, 1989. 3\n\n[34] Fereshteh Sadeghi and Sergey Levine. (CAD)2RL: Real singel-image \ufb02ight without a singel real image. In\n\nRSS, 2017. 3\n\n[35] Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for\n\nnavigation. In ICLR, 2018. 1, 3\n\n[36] Johannes L Sch\u00f6nberger, Marc Pollefeys, Andreas Geiger, and Torsten Sattler. Semantic visual localization.\n\nIn CVPR, 2018. 2\n\n[37] Johannes Lutz Sch\u00f6nberger and Jan-Michael Frahm. Structure-from-motion revisited. In CVPR, 2016. 6\n\n[38] Johannes Lutz Sch\u00f6nberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. Pixelwise view\n\nselection for unstructured multi-view stereo. In ECCV, 2016. 6\n\n[39] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic\n\nscene completion from a single depth image. In CVPR, 2018. 5\n\n[40] Tristan Swedish and Ramesh Raskar. Deep visual teach and repeat on path networks. In CVPR, 2018. 2\n\n[41] Jingwei Zhang, Lei Tai, Joschka Boedecker, Wolfram Burgard, and Ming Liu. Neural slam. arXiv preprint\n\narXiv:1706.09520, 2017. 3\n\n[42] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and\n\nego-motion from video. In CVPR, 2017. 2\n\n[43] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi.\n\nTarget-driven visual navigation in indoor scenes using deep reinforcement learning. In ICRA, 2017. 1\n\n10\n\n\f", "award": [], "sourceid": 431, "authors": [{"given_name": "Ashish", "family_name": "Kumar", "institution": "UC Berkeley"}, {"given_name": "Saurabh", "family_name": "Gupta", "institution": "UC Berkeley / FAIR / UIUC"}, {"given_name": "David", "family_name": "Fouhey", "institution": "UC Berkeley"}, {"given_name": "Sergey", "family_name": "Levine", "institution": "UC Berkeley"}, {"given_name": "Jitendra", "family_name": "Malik", "institution": "University of California at Berkley"}]}