{"title": "Geometry-Aware Recurrent Neural Networks for Active Visual Recognition", "book": "Advances in Neural Information Processing Systems", "page_first": 5081, "page_last": 5091, "abstract": "We present recurrent geometry-aware neural networks that integrate visual in-\nformation across multiple views of a scene into 3D latent feature tensors, while\nmaintaining an one-to-one mapping between 3D physical locations in the world\nscene and latent feature locations. Object detection, object segmentation, and 3D\nreconstruction is then carried out directly using the constructed 3D feature memory,\nas opposed to any of the input 2D images. The proposed models are equipped\nwith differentiable egomotion-aware feature warping and (learned) depth-aware\nunprojection operations to achieve geometrically consistent mapping between the\nfeatures in the input frame and the constructed latent model of the scene. We\nempirically show the proposed model generalizes much better than geometry-\nunaware LSTM/GRU networks, especially under the presence of multiple objects\nand cross-object occlusions. Combined with active view selection policies, our\nmodel learns to select informative viewpoints to integrate information from by\n\u201cundoing\" cross-object occlusions, seamlessly combining geometry with learning\nfrom experience.", "full_text": "Geometry-Aware Recurrent Neural Networks for\n\nActive Visual Recognition\n\nRicson Cheng\u2217, Ziyan Wang\u2217, Katerina Fragkiadaki\n\n{ricsonc,ziyanw1}@andrew.cmu.edu, katef@cs.cmu.edu\n\nCarnegie Mellon University\n\nPittsburgh, PA 15213\n\nAbstract\n\nWe present recurrent geometry-aware neural networks that integrate visual in-\nformation across multiple views of a scene into 3D latent feature tensors, while\nmaintaining an one-to-one mapping between 3D physical locations in the world\nscene and latent feature locations. Object detection, object segmentation, and 3D\nreconstruction is then carried out directly using the constructed 3D feature memory,\nas opposed to any of the input 2D images. The proposed models are equipped\nwith differentiable egomotion-aware feature warping and (learned) depth-aware\nunprojection operations to achieve geometrically consistent mapping between the\nfeatures in the input frame and the constructed latent model of the scene. We\nempirically show the proposed model generalizes much better than geometry-\nunaware LSTM/GRU networks, especially under the presence of multiple objects\nand cross-object occlusions. Combined with active view selection policies, our\nmodel learns to select informative viewpoints to integrate information from by\n\u201cundoing\" cross-object occlusions, seamlessly combining geometry with learning\nfrom experience.\n\n1\n\nIntroduction\n\nCross-object occlusions remain an important source of failures for current state-of-the-art object\ndetectors [29], which, despite their formidable performance increase in recent years, still carry the\nbiases and idiosyncrasies of the data they were trained on [16]: static images from Imagenet and\nCOCO datasets. Thus, detectors do well on viewpoints that human photographers tend to prefer, but\nmay fail on images (non skillfully) captured by robotic agents, that often feature severe cross-object\nocclusions and viewpoints which are unusual for humans. Robotic agents on the other hand do not\nabide by the constraints of passive vision: they can actively choose where to look in order to recover\nfrom occlusions and dif\ufb01cult viewpoints.\nActively selecting camera views for \u201cundoing\" occlusions and recovering missing information has\nbeen identi\ufb01ed as an important \ufb01eld of research since as early as 1980\u2019s, under the name active\nvision [1].\nIt has been argued that without such active view selection, no formal performance\nguarantees can be provided for visual recognition algorithms [32]. Yet, 1980\u2019s active vision was\nnot equipped with deep neural detectors, memory modules, or view selection policies, and often\nattempted tasks and imagery that would appear elementary with current detectors, even from a single\ncamera view. There has been recent methods that have attempted to resurrect the active vision\nparadigm [23, 9, 4, 18]. Paradoxically, they either focus on sequential processing of selected visual\nglimpses from a passive static image [23, 9, 4], and thus they are not able to \u201cundo\" occlusions, or\nthey consider 3D reconstruction of isolated objects [18, 43, 20], while one can argue that isolating\nobjects from clutter is precisely the premise of having an active camera.\n\n\u2217Indicates equal contribution\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fWe propose geometry-aware recurrent networks and loss functions for active object detection,\nsegmentation and 3D reconstruction in cluttered scenes, following the old active vision premise.\nOur main contribution is a network architecture that accumulates information across camera views into\na 3D feature map of the scene, in a geometrically consistent manner, so that information regarding\nthe same 3D physical point across views is placed nearby in the tensor. Object segmentation,\nclassi\ufb01cation and 3D reconstruction is directly solved for from the output of the 3D latent\nfeature tensor, as opposed to in 2D input images. This is drastically different from the paradigm\nof tracking by detection [40] where we need to commit to object detections in each image and\npost-correspond them in time, and may suffer from early erroneous decisions. Our model instead\naccumulates visual information across multiple views in a space-aware latent space, before detecting\nthe objects, and thus detection performance strictly improves with more views. Projecting the detected\n3D object segments generates amodal boxes and segmentations in the (potentially heavily) occluded\ninput 2D views.\nIn a nutshell, our method works as follows. At each time step, an active observer selects a nearby\ncamera viewpoint. Our model then \u201cunprojects\" RGB, (learned) depth and object foreground mask\ninto a 3D feature tensor, which is rotated to match the pose of the \ufb01rst camera view, using the relative\negomotion, which is assumed known. The warped feature tensor then updates a 3D latent state map of\na convolutional Gated Recurrent Unit (GRU) layer. Geometric consistency ensures that information\nfrom 2D projections of the same 3D physical point are placed nearby in the 3D feature memory tensor,\na key for generalization. GRU convolutional \ufb01lters can then \u201csee\" information regarding the same 3D\nconcept in their vicinity. They transform it into higher level features optimized for object instance\nsegmentation, classi\ufb01cation and voxel occupancy prediction, directly in 3D. Our geometry-aware\nrecurrent network is trained end-to-end in a differentiable manner, and our active view selection is\ntrained with reinforcement rewarded by state estimation accuracy at each time step.\nWe empirically show the proposed 3D recurrent network outperforms by a large margin 2D recurrent\nnetwork alternatives, convolutional or not, which do not effectively generalize in cluttered scenes due\nto the combinatorial explosion of occlusion con\ufb01gurations. Our view selection policy outperforms\nnaive or random view selection alternatives. The proposed method marries pure geometric localization\nand 3D pointcloud mapping [31] with feature learning from experience, while being robust to mistakes\nof depth and egomotion estimation. In summary our contributions are:\n\n\u2022 Introducing the problem of active state estimation, namely, selecting camera views for jointly\noptimizing object instance segmentation, classi\ufb01cation and 3D reconstruction, a problem\nvery relevant for robotic computer vision.\n\u2022 Proposing a geometry-aware recurrent neural network that accumulates feature information\ndirectly in 3D. The proposed model can be used for both passive as well as actively collected\nvideo sequences.\n\u2022 Conducting extensive experiments and ablations against alternative recurrent architectures\n\nin both \u201ceasy\" and \u201chard\" environments.\n\nOur codes will be made available at ricsonc.github.io/geometryaware.\n\n2 Related Work\n\nActive vision An active vision system is one that can manipulate the viewpoint of the camera(s) in\norder to explore the environment and get better information from it [3, 39]. Active visual recognition\nattracted attention as early as 1980\u2019s. Aloimonos et al. [1] suggested many problems such as shape\nfrom contour and structure from motion which, while underconstrained for a passive observer, can\neasily be solved by an active one [7, 30]. Active control of the camera viewpoint also helps in focusing\ncomputational resources on the relevant elements of the scene [35]. The importance of active data\ncollection for sensorimotor coordination has been highlighted in the famous experiment of Held\nand Hein in 1963 [14], involving two kittens engaged soon after birth to move and see in a carousel\napparatus. The active kitten that could move freely was able to develop healthy visual perception, but\nthe passively moved around kitten suffered signi\ufb01cant de\ufb01cits. Despite its importance, active vision\nwas put aside potentially due to the slow development of autonomous robotics agents, necessary\nfor the realization of its full potential. Recently, the active vision paradigm has been resurrected by\napproaches that perform selective attention [28] and selective sequential image glimpse processing in\n\n2\n\n\forder to accelerate object localization and recognition in 2D static images [23, 9, 4]. Fewer methods\nhave addressed camera view selection for a moving observer [2, 45, 22], e.g. for object detection\n[17], for object recognition [19], for navigation [24, 11] or for object 3d reconstruction, in synthetic\nenvironments [18]. Since we cannot backpropagate through viewpoint selection, reinforcement\nlearning methods have often been used to train view or glimpse selection policies. Many of the\naforementioned approaches use some type of recurrent network [23, 9, 4, 25, 20] that learns to\nintegrate information across views or glimpses. Our work proposes a particular architecture for such\nrecurrent network that takes into account the egomotion of the observer.\n\nDeep geometry There has recently been great interest in integrating learning and geometry for\nsingle view 3D object reconstruction [36, 41], depth and egomotion estimation from pairs of frames\n[38, 44], depth estimation from stereo images [8], estimating 3D locations of human keypoints from\n2D keypoint heatmaps [42, 37]. Many of those works use neural network architectures equipped\nwith some form of differentiable camera projection, so that 3D desired estimates can be supervised\ndirectly using 2D quantities. They do not accumulate information across multiple camera viewpoints.\nFor example, Tulsiani et al. [36], Wu et al. [41] and Zhou et al.[44] use a single image frame as input\nto predict 3D reconstruction of a single object, or a 2D depth map of the entire scene, though they\nuse multiple views to obtain supervision in the form of depth re-projection error. Works that use\nreprojection error as self-supervision at test-time [42, 37, 38, 44, 36, 41] cannot operate in an online,\ninteractive mode since backpropagating error gradients takes time. Learnt stereo machines (LSM) [20]\nintegrates RGB information along sequences of random camera viewpoints in a 3D feature memory\ntensor, in an egomotion stabilized way, similar to our method. We compare our model to LSM [20]\nin Section 4 and show that using learning-based depth estimation and early feature unprojection\ngeneralizes better in the presence of multiple objects that depth-unaware late unprojection, used in\nLSM [20]. 3D ShapeNets [43] uses a deep generative model that given a depth map as input predicts\na 3D occupancy voxel grid, and has an unprojection operation similar to the one we propose. It is\ntrained using contrastive divergence while our model is a simple recurrent discriminative network with\nstraightforward training. Both LSM [20] and 3D ShapeNets [43] consider the problem of single object\n3D reconstruction. Our work instead considers segmentation, classi\ufb01cation and 3D reconstruction of\nmultiple objects in clutter. Pointnets [27] are neural network architectures that operate directly on\n3D pointcloud input, as opposed to pixel grids. They do not accumulate information across multiple\nviews.\nPurely geometric approaches, such as Simultaneous Localization and Mapping methods [31, 21]\nbuild a 3D map of the scene while estimating the motion of the observer. Our method builds a feature\nmap instead, which captures both the geometry and the semantics of the scene. Each point of the map\nis a feature vector as opposed to an occupancy label. SLAM method need to see the scene from all\nviewpoints in order to reconstruct it, they cannot map the invisible. Our model instead learns to \ufb01ll in\nmissing views from experience.\nMapNet [15], IQA [10] and Neural Map [26] construct 2D birdview maps of the scene by taking\ninto account the egomotion of the observer, similar to our method. MapNet further estimates the\negomotion, as opposed to assuming it known as the rest of the methods, including ours. They consider\nthe tasks of navigation [15, 26] and interactive question answering [10], while we consider the task\nof object detection, segmentation and classi\ufb01cation. In IQA, objects are detected in each frame and\ndetections are aggregated in the birdview map, whereas we detect objects using the feature map\ndirectly as input. If an object is too heavily occluded to be reliably detected in any single view, it can\nstill be inferred given the integrated information across all views.\n\n3 Active vision with geometry-aware recurrent neural networks\n\nWe consider an active observer, e.g., a robotic agent, equipped with a perspective camera model. At\neach timestep, the agent selects a nearby camera viewpoint. The RGB, depth and foreground mask of\nthe selected view are then \u201cunprojected\" into a 3D tensor and transformed in order to compensate for\nthe egomotion of the observer, namely, the translation and rotation between the current and previous\nviews. The oriented tensor is then fed to a recurrent neural network that builds a latent model of the\nscene, as illustrated in Figure 1. We describe our geometry-aware recurrent model and viewpoint\nselection policy right below.\n\n3\n\n\fFigure 1: Active vision with geometry-aware recurrent neural networks. Our active agent se-\nquentially accumulates visual information in a 3D geometrically-consistent feature map of the visual\nscene. At each frame, the agent selects a nearby camera view, conditioned on the 3D feature map\nof the scene thus far, and the current RGB image. It predicts a corresponding 2D depth map and\nforeground object mask. RGB, depth and segmentation maps are unprojected in corresponding 3D\nfeature tensors, mapped using the relative egomotion to the coordinate frame of the cumulative 3D\nfeature map built thus far, and update a 3D convolutional GRU memory. We visualize 3 channels of\nthe memory. The output of the 3D GRU memory is then mapped via a 3D encoder-decoder network\nto segmentation embeddings, object category probabilities, and occupancy for every voxel in the 3D\ngrid.\n\n3.1 Geometry-aware recurrent neural network\n\nOur memory model is a recurrent neural network whose latent state corresponds to a 3D feature map\nof the visual scene. The memory map is updated with each new camera view in a geometrically\nconsistent manner, so that information from 2D pixel projections that correspond to the same\n3D physical point end up nearby in the tensor. This permits convolutional operations to have a\ncorrespondent input across views, as opposed to varying with the motion of the observer.\n\nUnprojection In an RGB image, we predict the corresponding 2D depth and object foreground\nmap using a 2D convolutional encoder-decoder network with skip-connections. We then unproject\nthose to \ufb01ll in an initial 3D feature tensor as follows:\nFor each feature voxel Vi,j,k in the 3D tensor we compute the pixel location (x, y) which it projects\nonto, from the current camera view. Then, Vi,j,k is \ufb01lled with the bilinearly interpolated RGB, depth\nand mask at (x, y). All voxels lying along the same ray shot from the camera will be \ufb01lled with\nnearly the same RGB, depth and mask, although in practice, since the centroids of two voxels are\nrarely on the exact same camera ray, the features will differ slightly.\nWe further compute (i) a grid of depth values, so that Vi,j,k = k and (ii) a binary voxel occupancy\ngrid that contains the thin shell of voxels directly visible from the current camera view. We compute\nthis by \ufb01lling all voxels whose unprojected depth value equals the grid depth value. Thus, our\nunprojection operation results in an initial feature tensor M64\u00d764\u00d764\u00d77: 3 channels for RGB, 2\nchannels for depth and mask, 1 channel for the surface occupancy, and 1 channel for the grid of depth\nvalues. We visualize the unprojection operation for one horizontal slice and one channel of the 3D\nfeature tensor in Figure 1.\n\nRecurrent memory update\nIn order for the 2D projections of the same physical point to end\nup closeby in our 3D memory tensor, in each view, we rotate unprojected feature tensors to the\norientation of the \ufb01rst camera view. This requires only relative camera pose (egomotion). We will\nassume ground-truth egomotion information throughout the paper since an active observer does have\naccess to its approximate egomotion, and leave egomotion estimation as future work. In Section 4,\nwe quantify how robust our recurrent model is to inaccurate egomotion.\n\n4\n\n\fOnce the feature tensor has been properly oriented, we feed it as input to a 3D convolutional Gated\nRecurrent Unit [6] layer. The operation of the GRU memory layer update can be written as:\n\nht+1 = ut \u25e6 ht + (1 \u2212 ut) \u25e6 tanh(conv3d([xt, rt \u25e6 ht], Wh))\nut, rt = \u03c3(conv3d([xt, ht], [Wu, Wr]))\n\n(1)\n(2)\nwhere ut and rt are the GRU update gate and reset gate respectively. We experimented with max and\naverage pooling operations for such memory update, but found that the GRU memory update worked\nbest.\n\nObject reconstruction, instance segmentation and classi\ufb01cation Our model is trained in a su-\npervised manner to predict object instance segmentation, object classi\ufb01cation and 3D object recon-\nstruction (voxel occupancy) using 3D groundtruth object bounding boxes, masks, and occupancy\nvoxel grids available in simulator environments.\nThe output of the 3D GRU memory is fed into a 3D convolutional encoder/decoder network with\nskip-connections to produce the \ufb01nal set of outputs: (i) a 3D sigmoid output which predicts voxel\noccupancy, (ii) a 3D segmentation embedding feature tensor, and (iii) a multiclass softmax output at\nevery voxel for category labelling. The segmentation embeddings and classi\ufb01cation outputs have half\nthe resolution of the occupancy grid.\nWe train for 3D reconstruction (voxel occupancy) using a standard binary cross-entropy loss. We train\nfor object instance segmentation by learning voxel segmentation embeddings [13]. Clustering using\nthe learnt embedding distances provides a set of voxel clusters, ideally, each one capturing an object\nin the scene. We use metric learning and a standard contrastive loss [12], that brings segmentation\nembeddings of voxels of the same object instance close, and pushes segmentation embeddings of\ndifferent object instances (and the empty space) apart. During training, we sample the same number\nof voxel examples from each object instance to keep the loss balanced, not proportional to the volume\nof each object. We convert the voxel feature embeddings into a voxel segmentation map as follows:\nWe discard voxel whose predicted occupancy is less than 0.5. We cluster the remaining voxels using\nk-means, with k set to the number of objects in the scene. When the number of objects is unknown,\nwe oversegment the scene using a large k, as shown in Figure 2. We then iteratively merge clusters\nwhose ratio of the number of adjacent voxels to the volume of the larger of the two clusters, R is\nthe largest. We stop merging when R < 1.5. In our scenario, such clustering is much facilitated by\nthe fact that objects live in 3D and free space effectively separates the different objects, preventing\nclusters to \u201cleak\" across different object instances.\nFor each voxel cluster, we compute the average of the logits predicted for the voxels in that group,\nand use that as the predicted class of the object. We train with cross-entropy loss between the average\nlogit and the ground truth class label of the (groundtruth) shape in the scene which has the highest\nIntersection over Union (IoU) with each voxel cluster.\n\n3.2 Learning a view selection policy\n\nWe train a view selection policy to predict at each timestep a distribution over eight adjacent camera\nviews in the neighborhood of the current one. Our policy network is a convolutional neural network\nwith two branches, one 2D and one 3D. The input to the 3D branch is the output of the GRU memory\nlayer, and the input to the 2D branch is the current RGB image. They are both mapped, using 3D\nand 2D convolutions respectively, to a \ufb01nal 1D vector after linearization and concatenation. A fully\nconnected layer with softmax activation produces the \ufb01nal categorical distribution over 8 possible\ndirections. Description of the detailed architecture of the view selection policy network is included in\nthe Appendix.\n\nFigure 2: Instance segmentation The left image in each of the 3 pairs shows the oversegmented\nscene using k = 8. We iteratively merge the clusters as described in the text to obtain the \ufb01nal\nsegmentation result.\n\n5\n\n\fWe train our policy network with REINFORCE [34] with a Monte Carlo baseline. We tried more\nadvanced Actor-Critic methods, but we did not \ufb01nd a difference empirically. The policy \u03c0(at|st, \u03b8)\nis iteratively updated with gradient ascent to maximize the expectation of total reward. The policy\ngradient is g(s) = Et[\u2207\u03b8 log(\u03c0\u03b8(at|st, \u03b8))Rt] where Rt is the reward of step t. Et[\u00b7] is the empirical\nexpectation of the average over a \ufb01nite number of sampled time steps. Though our agent solves\nfor object segmentation, classi\ufb01cation and 3D reconstruction, we found it adequate to provide\nreconstruction-driven rewards Rt, as the increase in Intesection over Union (IoU) of the discretized\nvoxel occupancy from each view to the next.\n\n4 Experiments\n\nWe test our models in two types of simulated environments: i) scenes we create using synthetic 3D\nobject models from ShapeNet [5]. The generated scenes contain single or multiple objects on a table\nsurface. ii) scenes from the SUNCG [33] dataset, sampled around an object of interest. Each 3D\nscene is rendered from a viewing sphere which has 3 \u00d7 18 possible views with 3 camera elevations\n(20\u25e6, 40\u25e6, 60\u25e6) and 18 azimuths (0\u25e6, 20\u25e6, . . . , 340\u25e6). The agent is allowed to move to one of the 8\nadjacent views at every time step. We voxelize each 3D scene at a resolution of 64 \u00d7 64 \u00d7 64.\nWe pretrain our geometry-aware recurrent neural network using randomly selected view sequences\nso that it does not over\ufb01t to the biases of the view selection policy. We then train our view selection\npolicy while the weights of the recurrent neural network are kept \ufb01xed.\n\nMulti-view reconstruction of single objects We compare the performance of the geometry-aware\nrecurrent network against alternatives on the task of 3D reconstruction of single objects using\nrandomly selected camera views (no active vision). We consider three object categories from\nShapeNet: chairs, cars, and airplanes. We split the data into training, validation, and test according to\n[5]. We train a single 3d reconstruction model for all categories using supervision from groundtruth\n3D voxel occupancy. We compare the proposed geometry-aware recurrent net ours against the\nfollowing baselines:\n\n1. 1D-LSTM: an LSTM network, similar to the one used in [41, 18]. In each frame, given RGB\nand depth as input, it produces through a 2D convolutional encoder a feature embedding\nvector by \ufb02attening the last 2D convolutional feature map. The embedding vector is fed into\na (fully-connected) LSTM layer and the output of the LSTM is decoded into a 3D occupancy\ngrid through a series of 3D convolutions. We detail its architecture in the Appendix.\n\n2. Learnt Stereo Machines (LSM) of [20]. This model aggregates information using a 3D GRU\nlayer, similar to ours, by unprojecting convolutional features extracted from RGB images\nusing absolute groundtruth camera poses. Absolute pose may be hard to obtain and it may\nnot be meaningful in the presence of multiple objects, our method only requires relative pose.\nThey use a late unprojection as opposed to our proposed early depth-aware unprojection.\nWe use the publicly available code of [20] and retrain their model for our setup.\n\n3. LSMdepth. An LSM alternative that uses depth as an additional input channel and performs\n\nlate unprojection on features extracted from the concatenation of depth and RGB.\n\nAll methods are trained and tested on sequences of randomly selected nearby camera views.\nWe report Intersection over Union (IoU) between the predicted and groundtruth occupancy grids in\neach of four camera views for our model and baselines in the test set in Table 1. More visualization\nresults are included in the supplementary material. The proposed geometry-aware recurrent neural\nnetwork outperforms the baselines. The LSM outperforms our method on the \ufb01rst input view.\nHowever, after aggregating information from more views, our method consistently outperforms all\nalternatives.\nFor SUNCG, we similarly test our proposed model on 3D reconstruction of (usually single) objects\nusing randomly selected camera views as input. Intersection over Union (IoU) between the predicted\nand groundtruth voxel occupancy grids in each of four consecutive camera views for our model and\nLSTM baseline are shown in Table 2. Qualitative results are included in the Appendix.\n\nActive view selection We evaluate the performance of our active view selection policy on 3D\nreconstruction of single and multiple objects. At each frame, our active observer learns to select an\n\n6\n\n\f1D-LSTM\n\nLSM\n\nLSM+gt depth\nours+gt depth\n\nview-1\n0.57\n0.63\n0.65\n0.55\n\nview-2\n0.59\n0.66\n0.68\n0.69\n\nsingle object\n\nmulti-objects\n\nview-3\n0.60\n0.68\n0.69\n0.72\n\n-\n\nview-4\n0.60\n0.69\n0.70\n0.73\n\nview-1\n0.11\n0.43\n0.48\n0.47\n0.45\n\nview-2\n0.15\n0.47\n0.51\n0.58\n0.56\n\nview-3\n0.17\n0.51\n0.54\n0.62\n0.60\n\nview-4\n0.20\n0.53\n0.56\n0.64\n0.62\n\n-\n\nours+learnt depth\nTable 1: 3D voxel occupancy prediction using randomly selected camera views. We show Inter-\nsection over Union (IoU) between the prediction and groundtruth 3D voxel grids. The performance\nof our model improves with more views. Using estimated depth as input does slightly worse than\nusing groundtruth depth.\n\n-\n\n-\n\nview-1\n1D-LSTM 0.12\n0.24\n\nours\n\nview-2\n0.16\n0.28\n\nview-3\n0.16\n0.31\n\nview-4\n0.18\n0.32\n\n3D voxel occupancy prediction using randomly selected camera views. We show\nTable 2:\nIntersection over Union (IoU) between the predicted and groundtruth 3D voxel occupancies on\nSUNCG scenes.\n\nadjacent \u2014as opposed to random [18]\u2014 view to jump to, in an attempt to mimic realistic deployment\nconditions on a robotic agent. We compare our learned view selection policy with four baselines:\n\n1. random, which randomly selects neighboring views to visit, and\n2. oneway, which moves the camera along the same direction at each timestep so that the agent\n\nmaximizes the horizontal or vertical baseline.\n\n3. 1-step greedy, similar to the one used in [43], which always selects the next view by\nmaximizing the immediate increase in IoU for a single step as opposed to a \ufb01nite horizon.\n4. oracle, which selects the best among all possible view trajectories through exhaustive search.\nSince it is too expensive to try out all possible four-view-long trajectories, we sample 100\nfour-view-long trajectories and take the best as the oracle trajectory. This serves as an\nupper-bound for our view selection policy.\n\nWe show qualitative results in Figure 3. For quantitative results on random and oneway policy, please\nsee supplementary materials. The learned view selection outperforms the baselines for both single\nand multiple objects present. It better reconstructs detailed object parts, like the arm of a chair or\nthe inside part of a mug. Compared to the oneway policy, the active policy select views that \u201cundo\"\nocclusions and recover most of the object.\nWe trained our policy using three random seeds and report the mean and variance.\nThe percent increase in IoU between predicted and groundtruth voxel occupanices over single-view\npredictions is reported in Table 3. We can see that our learned policy is worse than 1-step greedy\npolicy in the \ufb01rst view, but outperforms it as more views become available. This is not surprising,\nas our learned policy was trained to select views that maximize the cumulative future rewards over\na time horizon of four views. Qualitative results of the view selection policy are included in the\nsupplementary materials.\n\nMulti-view, multi-object segmentation, classi\ufb01cation and 3D reconstruction No previous work\nhas reported results on the task of object detection, segmentation and 3D reconstruction under the\npresence of multiple objects occluding each other. An important contribution of this paper is to\naddress joint object instance segmentation, object category prediction and 3D reconstruction while\naccumulating information across multiple views in order to reveal occluded objects. We choose\nobjects from four different categories, helmet, mug, bowl, and camera, that are similar in size. We\nplace two objects in each scene between 0.25 and 0.35 units from the origin. The line passing through\nthe centroid of both objects also goes through the origin. The scene is randomly rotated between\n\u221290\u25e6 and 90\u25e6 degrees. Although we only generated 332 scenes, our network is able to perform and\ngeneralize well. We uses 70% of the scenes for training and the rest for testing.\n\n7\n\n\ft\nu\np\nn\nI\n\nD\n2\n\nM\nT\nS\nL\n\nh\nt\np\ne\nd\nM\nS\nL\n\ns\nr\nu\nO\n\nt\nu\np\nn\nI\n\ne\nv\ni\nt\nc\na\n\ne\nv\ni\nt\nc\na\n\ns\nr\nu\nO\n\nt\ng\n\nFigure 3: 3D reconstruction results on single and multiple objects using different methods. Note\nthe ability of our model to model the thin armrest of the chair without blending it into the body.\n\nt\nu\np\nn\nI\n\ns\nr\nu\nO\n\nt\ng\n\nFigure 4: 3D segmentation on multi-object scenes.\n\nWe retrain the model of [20] using our proposed state estimation loss functions and with absolute\ngroundtruth camera poses as a comparison. We report multi-object 3D reconstruction (voxel occu-\npancy prediction) in Table 1 and classi\ufb01cation and segmentation in Table 4. For 3D reconstruction,\nlearned depth always results in lower reconstruction IoU in each view. The proposed architecture\noutperforms LSM, and depth input always helps, even if it is estimated by a network as opposed to\ngroundtruth. This result is encouraging given the improvement of depth sensors currently available\nthat the proposed geometry-aware architecture can maximally take advantage of. For object segmen-\ntation, we compute the 3D IoU using the optimal matching between the segmented 3D voxel groups\nrecovered from k-means and the groundtruth 3D object voxel grids. We report the average IoU across\nall objects in a scene. We also report classi\ufb01cation accuracy. Table 4 shows the quantitative results,\nand Figure 4 and 5 shows visualizations of the instance segmentation and classi\ufb01cation.\n\nview-1\n\nview-2\n\nsingle object\n\nview-3\n\n32.7%\n\n38.1%\nmultiple objects\n40.4%\n\n42.6%\n\n29.1%\n\n32.7%\n\n36.2%\n\n29.8%\n\nview-4\n\n33.3%\n\n38.1%\n\n40.4%\n\n44.7%\n\n1-step greedy\n\nActive\nOracle\n\n0%\n0%\n0%\n\n1-step greedy\n\nActive\nOracle\n\n0%\n0%\n0%\n\n27.7%\u00b13.23% 27.7%\u00b13.23% 36.6%\u00b11.17%\n\n31.3%\u00b12.26% 39.0%\u00b11.99% 41.8%\u00b12.36%\n\nTable 3:\nPercent increase in Intersection over Union (IoU) between 3D reconstructions and\ngroundtruth over the single-view reconstruction. The \ufb01rst view is the same for all policies, so\nthe IoU for the \ufb01rst view is the same.\n\n8\n\n\f3D voxel occupancy IoU\n\n3D segmentation IoU\nClassi\ufb01cation accuracy\n\nview-1\n0.54\n0.60\n0.56\n\nview-2\n0.63\n0.69\n0.83\n\nview-3\n0.64\n0.70\n0.83\n\nview-4\n0.65\n0.71\n0.83\n\nTable 4: 3D object segmentation, classi\ufb01cation and 3D reconstruction using two objects per\nscene and four object categories. As expected, both reconstruction and segmentation IoU increase\nwith more views. The four object categories are roughly balanced and always guessing the most\ncommon category yields 32% accuracy.\n\nFigure 5: Object segmentation under occlusions. We are able to project the segmented 3D voxels\nand obtain amodal bounding boxes and segmentation masks, even in the presence of heavy occlusions.\nWe show segmentations speci\ufb01cally from the most heavily occluded camera views.\n\nSensitivity to egomotion estimation errors Our reconstruction method uses the relative camera\npose between views in order to align the unprojected feature tensors, before feeding them to a 3D\nGRU layer. However, there may be egomotion estimation errors in a realistic scenario. We test the\nrobustness of our model to such errors by adding random noise to azimuth and elevation. We consider\nnoise distributed normally with \u00b5 = 0 and \u03c3 = 5 degrees for both the azimuth and elevation. We\ntest noise which is independent for each view, and also noise that accumulates across views, so that\nthe total error at view i is the error at view i \u2212 1 plus the additional error at view i. We test two\nvariants of our method: one trained without noisy egomotion, and one trained with (independent)\nnoisy egomotion. We show quantitative results in Table 5.\n\nTrained without noise Trained with noise\n\ngroundtruth egomotion\n\nIndependent egomotion noise\nAccumulating egomotion noise\n\n0.64\n0.56\n0.56\n\n0.63\n0.59\n0.58\n\nTable 5: 3D reconstruction under noisy egomotion for the chair category. We show IoU for the\ndifferent test-time scenarios listed in the left column. The performance of our model degrades\ngracefully with noise in egomotion estimation. Moreover, when trained with such noise, it become\nmore resistant to such errors.\n\n5 Discussion - Future Work\n\nWe presented a method for active object detection, segmentation and 3D reconstruction in cluttered\nstatic scenes, that selects camera viewpoints and integrates information into a 3D feature memory map,\nusing geometry-aware recurrent neural updates. Object detection, segmentation and 3D reconstruction\nare directly predicted from the constructed 3D feature map, as opposed to any single 2D view. As\na result, object delineation seamlessly improves with more views, with no need to commit to early\nerroneous object detections, which are hard to correct once made. We showed the proposed model\ngeneralizes better than geometry-unaware LSTMs/GRUs, especially in cluttered scenes. The main\ncontributing factor for the effectiveness of the proposed method is the representation: 3D space does\nnot suffer from instantaneous occlusions or foreshortenings. It is a stable space for information\nintegration.\nThe current framework has the following limitations: (i) it assumes ground-truth egomotion, (ii)\nit consumes lots of GPU memory for maintaining the full scene tensor, (iii) it cannot handle inde-\npendently moving objects, and (iv) it requires 3D groundtruth for object detection, which is very\nexpensive to obtain. Addressing the above limitations are interesting directions for future work.\n\n9\n\n\fReferences\n[1] J. Aloimonos, I. Weiss, and A. Bandyopadhyay. Active vision. In IJCV, 1988.\n[2] P. Ammirato, P. Poirson, E. Park, J. Kosecka, and A. C. Berg. A dataset for developing and\n\nbenchmarking active vision. In ICRA, 2017.\n\n[3] R. Bajcsy. Active perception. In in Proceedings of the IEEE, 1988.\n[4] J. Caicedo and S. Lazebnik. Active object localization with deep reinforcement learning. In\n\nICCV, 2015.\n\n[5] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva,\nS. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information-Rich 3D Model Repository.\nTechnical Report arXiv:1512.03012 [cs.GR], Stanford University \u2014 Princeton University \u2014\nToyota Technological Institute at Chicago, 2015.\n\n[6] K. Cho, B. van Merrienboer, \u00c7. G\u00fcl\u00e7ehre, F. Bougares, H. Schwenk, and Y. Bengio. Learning\nphrase representations using RNN encoder-decoder for statistical machine translation. CoRR,\nabs/1406.1078, 2014.\n\n[7] J. Denzler, M. Zobel, and H. Niemann. Information theoretic focal length selection for real-time\nactive 3d object tracking. In Proceedings Ninth IEEE International Conference on Computer\nVision, pages 400\u2013407 vol.1, Oct 2003.\n\n[8] C. Godard, O. Mac Aodha, and G. J. Brostow. Unsupervised monocular depth estimation with\n\nleft-right consistency. CoRR, abs/1609.03677, 2016.\n\n[9] A. Gonzalez-Garcia, A. Vezhnevets, and V. Ferrari. An active search strategy for ef\ufb01cient\nobject class detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern\nRecognition, pages 3022\u20133031, 2015.\n\n[10] D. Gordon, A. Kembhavi, M. Rastegari, J. Redmon, D. Fox, and A. Farhadi. IQA: visual\n\nquestion answering in interactive environments. CoRR, abs/1712.03316, 2017.\n\n[11] S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik. Cognitive mapping and planning\nfor visual navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern\nRecognition, pages 2616\u20132625, 2017.\n\n[12] R. Hadsell, S. Chopra, and Y. Lecun. Dimensionality reduction by learning an invariant mapping.\n\nIn CVPR, 2006.\n\n[13] A. W. Harley, K. G. Derpanis, and I. Kokkinos. Learning dense convolutional embeddings for\n\nsemantic segmentation. CoRR, abs/1511.04377, 2015.\n\n[14] R. Held and A. Hein. Movement-produced stimulation in the development of visually guided\n\nbehavior. In Journal of Comparative and Physiological Psychology, 1963.\n\n[15] J. F. Henriques and A. Vedaldi. Mapnet: An allocentric spatial memory for mapping environ-\nments. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,\n2018.\n\n[16] J. Hoffman, E. Tzeng, T. Park, J. Zhu, P. Isola, K. Saenko, A. A. Efros, and T. Darrell. Cycada:\n\nCycle-consistent adversarial domain adaptation. CoRR, abs/1711.03213, 2017.\n\n[17] D. Jayaraman and K. Grauman. Look-ahead before you leap: end-to-end active recognition by\nforecasting the effect of motion. In European Conference on Computer Vision, pages 489\u2013505.\nSpringer, 2016.\n\n[18] D. Jayaraman and K. Grauman. Learning to look around. CoRR, abs/1709.00507, 2017.\n[19] E. Johns, S. Leutenegger, and A. J. Davison. Pairwise decomposition of image sequences for\nactive multi-view recognition. In Proceedings of the IEEE Conference on Computer Vision and\nPattern Recognition, pages 3813\u20133822, 2016.\n\n[20] A. Kar, C. H\u00e4ne, and J. Malik. Learning a multi-view stereo machine. CoRR, abs/1708.05375,\n\n2017.\n\nIn CVPR, 2016.\n\n[21] C. Kerl, J. Sturm, and D. Cremers. Dense visual SLAM for RGB-D cameras. In IROS, 2013.\n[22] M. Malmir, K. Sikka, D. Forster, J. R. Movellan, and G. Cottrell. Deep q-learning for active\n\nrecognition of germs. In BMVC, 2015.\n\n[23] S. Mathe, A. Pirinen, and C. Sminchisescu. Reinforcement learning for visual object detection.\n\n[24] P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. J. Ballard, A. Banino, M. Denil, R. Goroshin,\nL. Sifre, K. Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint\narXiv:1611.03673, 2016.\n\n[25] V. Mnih, N. Heess, A. Graves, and K. Kavukcuoglu. Recurrent models of visual attention.\n\nCoRR, abs/1406.6247, 2014.\n\n10\n\n\f[26] E. Parisotto and R. Salakhutdinov. Neural map: Structured memory for deep reinforcement\n\nlearning. CoRR, abs/1702.08360, 2017.\n\n[27] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d\n\nclassi\ufb01cation and segmentation. CoRR, abs/1612.00593, 2016.\n\n[28] M. Ranzato. On learning where to look. In arXiv preprint arXiv:1405.5488, 2014.\n[29] S. Ren, K. He, R. B. Girshick, and J. Sun. Faster R-CNN: towards real-time object detection\n\nwith region proposal networks. CoRR, abs/1506.01497, 2015.\n\n[30] E. Rivlin and H. Rotstein. Control of a camera for active vision: Foveal vision, smooth tracking\n\n[31] T. Sch\u00f6ps, J. Engel, and D. Cremers. Semi-dense visual odometry for AR on a smartphone. In\n\nand saccade. In IJCV, 2000.\n\nISMAR, 2014.\n\n[32] S. Soatto. Actionable information in vision. In ICCV, 2009.\n[33] S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser. Semantic scene completion\nfrom a single depth image. IEEE Conference on Computer Vision and Pattern Recognition,\n2017.\n\n[34] R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour. Policy gradient methods for rein-\nforcement learning with function approximation. In Advances in neural information processing\nsystems, pages 1057\u20131063, 2000.\n\n[35] B. W. Tatler, M. M. Hayhoe, M. F. Land, and D. H. Ballard. Eye guidance in natural vision:\n\nReinterpreting salience. Journal of Vision, 11(5):5, 2011.\n\n[36] S. Tulsiani, T. Zhou, A. A. Efros, and J. Malik. Multi-view supervision for single-view\n\nreconstruction via differentiable ray consistency. CoRR, abs/1704.06254, 2017.\n\n[37] H. F. Tung, A. Harley, W. Seto, and K. Fragkiadaki. Adversarial inverse graphics networks:\nLearning 2d-to-3d lifting and image-to-image translation with unpaired supervision. ICCV,\n2017.\n\n[38] S. Vijayanarasimhan, S. Ricco, C. Schmid, R. Sukthankar, and K. Fragkiadaki. Sfm-net:\n\nLearning of structure and motion from video. arXiv preprint arXiv:1704.07804, 2017.\n\n[39] D. Wilkes and J. Tsotsos. Active object recognition. In CVPR, 1992.\n[40] L. L. Wong, L. P. Kaelbling, and T. Lozano-Perez. Constructing semantic world models from\npartial views. In Robotics: Science and Systems (RSS) Workshop on Robots in Clutter, 2013.\n[41] J. Wu, Y. Wang, T. Xue, X. Sun, W. T. Freeman, and J. B. Tenenbaum. Marrnet: 3d shape\n\nreconstruction via 2.5d sketches. CoRR, abs/1711.03129, 2017.\n\n[42] J. Wu, T. Xue, J. J. Lim, Y. Tian, J. B. Tenenbaum, A. Torralba, and W. T. Freeman. 3d interpreter\nnetworks for viewer-centered wireframe modeling. International Journal of Computer Vision\n(IJCV), 2018.\n\n[43] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: A deep\nrepresentation for volumetric shapes. In CVPR, pages 1912\u20131920. IEEE Computer Society,\n2015.\n\n[44] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe. Unsupervised learning of depth and ego-\n\nmotion from video. In arxiv, 2017.\n\n[45] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven\n\nvisual navigation in indoor scenes using deep reinforcement learning. In ICRA, 2017.\n\n11\n\n\f", "award": [], "sourceid": 2453, "authors": [{"given_name": "Ricson", "family_name": "Cheng", "institution": "Carnegie Mellon University"}, {"given_name": "Ziyan", "family_name": "Wang", "institution": "Carnegie Mellon University"}, {"given_name": "Katerina", "family_name": "Fragkiadaki", "institution": "Carnegie Mellon University"}]}