{"title": "CNN^{2}: Viewpoint Generalization via a Binocular Vision", "book": "Advances in Neural Information Processing Systems", "page_first": 1988, "page_last": 2000, "abstract": "The Convolutional Neural Networks (CNNs) have laid the foundation for many techniques in various applications. Despite achieving remarkable performance in some tasks, the 3D viewpoint generalizability of CNNs is still far behind humans visual capabilities. Although recent efforts, such as the Capsule Networks, have been made to address this issue, these new models are either hard to train and/or incompatible with existing CNN-based techniques specialized for different applications. Observing that humans use binocular vision to understand the world, we study in this paper whether the 3D viewpoint generalizability of CNNs can be achieved via a binocular vision. We propose CNN^{2}, a CNN that takes two images as input, which resembles the process of an object being viewed from the left eye and the right eye. CNN^{2} uses novel augmentation, pooling, and convolutional layers to learn a sense of three-dimensionality in a recursive manner. Empirical evaluation shows that CNN^{2} has improved viewpoint generalizability compared to vanilla CNNs. Furthermore, CNN^{2} is easy to implement and train, and is compatible with existing CNN-based specialized techniques for different applications.", "full_text": "CNN2: Viewpoint Generalization via\n\na Binocular Vision\n\nWei-Da Chen\n\nDepartment of Computer Science\nNational Tsing-Hua University\n\nTaiwan, R.O.C.\n\nwdchen@datalab.cs.nthu.edu.tw\n\nShan-Hung Wu\n\nDepartment of Computer Science\nNational Tsing-Hua University\n\nTaiwan, R.O.C.\n\nshwu@cs.nthu.edu.tw\n\nAbstract\n\nThe Convolutional Neural Networks (CNNs) have laid the foundation for many\ntechniques in various applications. Despite achieving remarkable performance in\nsome tasks, the 3D viewpoint generalizability of CNNs is still far behind humans\nvisual capabilities. Although recent efforts, such as the Capsule Networks, have\nbeen made to address this issue, these new models are either hard to train and/or\nincompatible with existing CNN-based techniques specialized for different appli-\ncations. Observing that humans use binocular vision to understand the world, we\nstudy in this paper whether the 3D viewpoint generalizability of CNNs can be\nachieved via a binocular vision. We propose CNN2, a CNN that takes two images as\ninput, which resembles the process of an object being viewed from the left eye and\nthe right eye. CNN2 uses novel augmentation, pooling, and convolutional layers to\nlearn a sense of three-dimensionality in a recursive manner. Empirical evaluation\nshows that CNN2 has improved viewpoint generalizability compared to vanilla\nCNNs. Furthermore, CNN2 is easy to implement and train, and is compatible with\nexisting CNN-based specialized techniques for different applications.\n\n1\n\nIntroduction\n\nConvolutional Neural Networks (CNNs, LeCun et al. (1989, 1998)) are models inspired by how the\nanimal visual cortex works (Hubel and Wiesel (1962)) and are computationally modelled (Fukushima\nand Miyake (1982)) based on local connectivities between neurons and hierarchically organized\ntransformations of an image. CNNs have greatly advanced the state-of-the-art performance of visual\nrecognition tasks, such as image classi\ufb01cation (Real et al. (2018); He et al. (2016); Krizhevsky et al.\n(2012)), localization and detection (Lin et al. (2017b); Redmon et al. (2016)), segmentation (He\net al. (2017); Long et al. (2015)), and have driven the development of various specialized techniques\nfor applications in natural language processing (Gehring et al. (2017a,b)), search (McDonald et al.\n(2018); Dai et al. (2018)), mapping (Liu et al. (2017); Zhu et al. (2017)), medicine (Esteva et al.\n(2019)), drones (Kim et al. (2017); Kyrkou et al. (2018)), and self-driving cars (Codevilla et al.\n(2018); Bojarski et al. (2016)).\nDespite giving impressive performance in many applications, CNNs still have a long way to go in\nterms of being comparable to human\u2019s visual ability. One important aspects where vanilla CNNs\nfall short is referred to as transformation generalizability\u2014the ability to generalize what have been\nlearned from training images to understand the transformed images at test time. While there are\nmany studies (Jaderberg et al. (2015); Maninis et al. (2016); Cheng et al. (2016); Laptev et al. (2016);\nWorrall et al. (2017); Hinton et al. (2018); Cheng et al. (2019); Ecker et al. (2019)) that address 2D\ntransformations (e.g., rotation, scaling, and sheering), few efforts have been made towards a more\nchallenging goal called 3D viewpoint generalization; that is, to understand images of 3D objects with\nunseen viewpoint translation at test time.\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fFigure 1: A simple extension (LeCun et al. (2004)) of the CNN architecture for binocular images.\nThe two images from the left eye and the right eye are merged and then fed into regular CNN layers.\n\nA well-known branch of studies targeting 3D viewpoint generalizability is the capsule networks\n(Hinton et al. (2011); Sabour et al. (2017); Hinton et al. (2018)), which represent an object or a\npart of an object as a collection of neurons called a capsule. It organizes different capsules in a\nparse tree where the output of lower-level capsules is dynamically routed to upper-level capsules\nusing an agreement protocol. The capsule networks show some promising results that deserve\nfurther investigation. Some researchers who have done so, such as Peer et al. (2018), found out\nthat capsule networks are harder to train than conventional CNNs, because the capsules increase the\nnumber of model parameters. Also, the iterative routing-by-agreement algorithm used for training\nis time consuming and does not ensure the emergence of a parse tree in the network. Additionally,\nthe architecture of capsule networks is not compatible with CNNs, which prevents the large CNN\necosystem from being able to add values to and bene\ufb01t from the capsule nets.\nThe above drawbacks motivate us to seek for a more generalizable model that is compatible with\nexisting CNN-based techniques. An obvious difference between how humans and machines view\nan object is that humans visualize using two eyes. Fortunately nowadays, binocular images can be\neasily collected. For instance, majority of people are using their smartphones, which are now usually\nequipped with dual or more lens (Moura et al. (2014)), as cameras to record daily events. As another\nexample, one can extract two nearby frames in online videos to construct a large binocular image\ndataset.\nIn this paper, we propose CNN2, a convolutional neural network with improved 3D viewpoint\ngeneralizability by taking two binocular images as input. Unlike a simple CNN extension (LeCun\net al. (2004), as shown in Figure 1) that stacks up two images along the channel dimension and then\nfeeds them to a regular CNN network, CNN2 explicitly models some priors from binocular vision.\nWe apply contrastive channel augmentation to the respective images so they are scanned by \ufb01lters\n(or kernels) in two parallel, complementary feedforward pathways. This resembles the dual-path\ncentral visual pathways (Wurtz et al. (2000); Milner and Goodale (2006)) in human brains. After\nthe augmentation, the CNN2 employs a novel concentric multi-scale pooling layers that are applied\nbefore the convolutional layers to learn the in-focus and out-of-focus features. Such a design is\ninspired by the interactions between the V1 and V2 visual cortices in human visual cortex system\n(Biederman (1987); Reid and Alonso (1995); Murphy et al. (1999)). We conduct experiments using\nbinocular images from the SmallNORB (LeCun et al. (2004)), ModelNet (Wu et al. (2015)) and\nlarger-scale RGB-D Object (Lai et al. (2011)) datasets. The results demonstrate that CNN2 can learn\na sense of three-dimensionality in a recursive manner and has improved 3D viewpoint generalizability.\nFurthermore, CNN2 is easy to implement and train, and is compatible with existing CNN-based\nspecialized techniques for different vision applications.\n\n2 Model Design of CNN2\n\nFor ease of presentation, we consider a supervised learning task: given a task model f and a binocular\nimage set D = (X ,Y) = {(x(i)\nR , y(i))}i where each x(i)\nL and x(i)\nR represent the images taken\nfrom the left eye and the right eye viewpoints, respectively. Our goal is to design an embedding\nmodel g such that, after being trained using D, it can help \u02c6y\n(cid:48)\n= f (g(x(cid:48)\nR)) predict the correct\nlabel y(cid:48) of a pair (x(cid:48)\n\nR) of binocular images taken from an unseen viewpoint at test time.\n\nL, x(cid:48)\n\nL , x(i)\n\nL, x(cid:48)\n\n2\n\n\u00a0xLxRClassifier\u00a0\u00a0ConvPooling\u00a0ConvPooling\u00a0ConvPooling\u00a0merge\fFigure 2: CNN2 model architecture that has two feedforward pathways providing the dual parallax\naugmentation at different abstraction levels. h \u2208 RW\u00d7H\u00d7C: raw feature map. \u02dch \u2208 RW\u00d7H\u00d72C:\n\u02d9h \u2208 RW\u00d7H\u00d72CS: output of the concentric multi-scale (CM)\nparallax augmented feature map.\npooling, where S is the number of scales.\n\nOne naive idea to improve the 3D viewpoint generalizability is to learn a depth map (Godard et al.\n(2017); Kendall et al. (2017)) from a pair of binocular images, treat the depth map as a new channel\nin the input (left or right eye) image, and feed the augmented image to a regular CNN just like the\none shown in Figure 1. However, the depth information is only a subset of the knowledge that can be\nlearned from binocular vision. Studies in neuroscience have found out that human\u2019s visual system\ncan detect stereoscopic edges (Von Der Heydt et al. (2000)), foreground and background (Qiu and\nVon Der Heydt (2005); Maruko et al. (2008)), and illusory contours of objects extrapolated from seen\nangles (von der Heydt et al. (1984); Anzai et al. (2007)) from binocular images. Hence, our goal is to\ndesign a model g that is able to capture these generic patterns.\nNext, we present the CNN2 that is able to recognize generic binocular vision patterns recursively at\ndifferent layers. It can be jointly trained with the task model f in an end-to-end manner.\nDual Feedforward Pathways. Figure 2 shows the architecture of CNN2. Unlike a regular CNN that\nhas only one feedforward pathway, the CNN2 employs two parallel, yet complementary, feedforward\npathways for the left and right eye images, respectively. At each layer, the binocular images or\nfeature maps are combined and then split by following the dual parallax augmentation procedure.\nSpeci\ufb01cally, given a pair of binocular images or feature maps (hL \u2208 RW\u00d7H\u00d7C, hR \u2208 RW\u00d7H\u00d7C),\nwe augment hL by adding the parallax hR \u2212 hL as new channels. Similarly, we augment hR by\nhL \u2212 hR. The two augmented maps (\u02dchL \u2208 RW\u00d7H\u00d72C, \u02dchR \u2208 RW\u00d7H\u00d72C) contain the information\nfrom both eyes, but on different bases (de\ufb01ned by the three original channels). Then each augmented\nmap is fed into the next layer either through the left or right pathway. This allows the \ufb01lters (or\nkernels) in convolutional layers to recursively detect stereoscopic features at different abstraction\nlevels by looking into the parallax. The small differences between the two input images at the pixel\nlevel and at shallow layers may add up to a big difference at a deeper layer, as discovered in human\nvisual system (Biederman (1987); Murphy et al. (1999); Reid and Alonso (1995)).\nConcentric Multi-Scale Pooling: Human and camera lens both re\ufb02ect the light following the\nprinciples of optics, and objects become blurry when they are out of focus. In addition to parallax\naugmentation, by comparing clear and blurred features from the previous layer, we allow a \ufb01lter\nto detect stereoscopic patterns. We introduce a new type of pooling layers, called the concentric\nmulti-scale (CM) pooling. Figure 3 shows how the CM pooling works. Formally, let \u02dch \u2208 RW\u00d7H\u00d72C\nbe an augmented image or feature map and suppose there are S given scales. At each scale s =\n0, 1,\u00b7\u00b7\u00b7 , S \u2212 1, we \ufb01rst obtain a temporary map e(s) \u2208 RW\u00d7H\u00d72C (assuming zero padding), where\n\ni,j,c = poolp,q:i\u2212s\u2264p\u2264i+s and j\u2212s\u2264q\u2264j+s{\u02dchp,q,c}\ne(s)\n\nand pool{\u00b7} is a pooling operation (e.g., max{\u00b7} or avg{\u00b7}). Then, these temporary maps are stacked\nup along the channel dimension to produce \u02d9h \u2208 RW\u00d7H\u00d72CS. Unlike conventional pooling layers\nthat come after the convolutional layers, the CM pooling layers are placed before the convolutional\nlayers. This aids the \ufb01lter in the next layer to easily detect stereoscopic patterns, by contrasting\nblurry features with clear features. The translation invariance created by an e(s) at a large scale (s)\n\n3\n\nCM\u00a0PoolingConv\u00a0CM\u00a0PoolingConv\u00a0CM\u00a0PoolingConv\u00a0hh\u0303\u00a0h\u02d9augmentaugmentaddClassifier\u00a0\u00a0augment\u00a0\u00a0xLxRCM\u00a0PoolingConv\u00a0CM\u00a0PoolingConv\u00a0CM\u00a0PoolingConv\u00a0augmentaugment\fFigure 3: Concentric multi-scale pooling that is\nplaced before a convolution in the CNN2. It en-\nables a \ufb01lter to easily detect stereoscopic patterns\nby contrasting in-focus features with out-of-focus\nfeatures.\n\ndetects blurry features in the background, while an e(s) at a small scale detects clear features in the\nforeground.\nNote that the feature map \u02d9h produced by a\nCM pooling layer is equivariant to input trans-\nlation. The CNN2 does not use conventional\npooling layers that are known to introduce\ntranslation invariance and decrease viewpoint\ngeneralizability (Hinton et al. (2011); Sabour\net al. (2017)). A drawback of the CNN2 is\nthat a feature map at a hidden layer will have\nthe same (large) width and height as that of\nthe input image, which could slow down the\nspeed of computation. Additionally, there is\nan increase in the number of \ufb01lter weights\ndue to a larger number of channels (2CS) in\n\u02d9h. These problems can be mitigated by us-\ning fewer \ufb01lters at each layer. Empirically, we\nfound that the CNN2 requires much fewer \ufb01lters\nthan the conventional CNNs for the same sat-\nisfactory performance. Also note that CNN2\ndoes not modify the convolutional layers in\nregular CNNs. This means that the CNN2 is compatible with the existing convolution-based\nenhancement techniques and can contribute to and bene\ufb01t from the rich CNN ecosystem.\nInspiration from Human Visual System.\nWhile the effectiveness of the CNN2 solely\ndepends on engineering efforts, the design of\nCNN2 model is loosely inspired by how the\nhuman visual system works. Figure 4 shows\nan oversimpli\ufb01ed version of the mammals\u2019 vi-\nsual system (Wurtz et al. (2000); Milner and\nGoodale (2006)). The visual information mainly\n\ufb02ows through the central visual pathways in the\nbrain. Although recent studies (Kheradpisheh\net al. (2016); Wallis et al. (2017); Laskar et al.\n(2018); Long and Konkle (2018)) have found\ncorrespondence between the activations of CNN\nlayers and the neuron responses in human\u2019s vi-\nsual cortex system, the CNNs are still differ-\nent from human visual system in many ways.\nOne key difference is that the CNNs have only\none feedforward pathway. On the other hand,\nthe CNN2 employs two feedforward pathways,\nwhich resembles the left and right halves of the\ncentral visual pathway in two sides of our brains.\nThe dual parallax augmentation at the input layer\nof the CNN2 corresponds to the optic chiasma\nin human visual system, where the information\ncoming from both eyes is combined, augmented,\nand then split. At deep layers, it resembles the\ninteractions between the left and right sides of\nthe brain which are known to have their own bias (Gotts et al. (2013)). For more discussion about the\ncorrespondency between the CNN2 components and human visual system, please refer to Section 2\nof the supplementary materials.\n\nFigure 4: Visual system of mammals (Wurtz et al.\n(2000); Milner and Goodale (2006)). The elec-\ntrical pulses from the two eyes are merged at the\noptic chiasma and then sent to the right and left\nbrains separately following two visual pathways.\nThe pulses will then \ufb01nally reach the visual cor-\ntex system (Biederman (1987); Reid and Alonso\n(1995); Murphy et al. (1999); Gotts et al. (2013))\nwhere the visual image is heavily processed by the\ninteraction between the right and left brains with\nrespective bias.\n\n4\n\nh\u0303\u00a0e(S\u22121)e(1)e(0)Optic\u00a0NerveOptic\u00a0Chiasm\u00a0Lateral\u00a0Geniculate\u00a0\u00a0Nucleus\u00a0(LGN)\u00a0Visual\u00a0Cortex\u00a0\u00a0System\u00a0\f(a)\n\n(b)\n\n(c)\n\nFigure 5: Examples of left eye images taken from different viewpoints in (a) ModelNet2D dataset\n(chairs), (b) SmallNORB dataset (humans), and (c) RGB-D Object dataset (\ufb02ashlights).\n\n3 Further Related Work\n\nHere, we review further related works that are not mentioned in Sections 1 and 2. For a complete\ndiscussion of the related work, please refer to Section 1 of the supplementary materials.\n3D Viewpoint Generalization. In addition to the capsule networks (Hinton et al. (2011); Sabour et al.\n(2017); Hinton et al. (2018)), another way to viewpoint generalization is using voxel discretization\n(Su et al. (2015); Qi et al. (2016); Yan et al. (2016); Qi et al. (2017)), which reconstructs manifold\n(and non-manifold) surfaces in the 3D space from point clouds using voxels as an intermediate\nrepresentation . However, these models require either the voxel-level supervision or omnidirectional\nimages as input, which are both expensive to collect in practice. Binocular Vision. Binocular images\nhave been used for learning the depth information. Godard et al. (2017) utilize binocular images\nto make a model learn the depth map in an unsupervised manner. Kendall et al. (2017) exploit the\ngeometry and context information in binocular images to let a model learn the disparity map of a\nstereogram. However, few studies (LeCun et al. (2004), whose architecture is shown in Figure 1)\nhave been made to understand the impact of binocular vision on CNN generalizability. Multi-Scale\nFeature Representations. A CNN2 layer extracts features at multiple scales, thus is related to the\nwork on multi-scale feature learning (Yang and Ramanan (2015); Cai et al. (2016); Lin et al. (2017a);\nChen et al. (2019)). Unlike most existing models that concatenate the multi-scale features to learn\npatterns, CNN2 pools (via the CM pooling) multi-scale features to make them of equal size and then\nstack them up along the channel dimension. The location information encoded in different feature\nmaps are aligned. This allows the next convolutional layer to learn location independent patterns (and\na sense of 3D dimensionality) by contracting the features at different scales. Pooling Strategies. Our\nCM pooling is cosmetically similar to some existing pooling techniques (He et al. (2014); Gong et al.\n(2014); Qi et al. (2018)). The spatial pyramid pooling (He et al. (2014)) pools image pixels using\nprede\ufb01ned patches, which require domain-speci\ufb01c knowledge to de\ufb01ne. The multi-scale orderless\npooling (Gong et al. (2014)) outputs feature maps of different sizes, but these maps are not \u201czoomed\u201d\nto equal size and then stacked up along the channel dimension to help the \ufb01lters contract features at\ndifferent scales at the same location. Qi et al. (2018) propose a concentric circle pooling strategy to\nachieve rotation invariance, where multiple \ufb01lters scan an image or feature map following concentric\nwindow-sliding paths. Their term \u201cconcentric\u201d is different from ours.\n\n4 Experiments\n\nIn this section, we evaluate the performance of CNN2 using three binocular image datasets: 1) the\nModelNet2D dataset rendered from ModelNet40 (Wu et al. (2015)) following the settings used by\nLeCun et al. (2004), 2) the SmallNORB dataset (LeCun et al. (2004)), and 3) the RGB-D Object\ndataset (Lai et al. (2011)), which consist of 12,311 grayscale, 48,600 grayscale, and 250,000 color\nimages taken from different azimuths with 5-, 20-, and 10-degree ticks, respectively. Figure 5 shows\nsome example images from these datasets. Only the SmallNORB dataset provides binocular images.\nFor the rest of the datasets, we use pairs of images having successive azimuths degrees to simulate\nbinocular images. We also sample 5 classes of objects from each dataset that look different from each\nother in any azimuths degree. For more information about the datasets and preprocessing, please refer\nto Section 3.1 of the supplementary materials. Note that 3D viewpoint generalization is a dif\ufb01cult and\nchallenging problem, wherein majority of existing work were only evaluated on grayscale datasets.\nTo the best of our knowledge, this is the \ufb01rst work that conducted experiments on colored datasets for\n3D viewpoint generalization.\nWe implement CNN2 and the following baselines using TensorFlow (Abadi et al. (2016)). Vanilla\nCNN. This is a simple CNN extension (LeCun et al. (2004)) whose architecture is shown in Figure\n1. CapsuleNet. This is capsule network with EM routing (Hinton et al. (2018)). It uses the matrix\ncapsules to capture the activation along with a pose matrix. PTN. The perspective transformer\n\n5\n\n\fTable 1: The number of parameters in different models for the grayscale (ModelNet2D and Small-\nNORB) and RGB-D Object datasets.\n\nGrayscale\n\nRGB-D Object\n\nVanilla CNN BL-Net Monodepth\n19M+333K\n19M+427K\n\n333K\n421K\n\n411K\n489K\n\nPTN CapsuleNet CNN2 CNN2+BL\n12M\n12M\n\n341K\n493K\n\n362K\n568K\n\n407K\n506K\n\nTable 2: Average test accuracy of different models over unseen viewpoints and the time required to\ntrain these models. The pair of numbers in Monodepth denotes the training time for the depth map\ngenerator (stage 1) and CNN (stage 2), respectively. The training of PTN and CapsuleNet on the\nRGB-D dataset did not converge.\n\nModels\n\nVanilla CNN\n\nBL-Net\n\nMonodepth\n\nPTN\n\nCapsuleNet\n\nCNN2\n\nCNN2+BL\n\nModelNet2D\n\nSmallNORB\n\nRGB-D Object\n\nAcc. (Unseen) Time (min) Acc. (Unseen) Time (min) Acc. (Unseen) Time (min)\n\n0.907\n0.903\n0.910\n0.879\n0.921\n0.941\n0.918\n\n138\n109\n\n143+127\n\n159\n478\n91\n115\n\n0.722\n0.751\n0.783\n0.714\n0.835\n0.865\n0.787\n\n231\n192\n\n168+150\n\n273\n1328\n121\n251\n\n0.795\n0.829\n0.802\n0.427\n0.476\n0.868\n0.778\n\n313\n288\n\n612+301\n\n-\n-\n236\n315\n\nnetwork (Yan et al. (2016)) that outputs 3D voxels. The original paper assumes omnidirectional\nimages of an object as the input. Here, we feed only the images within a particular range of view\nangles that is available at training time (see Section 4.1 for more details about the range) to the\nnetwork to get output voxels. Then, we feed the voxels into a 3D convolutional neural network for\nclassi\ufb01cation. We follow the settings described in the original paper (Yan et al. (2016)) and the study\n(Maturana and Scherer (2015)) to train the entire model from end to end. Monodepth. A model\nbased on the depth information, which is explicitly learned from the binocular images. The original\nMonodepth network (Godard et al. (2017)) is a model that outputs the depth map for a given pair\nof binocular images. It can be trained in an unsupervised manner. We create a two-stage training\nprocess here. In the \ufb01rst stage (pre-training stage), we train a Monodepth network and use it to\ngenerate a depth map. Then, in the second stage, we add the depth map into the the left eye image as\nan additional channel and feed the augmented image to a CNN. The CNN architecture is the same as\nthat used in Vanilla CNN. We follow the settings described in the Monodepth network paper (Godard\net al. (2017)) to train the model for stage one. BL-Net. This network is composed of a concatenation\nof Big-Little module (BL-module), which aims to extract multi-scale feature representations with\na good trade-off between speed and accuracy. Here, we extend the Vanilla CNN by replacing its\narchitecture with the Big-Little network following the settings in Chen et al. (2019). CNN2+BL. To\nsee whether our CM pooling can help a model learn beyond the multi-scale features, we also replace\nthe blocks of layers of CNN2 with the BL-modules, while keeping the dual feedforward pathways\nand parallax augmentation.\nWe conduct experiments on a computer with an Intel Core i7-6900K CPU, 64 GB RAM, and an\nNVIDIA Geforce GTX 1070 GPU. We did not augment the data at training time in order to observe\nthe unbiased generalizability of different models. For each of the above models, we search for the\nbest architecture for a given dataset. Table 1 shows the number of parameters in different models.\nPlease see Section 3.2 of the supplementary materials for more details.\n\n4.1\n\n3D Viewpoint Generalization\n\nTo test the 3D viewpoint generalizability of different models, we train the models using (binocular)\nimages taken from a limited range of view angles and then test the model performance using images\ntaken from unlimited view angles. On the ModelNet2D dataset, we use the images taken from\nazimuths of degrees from 50 to 125 as the training set, degrees from 30 to 45 and from 130 to 145\nas the validation set, and unlimited degrees as the test set. On the SmallNORB dataset, we use the\nimages taken from azimuths of degrees from 20 to 80 as the training set, degrees at 0 and 100 as\nthe validation set, and the rest as the test set. On the RGB-D Object dataset, images of different\nobjects are taken from different viewpoints. So, we use images taken from one third of continuous\n\n6\n\n\f(a) ModelNet2D\n\n(b) SmallNORB\n\n(c) RGB-D Object\n\n(d) ModelNet2D\n\n(e) SmallNORB\n\n(f) RGB-D Object\n\nFigure 6: 3D viewpoint generalizability of models trained on each dataset. (a)-(c) Test accuracy at\ndifferent view angles with about two third of the view angles that are not seen at training time. (d)-(f)\nLearning curve (early stopped or truncated at 120 mins).\n\nviewpoints of each object as the training set and the remaining images as the test set. We further split\none third of the training images having continuous viewpoints as the validation set.\nThe average test accuracy of different models over all unseen angles and the time required to train\nthese models is shown in Table 2. It can be seen that CNN2 achieves higher accuracy than all\nthe baseline models. It also converges faster during the training process. Figure 6 shows how the\naccuracy of different models varies at different view angles and how the models learn over time. On\ngrayscale datasets (ModelNet2D and SmallNORB), both the CapsuleNet and CNN2 give signi\ufb01cantly\nbetter performance than the other baselines at challenging view angles where the objects look very\ndifferent from what they appeared at training time. However, the CNN2 is much faster to train\nthan the CapsuleNet. In fact, the learning speed of the CNN2 is even faster than the Vanilla CNN.\nNote that the CNN2 uses much fewer \ufb01lters (50) than the Vanilla CNN (112). This justi\ufb01es that the\npatterns detected by CNN2 \ufb01lters are useful for 3D viewpoint generalization. Also, by comparing\nthe performance of CNN2 and CNN2+BL, we know that the performance gain is not from merely\nextracting the multi-scale features. The CM pooling indeed helps the CNN2 \ufb01lters learn generic\nstereoscopic features by contracting the features at different scales. On the colored RGB-D Object\ndataset, the CNN2 still outperforms other baselines. The CapsuleNet and PTN perform poorly in this\ncase. We have searched different architectures for these models for better performance, but failed\n(see Section 3.2 of the supplementary materials). Our \ufb01ndings about the CapsuleNet is consistent\nwith Peer et al. (2018), who pointed out that the capsule networks are harder to train and the iterative\nrouting-by-agreement algorithm used for training does not ensure the emergence of a parse tree in the\nnetworks. As for the (voxel-based) PTN, we suspect that it has too high sample complexity to easily\nlearn from a color binocular dataset.\n\n4.2 Backward Compatibility\n\n2D Rotation Generalizability. The CNN2 does not change the convolution operation, which makes\nit compatible with the rich CNN ecosystem. To see how this can be bene\ufb01cial, we design a more\nchallenging task where the models are asked to predict the labels of images taken from unseen view\nangles and unseen 2D rotations at test time. We train and validate the models using the images\nfrom the ModelNet2D dataset that have 50% chance to be rotated 90 degrees clockwisely. At test\ntime, we feed the model with the images that is rotated either 180 or 270 degrees clockwisely (in\naddition to viewpoint shift described in Section 4.1). Without data augmentation and specialized\ntechniques, the convolution-based methods, including CNN2, give degraded performance in this\n\n7\n\nVanilla CNNBL-netMonoDepthPTNCapsuleNetCNN2 + BL-moduleCNN2901802700View Angle0.70.80.91.0Accuracy901802700View Angle0.30.40.50.60.70.80.91.0 090180270View Angle0.30.40.50.60.70.80.91.0 20406080100Training Time (mins)0.30.50.70.9Validation Accuracy20406080100Training Time (mins)0.30.50.70.9 20406080100Training Time (mins)0.30.50.70.9 \f(a)\n\n(b)\n\n(c)\n\nFigure 7: Backward compatibility. (a) 3D viewpoint + 2D rotation generalizability of models trained\non the ModelNet2D dataset. Test accuracy for objects with unseen rotations (180 and 270 degrees) at\ndifferent view angles. Angles outside [50, 125] are unseen. See Figure 6 for the legends. (b) CNN2 is\nbackward compatible with existing CNN-based techniques and can be readily enhanced to have 2D\nrotation generalizability. (c) Performance of CNN2 with monocular images from the RGB-D Object\ndataset. CNN2 is also backward compatible with single-eye image classi\ufb01cation tasks.\n\ntask, as shown in Figure 7(a). Only the non-convolutional CapsuleNet achieves stable performance\nacross viewpoints and rotations. However, there exists many CNN-based techniques that target 2D\nrotation generalizability, such as the spatial transformer networks (STN, Jaderberg et al. (2015)),\ngroup equivariant convolutional neural networks (GCNN, Cohen and Welling (2016)), and harmonic\nnetworks (HN, Worrall et al. (2017)). We integrate these methods into the CNN2 and get signi\ufb01cantly\nbetter performance, as shown in Figure 7(b). The performance boost is consistent on other datasets\n(see Section 3.3 of the supplementary materials). This demonstrates the potential of CNN2 for\nbene\ufb01tting from, and contributing to, many applications where CNNs thrive. Monocular Images.\nWith monocular images, the parallax channels contain all zeros, therefore the CNN2 degenerates into\na conventional CNN gracefully. Figure 7(c) shows the performance of degenerated CNN2 with the\nsingle-eye images from the RGB-D Object dataset. Although the degenerated CNN2 with monocular\nimages does not outperform the fully functional CNN2 with binocular inputs due to the lack of\nbinocular information, its performance is comparable with (if not surpasses) that of vanilla CNN\nbecause it models more prior than vanilla CNN. The CNN2 is compatible with single-eye image\nclassi\ufb01cation tasks.\n\n4.3 More Experiments\n\nAblation Study. Here, we investigate whether each designed component used by the CNN2 improves\n3D viewpoint generalizability. Following the settings described in Section 4.1, we compare the CNN2\nwith its variant where the weights along the dual feedforward pathways are tied. The results, as\nshown in Figure 8(a), indicate that having two feedforward pathways is indeed bene\ufb01cial. Next, we\ncompare the CNN2 with another version where parallax augmentation is dropped. As we can see\nfrom Figure 8(b), the parallax augmentation can improve the model generalizability at challenging\nview angles. Next, we test whether the concentric multi-scale (CM) pooling contributes to 3D\nviewpoint generalizability. We compare the CNN2 with a variant where the CM pooling layers are\nreplaced by conventional max pooling layers. The results, which are shown in Figure 8(c), con\ufb01rm\nits effectiveness. We can also see from Figure 8(d) that the standalone CM pooling is suf\ufb01cient\nto improve the generalizability of vanilla CNN. Pooling before Convolution. We also have an\ninteresting observation: while placing the pooling layers after the convolution layers give better\nperformance in regular CNNs, it hurts the generalizability of CNN2, as shown in Figure 8(e). This\nreminds us that something we took for granted in monocular vision may not be the best choice for\nthe binocular cases. Fusion of the Two Feedforward Pathways. To show that the fusion (i.e., dual\nparallax augmentation) of the two feedforward pathways at each layer is bene\ufb01cial, we compare\nCNN2 with two new baselines that perform early and late fusion in only the \ufb01rst and last layer,\nrespectively. Figure 8(f) shows the results on the RGB-D Object dataset. CNN2 outperforms other\nbaselines because it has fusion at all layers, which allows small differences between the feature maps\nin two paths to add up to a big difference at a deeper layer. Backbone Choices. The CNN2 can\nwork with different backbone architectures. To show this, we compare the performance of CNN2\nwith ResNet-50 and a toy ResNet as the backbone on the SmallNORB dataset. The toy ResNet is\nconsisted of 2 residual blocks and has similar number of parameters as CNN2. The results are shown\nin Figure 9(a). Although the SmallNORB dataset contains only grayscale images and looks easy,\n\n8\n\n901802700View Angle0.60.70.80.91.0Accuracy901802700View Angle0.60.70.80.91.0AccuracyCNN2CNN2 + STNCNN2 + GCNNCNN2 + HNCapsuleNet090180270View Angle0.30.40.50.60.70.80.91.0AccuracyCNN2 (Binocular)CNN2 (Monocular)CNN (Monocular)\f(a)\n\n(d)\n\n(b)\n\n(e)\n\n(c)\n\n(f)\n\nFigure 8: Ablation Study. (a) Single (weight-tied) vs. dual feedforward pathways. (b) CNN2 with vs.\nwithout parallax augmentation. (c) Max pooling (before convolution) vs. CM pooling. (d) The CM\npooling, by itself, can improve the performance of vanilla CNN on the RGB-D Object dataset. (e)\nPerforming pooling before convolution improves performance in CNN2, but not in vanilla CNN. (f)\nPerformance of CNN2 variants with different fusion strategies on the RGB-D Object dataset.\n\n(a)\n\n(b) CNN\n\n(c) CNN2\n\nFigure 9: (a) Performance of different models with stronger network backbone (ResNet) on the\nSmallNORB dataset. (b)(c) Confusion matrices of the predictions made by CNN and CNN2 on the\nRGBD-Object dataset.\n\nneither of the ResNet variants generalizes better than CNN2. A backbone like ResNet that is strong\nto make predictions at seen angles does not imply that it is strong at unseen angles, and it can still\nbene\ufb01t from CNN2 to have improved 3D viewpoint generalizability. Confusion Matrices. Finally,\nwe investigate how the predictions made by CNN2 differ from those of vanilla CNN. Figures 9(b)(c)\nshow the confusion matrices of the predictions made by CNN and CNN2 at unseen view angles on\nthe RGBD-Object dataset. The CNN2 outperforms CNN in most cases, except when classifying the\nclasses 1 (\ufb02ashlight) and 4 (stapler) that are similar in shape but different in texture at certain view\nangles. This suggests that the CNN2 relies more on shapes than textures to generalize, a bias that\nhumans have been shown to possess (Geirhos et al. (2019)).\n\n5 Conclusion\n\nWe propose the CNN2 that gives improved 3D viewpoint generalizability of CNNs via a binocular\nvision. The CNN2 uses dual feedforward pathways, recursive parallax augmentation, and the\nconcentric multi-scale pooling to learn stereoscopic features. One important research direction\nfollowing our work is to understand and visualize what have been learned by the \ufb01lters and how\nthey relate to that of human visualization. Furthermore, it would be interesting to apply CNN2 to\napplications wherein a generalized vision system is highly in-demand, such as in self-driving cars.\n\n9\n\n901802700View Angle0.70.80.91.0AccuracySingle PathwayDual Pathways901802700View Angle0.70.80.91.0 w/o augmentationw/ augmentation 901802700View Angle0.70.80.91.0 Single-ScaleMulti-Scale090180270View Angle0.30.40.50.60.70.80.91.0AccuracyCNN2CNNCNN+CM pool901802700View Angle0.50.60.70.80.91.0 CNN2 (conv->pool)CNN2 (pool->conv)CNN (pool->conv)CNN (conv->pool)090180270View Angle0.30.40.50.60.70.80.91.0AccuracyCNN2Early FusionLate Fusion901802700View Angle0.40.50.60.70.80.91.0AccuracyCNN2Toy ResnetResnet 5001234012347691070142102387340112700110000617010770236160318170200400600800100001234012341034100452146512014410011000064010900166260490402004006008001000\f6 Acknowledgments\n\nThis work is supported by the MOST Joint Research Center for AI Technology and All Vista\nHealthcare, Taiwan (MOST 108-2634-F-007-003-). We also thank the anonymous reviewers for their\ninsightful feedbacks.\n\nReferences\nMart\u00edn Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu\nDevin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensor\ufb02ow: a system for\nlarge-scale machine learning.\n\nAkiyuki Anzai, Xinmiao Peng, and David C Van Essen. 2007. Neurons in monkey visual area V2\n\nencode combinations of orientations. Nature neuroscience 10, 10 (2007), 1313.\n\nIrving Biederman. 1987. Recognition-by-components: a theory of human image understanding.\n\nPsychological review 94, 2 (1987), 115.\n\nMariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon\nGoyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. 2016. End to end\nlearning for self-driving cars. arXiv preprint arXiv:1604.07316 (2016).\n\nZhaowei Cai, Quanfu Fan, Rogerio S Feris, and Nuno Vasconcelos. 2016. A uni\ufb01ed multi-scale deep\n\nconvolutional neural network for fast object detection. In Proc. of ECCV.\n\nChun-Fu Chen, Quanfu Fan, Neil Mallinar, Tom Sercu, and Rogerio Feris. 2019. Big-little net: An\n\nef\ufb01cient multi-scale feature representation for visual and speech recognition. (2019).\n\nGong Cheng, Peicheng Zhou, and Junwei Han. 2016. Rifd-cnn: Rotation-invariant and \ufb01sher\n\ndiscriminative convolutional neural networks for object detection. In Proc. of CVPR.\n\nXiuyuan Cheng, Qiang Qiu, Robert Calderbank, and Guillermo Sapiro. 2019. RotDCF: Decomposi-\n\ntion of Convolutional Filters for Rotation-Equivariant Deep Networks. In Proc. of ICLR.\n\nFelipe Codevilla, Matthias Miiller, Antonio L\u00f3pez, Vladlen Koltun, and Alexey Dosovitskiy. 2018.\n\nEnd-to-End Driving Via Conditional Imitation Learning. In Proc. of ICRA.\n\nTaco Cohen and Max Welling. 2016. Group Equivariant Convolutional Networks. In Proc. of ICML.\n\nZhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018. Convolutional neural networks\nfor soft-matching n-grams in ad-hoc search. In Proc. of the Eleventh ACM International Conference\non Web Search and Data Mining. ACM, 126\u2013134.\n\nAlexander S. Ecker, Fabian H. Sinz, Emmanouil Froudarakis, Paul G. Fahey, Santiago A. Cadena,\nEdgar Y. Walker, Erick Cobos, Jacob Reimer, Andreas S. Tolias, and Matthias Bethge. 2019. A\nrotation-equivariant convolutional neural network model of primary visual cortex. In Proc. of\nICLR.\n\nAndre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov, Mark DePristo,\nKatherine Chou, Claire Cui, Greg Corrado, Sebastian Thrun, and Jeff Dean. 2019. A guide to deep\nlearning in healthcare. Nature Medicine 25 (2019).\n\nKunihiko Fukushima and Sei Miyake. 1982. Neocognitron: A self-organizing neural network model\nfor a mechanism of visual pattern recognition. In Competition and cooperation in neural nets.\nSpringer, 267\u2013285.\n\nJonas Gehring, Michael Auli, David Grangier, and Yann Dauphin. 2017a. A Convolutional Encoder\n\nModel for Neural Machine Translation. In Proc. of ACL.\n\nJonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017b. Convolu-\n\ntional sequence to sequence learning. Proc. of ICML (2017).\n\n10\n\n\fRobert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and\nWieland Brendel. 2019. ImageNet-trained CNNs are biased towards texture; increasing shape bias\nimproves accuracy and robustness.. In Proc. of ICLR.\n\nCl\u00e9ment Godard, Oisin Mac Aodha, and Gabriel J Brostow. 2017. Unsupervised monocular depth\n\nestimation with left-right consistency. In Proc. of CVPR. 6602\u20136611.\n\nYunchao Gong, Liwei Wang, Ruiqi Guo, and Svetlana Lazebnik. 2014. Multi-scale Orderless Pooling\n\nof Deep Convolutional Activation Features. (2014).\n\nStephen J Gotts, Hang Joon Jo, Gregory L Wallace, Ziad S Saad, Robert W Cox, and Alex Martin.\n2013. Two distinct forms of functional lateralization in the human brain. Proc. of the National\nAcademy of Sciences (2013), 201302581.\n\nKaiming He, Georgia Gkioxari, Piotr Doll\u00e1r, and Ross Girshick. 2017. Mask r-cnn. In Proc. of ICCV.\n\nIEEE, 2980\u20132988.\n\nKaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2014. Spatial Pyramid Pooling in Deep\n\nConvolutional Networks for Visual Recognition. In Proc. of ECCV.\n\nKaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image\n\nrecognition. In Proc. CVPR. 770\u2013778.\n\nGeoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. 2011. Transforming auto-encoders. In\n\nInternational Conference on Arti\ufb01cial Neural Networks.\n\nGeoffrey E Hinton, Sara Sabour, and Nicholas Frosst. 2018. Matrix capsules with EM routing. In\n\nProc. of ICLR.\n\nDavid H Hubel and Torsten N Wiesel. 1962. Receptive \ufb01elds, binocular interaction and functional\n\narchitecture in the cat\u2019s visual cortex. The Journal of physiology 160, 1 (1962), 106\u2013154.\n\nMax Jaderberg, Karen Simonyan, Andrew Zisserman, et al. 2015. Spatial transformer networks. In\n\nProc. of NIPS.\n\nAlex Kendall, Hayk Martirosyan, Saumitro Dasgupta, and Peter Henry. 2017. End-to-end learning of\n\ngeometry and context for deep stereo regression. In Proc. of ICCV. 66\u201375.\n\nSaeed Reza Kheradpisheh, Masoud Ghodrati, Mohammad Ganjtabesh, and Timoth\u00e9e Masquelier.\n2016. Deep networks can resemble human feed-forward vision in invariant object recognition.\nScienti\ufb01c reports 6 (2016), 32672.\n\nByungkwan Kim, Hyunseong Kang, and Seong-Ook Park. 2017. Drone Classi\ufb01cation Using Convo-\n\nlutional Neural Networks With Merged Doppler Images. (2017).\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classi\ufb01cation with deep\n\nconvolutional neural networks. In Proc. of NIPS. 1097\u20131105.\n\nChristos Kyrkou, George Plastiras, Theocharis Theocharides, Stylianos I Venieris, and Christos-\nSavvas Bouganis. 2018. DroNet: Ef\ufb01cient convolutional neural network detector for real-time\nUAV applications. In Design, Automation & Test in Europe Conference & Exhibition (DATE),\n2018. IEEE, 967\u2013972.\n\nKevin Lai, Liefeng Bo, Xiaofeng Ren, and Dieter Fox. 2011. A Large-Scale Hierarchical Multi-View\n\nRGB-D Object Dataset. (2011).\n\nDmitry Laptev, Nikolay Savinov, Joachim M Buhmann, and Marc Pollefeys. 2016. TI-POOLING:\ntransformation-invariant pooling for feature learning in convolutional neural networks. In Proc. of\nCVPR.\n\nMd Nasir Uddin Laskar, Luis G Sanchez Giraldo, and Odelia Schwartz. 2018. Correspondence\nof Deep Neural Networks and the Brain for Visual Textures. arXiv preprint arXiv:1806.02888\n(2018).\n\n11\n\n\fYann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne\nHubbard, and Lawrence D Jackel. 1989. Backpropagation applied to handwritten zip code\nrecognition. Neural computation 1, 4 (1989), 541\u2013551.\n\nYann LeCun, L\u00e9on Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning\n\napplied to document recognition. Proc. IEEE 86, 11 (1998), 2278\u20132324.\n\nYann LeCun, Fu Jie Huang, and Leon Bottou. 2004. Learning methods for generic object recognition\n\nwith invariance to pose and lighting. In Proc. of CVPR, Vol. 2. IEEE, II\u2013104.\n\nTsung-Yi Lin, Piotr Doll\u00e1r, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.\n\n2017a. Feature pyramid networks for object detection. In Proc. of CVPR.\n\nTsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. 2017b. Focal Loss for\n\nDense Object Detection. In Proc. of ICCV. IEEE, 2999\u20133007.\n\nMing-Yu Liu, Thomas Breuel, and Jan Kautz. 2017. Unsupervised image-to-image translation\n\nnetworks. In Proc. of NIPS. 700\u2013708.\n\nBria Long and Talia Konkle. 2018. The role of textural statistics vs. outer contours in deep CNN and\nneural responses to objects. http://konklab.fas.harvard.edu/ConferenceProceedings/\nLong_2018_CCN.pdf.\n\nJonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic\n\nsegmentation. In Proc. of CVPR. 3431\u20133440.\n\nKevis-Kokitsi Maninis, Jordi Pont-Tuset, Pablo Arbel\u00e1ez, and Luc Van Gool. 2016. Convolutional\n\noriented boundaries. In Proc. of ECCV.\n\nIchiro Maruko, Bin Zhang, Xiaofeng Tao, Jianliang Tong, Earl L Smith III, and Yuzo M Chino. 2008.\nPostnatal development of disparity sensitivity in visual area 2 (v2) of macaque monkeys. Journal\nof Neurophysiology 100, 5 (2008), 2486\u20132495.\n\nDaniel Maturana and Sebastian Scherer. 2015. VoxNet: A 3D Convolutional Neural Network for\nReal-Time Object Recognition. In Proc. of IEEE/RSJ International Conference on Intelligent\nRobots and Systems.\n\nRyan McDonald, George Brokos, and Ion Androutsopoulos. 2018. Deep Relevance Ranking using\n\nEnhanced Document-Query Interactions. In Proc. of EMNLP. 1849\u20131860.\n\nDavid Milner and Mel Goodale. 2006. The visual brain in action. Oxford University Press.\n\nThiago DO Moura, Takashiro Tsukamoto, Shuji Tanaka, Daniel Filgueiras, and Davies W de\nLima Monteiro. 2014. Design of a dual lens system for a micromachined optical setup. In\nProc. Microelectronics Technology and Devices (SBMicro). IEEE, 1\u20134.\n\nPenelope C Murphy, Simon G Duckett, and Adam M Sillito. 1999. Feedback connections to the\nlateral geniculate nucleus and cortical response properties. Science 286, 5444 (1999), 1552\u20131554.\n\nDavid Peer, Sebastian Stabinger, and Antonio Rodriguez-Sanchez. 2018. Training Deep Capsule\n\nNetworks. arXiv preprint arXiv:1812.09707 (2018).\n\nCharles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017. Pointnet: Deep learning on point\n\nsets for 3d classi\ufb01cation and segmentation. (2017).\n\nCharles R Qi, Hao Su, Matthias Nie\u00dfner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas. 2016.\n\nVolumetric and multi-view cnns for object classi\ufb01cation on 3d data. In Proc. of CVPR.\n\nKunlun Qi, Qingfeng Guan, Chao Yang, Feifei Peng, Shengyu Shen, and Huayi Wu. 2018. Concentric\nCircle Pooling in Deep Convolutional Networks for Remote Sensing Scene Classi\ufb01cation. Remote\nSensing (2018).\n\nFangtu T Qiu and R\u00fcdiger Von Der Heydt. 2005. Figure and ground in the visual cortex: V2 combines\n\nstereoscopic cues with Gestalt rules. Neuron 47, 1 (2005), 155\u2013166.\n\n12\n\n\fEsteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. 2018. Regularized evolution for\n\nimage classi\ufb01er architecture search. arXiv preprint arXiv:1802.01548 (2018).\n\nJoseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Uni\ufb01ed,\n\nreal-time object detection. In Proc. of CVPR. 779\u2013788.\n\nR Clay Reid and Jose-Manuel Alonso. 1995. Speci\ufb01city of monosynaptic connections from thalamus\n\nto visual cortex. Nature 378, 6554 (1995), 281.\n\nSara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017. Dynamic routing between capsules. In\n\nProc. of NIPS.\n\nHang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. 2015. Multi-view\n\nconvolutional neural networks for 3d shape recognition. In Proc. of ICCV.\n\nR\u00fcdiger von der Heydt, Esther Peterhans, and Gunter Baumgartner. 1984. Illusory contours and\n\ncortical neuron responses. Science 224, 4654 (1984), 1260\u20131262.\n\nR\u00fcdiger Von Der Heydt, Hong Zhou, and Howard S Friedman. 2000. Representation of stereoscopic\n\nedges in monkey visual cortex. Vision research 40, 15 (2000), 1955\u20131967.\n\nThomas SA Wallis, Christina M Funke, Alexander S Ecker, Leon A Gatys, Felix A Wichmann, and\nMatthias Bethge. 2017. A parametric texture model based on deep convolutional features closely\nmatches texture appearance for humans. Journal of vision 17, 12 (2017), 5\u20135.\n\nDaniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. 2017.\n\nHarmonic networks: Deep translation and rotation equivariance. In Proc. of CVPR.\n\nZhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong\n\nXiao. 2015. 3d shapenets: A deep representation for volumetric shapes. In Proc. of CVPR.\n\nRobert H Wurtz, Eric R Kandel, et al. 2000. Central visual pathways. Principles of neural science 4\n\n(2000), 523\u2013545.\n\nXinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. 2016. Perspective transformer\nnets: Learning single-view 3d object reconstruction without 3d supervision. In Proc. of NIPS.\n1696\u20131704.\n\nSongfan Yang and Deva Ramanan. 2015. Multi-scale recognition with DAG-CNNs. In Proc. of\n\nICCV.\n\nJun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired Image-to-Image\n\nTranslation using Cycle-Consistent Adversarial Networks. In Proc. of ICCV.\n\n13\n\n\f", "award": [], "sourceid": 1179, "authors": [{"given_name": "Wei-Da", "family_name": "Chen", "institution": "National Tsing Hua University"}, {"given_name": "Shan-Hung (Brandon)", "family_name": "Wu", "institution": "National Tsing Hua University"}]}