{"title": "LF-Net: Learning Local Features from Images", "book": "Advances in Neural Information Processing Systems", "page_first": 6234, "page_last": 6244, "abstract": "We present a novel deep architecture and a training strategy to learn a local feature pipeline from scratch, using collections of images without the need for human supervision. To do so we exploit depth and relative camera pose cues to create a virtual target that the network should achieve on one image, provided the outputs of the network for the other image. While this process is inherently non-differentiable, we show that we can optimize the network in a two-branch setup by confining it to one branch, while preserving differentiability in the other. We train our method on both indoor and outdoor datasets, with depth data from 3D sensors for the former, and depth estimates from an off-the-shelf Structure-from-Motion solution for the latter. Our models outperform the state of the art on sparse feature matching on both datasets, while running at 60+ fps for QVGA images.", "full_text": "LF-Net: Learning Local Features from Images\n\nYuki Ono\n\nSony Imaging Products & Solutions Inc.\n\nyuki.ono@sony.com\n\nEduard Trulls\n\n\u00c9cole Polytechnique F\u00e9d\u00e9rale de Lausanne\n\neduard.trulls@epfl.ch\n\nPascal Fua\n\n\u00c9cole Polytechnique F\u00e9d\u00e9rale de Lausanne\n\npascal.fua@epfl.ch\n\nVisual Computing Group, University of Victoria\n\nKwang Moo Yi\n\nkyi@uvic.ca\n\nAbstract\n\nWe present a novel deep architecture and a training strategy to learn a local feature\npipeline from scratch, using collections of images without the need for human\nsupervision. To do so we exploit depth and relative camera pose cues to create a\nvirtual target that the network should achieve on one image, provided the outputs of\nthe network for the other image. While this process is inherently non-differentiable,\nwe show that we can optimize the network in a two-branch setup by con\ufb01ning it to\none branch, while preserving differentiability in the other. We train our method on\nboth indoor and outdoor datasets, with depth data from 3D sensors for the former,\nand depth estimates from an off-the-shelf Structure-from-Motion solution for the\nlatter. Our models outperform the state of the art on sparse feature matching on\nboth datasets, while running at 60+ fps for QVGA images.\n\n1\n\nIntroduction\n\nEstablishing correspondences across images is at the heart of many Computer Vision algorithms,\nsuch as those for wide-baseline stereo, object detection, and image retrieval. With the emergence\nof SIFT [23], sparse methods that \ufb01nd interest points and then match them across images became\nthe de facto standard. In recent years, many of these approaches have been revisited using deep\nnets [11, 33, 48, 49], which has also sparked a revival for dense matching [9, 43, 45, 52, 53].\nHowever, dense methods tend to fail in complex scenes with occlusions [49], while sparse methods\nstill suffer from severe limitations. Some can only train individual parts of the feature extraction\npipeline [33] while others can be trained end-to-end but still require the output of hand-crafted\ndetectors to initialize the training process [11, 48, 49]. For the former, reported gains in performance\nmay fade away when they are integrated into the full pipeline. For the latter, parts of the image which\nhand-crafted detectors miss are simply discarded for training.\nIn this paper, we propose a sparse-matching method with a novel deep architecture, which we\nname LF-Net, for Local Feature Network, that is trainable end-to-end and does not require using\na hand-crafted detector to generate training data. Instead, we use image pairs for which we know\nthe relative pose and corresponding depth maps, which can be obtained either with laser scanners or\nshape-from-structure algorithms [34], without any further annotation.\nBeing thus given dense correspondence data, we could train a feature extraction pipeline by selecting\na number of keypoints over two images, computing descriptors for each keypoint, using the ground\ntruth to determine which ones match correctly across images, and use those to learn good descriptors.\nThis is, however, not feasible in practice. First, extracting multiple maxima from a score map is\ninherently not differentiable. Second, performing this operation over each image produces two\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fdisjoint sets of keypoints which will typically produce very few ground truth matches, which we need\nto train the descriptor network, and in turn guide the detector towards keypoints which are distinctive\nand good for matching.\nWe therefore propose to create a virtual target response for the network, using the ground-truth\ngeometry in a non-differentiable way. Speci\ufb01cally, we run our detector on the \ufb01rst image, \ufb01nd the\nmaxima, and then optimize the weights so that when run on the second image it produces a clean\nresponse map with sharp maxima at the right locations. Moreover, we warp the keypoints selected\nin this manner to the other image using the ground truth, guaranteeing a large pool of ground truth\nmatches. Note that while we break differentiability in one branch, the other one can be trained end to\nend, which lets us learn discriminative features by learning the entire pipeline at once. We show that\nour method greatly outperforms the state-of-the-art.\n\n2 Related work\n\nSince the appearance of SIFT [23], local features have played a crucial role in computer vision,\nbecoming the de facto standard for wide-baseline image matching [14]. They are versatile [23, 29, 47]\nand remain useful in many scenarios. This remains true even in competition with deep network\nalternatives, which typically involve dense matching [9, 43, 45, 52, 53] and tend to work best on\nnarrow baselines, as they can suffer from occlusions, which local features are robust against.\nTypically, feature extraction and matching comprises three stages: \ufb01nding interest points, estimating\ntheir orientation, and creating a descriptor for each. SIFT [23], along with more recent methods [1,\n5, 32, 48] implements the entire pipeline. However, many other approaches target some of their\nindividual components, be it feature point extraction [31, 44], orientation estimation [50], or descriptor\ngeneration [36, 40, 41]. One problem with this approach is that increasing the performance of one\ncomponent does not necessarily translate into overall improvements [35, 48].\nNext, we brie\ufb02y introduce some representative algorithms below, separating those that rely on\nhand-crafted features from those that use Machine Learning techniques extensively.\nHand-crafted. SIFT [23] was the \ufb01rst widely successful attempt at designing an integrated solu-\ntion for local feature extraction. Many subsequent efforts focused on reducing its computational\nrequirements. For instance, SURF [5] used Haar \ufb01lters and integral images for fast keypoint detection\nand descriptor extraction. DAISY [41] computed dense descriptors ef\ufb01ciently from convolutions of\noriented gradient maps. The literature on this topic is very extensive\u2014we refer the reader to [28].\nLearned. While methods such as FAST [30] used machine learning techniques to extract keypoints,\nmost early efforts in this area targeted descriptors, e.g.using metric learning [38] or convex opti-\nmization [37]. However, with the advent of deep learning, there has been a renewed push towards\nreplacing all the components of the standard pipeline by convolutional neural networks.\n\u2013 Keypoints. In [44], piecewise-linear convolutional \ufb01lters were used to make keypoint detection\nrobust to severe lighting changes. In [33], neural networks are trained to rank keypoints. The latter\nis relevant to our work because no annotations are required to train the keypoint detector, but both\nmethods are optimized for repeatability and not for the quality of the associated descriptors. Deep\nnetworks have also been used to learn covariant feature detectors, particularly towards invariance\nagainst af\ufb01ne transformations due to viewpoint changes [22, 26].\n\u2013 Orientations. The method of [50] is the only one we know of that focuses on improving orientation\nestimates. It uses a siamese network to predict the orientations that minimize the distance between\nthe orientation-dependent descriptors of matching keypoints, assuming that the keypoints have been\nextracted using some other technique.\n\u2013 Descriptors. The bulk of methods focus on descriptors. In [13, 51], the comparison metric is\nlearned by training Siamese networks. Later works, starting with [36], rely on hard sample mining\nfor training and the l2 norm for comparisons. A triplet-based loss function was introduced in [3],\nand in [25], negative samples are mined over the entire training batch. More recent efforts further\nincreased performance using spectral pooling [46] and novel loss formulations [19]. However, none\nof these take into account what kind of keypoint they are working and typically use only SIFT.\nCrucially, performance improvements in popular benchmarks for a single one of either of these\nthree components do not always survive when evaluating the whole pipeline [35, 48]. For example,\n\n2\n\n\fDetector\n\n\ud835\udc65\ud835\udc65\ud835\udc56\ud835\udc56,\ud835\udc66\ud835\udc66\ud835\udc56\ud835\udc56,\ud835\udc60\ud835\udc60\ud835\udc56\ud835\udc56,\ud835\udf03\ud835\udf03\ud835\udc56\ud835\udc56\n\nSTN\n\nDescriptor\n\npatches\n\n\ud835\udc6b\ud835\udc6b\ud835\udc56\ud835\udc56\ud835\udc58\ud835\udc58\n\n(a) The LF-Net architecture. The detector network generates a scale-space score map along with dense orientation\nestimates, which are used to select the keypoints. Image patches around the chosen keypoints are cropped with a\ndifferentiable sampler (STN) and fed to the descriptor network, which generates a descriptor for each patch.\n\n(b) For training we use a two-branch LF-Net, containing two identical copies of the network, processing two\ncorresponding images Ii and Ij. Branch j (right) is used to generate a supervision signal for branch i (left),\ncreated by warping the results from i to j. As this is not differentiable, we optimize only over branch i, and\nupdate the network copy for branch j in the next iteration. We omit the samplers in this \ufb01gure, for simplicity.\n\nFigure 1: (a) The Local Feature Network (LF-Net). (b) Training with two LF-Nets.\n\nkeypoints are often evaluated on repeatability, which can be misleading because they may be repeat-\nable but useless for matching purposes. Descriptors can prove very robust against photometric and\ngeometric transformations, but this may be unnecessary or even counterproductive when patches are\nwell-aligned, and results on the most common benchmark [7] are heavily saturated.\nThis was demonstrated in [48], which integrated previous efforts [36, 44, 50] into a fully-differentiable\narchitecture, reformulating the entire keypoint extraction pipeline with deep networks. It showed\nthat not only is joint training necessary for optimal performance, but also that standard SIFT still\noutperforms many modern baselines. However, their approach still relies on SIFT keypoints for\ntraining, and as a result it can not learn where SIFT itself fails. Along the same lines, a deep network\nwas introduced in [11] to match images with a keypoint-based formulation, assuming a homography\nmodel. However, it was largely trained on synthetic images or real images with af\ufb01ne transformations,\nand its effectiveness on practical wide-baseline stereo problems remains unproven.\n\n3 Method\n\nFig. 1 depicts the LF-Net architecture (top), and our training pipeline with two LF-Nets (bottom).\nIn the following we \ufb01rst describe our network in Section 3.1. We break it down into its individual\ncomponents and detail how they are connected in order to build a complete feature extraction pipeline.\nIn Section 3.2, we introduce our training architecture, which is based on two LF-Net copies processing\nseparate images with non-differentiable components, along with the loss function used to learn the\nweights. In Section 3.3 we outline some technical details.\n\n3.1 LF-Net: a Local Feature Network\n\nLF-Net has two main components. The \ufb01rst one is a dense, multi-scale, fully convolutional network\nthat returns keypoint locations, scales, and orientations. It is designed to achieve fast inference time,\nand to be agnostic to image size. The second is a network that outputs local descriptors given patches\ncropped around the keypoints produced by the \ufb01rst network. We call them detector and descriptor.\n\n3\n\nGradient FlowGradient FlowDet.Det.warpscoremapcleaning\u2112\ud835\udc56\ud835\udc56\ud835\udc56\ud835\udc56Desc.Desc.warp\ud835\udc6b\ud835\udc6b\ud835\udc56\ud835\udc56\ud835\udc58\ud835\udc58\ufffd\ud835\udc6b\ud835\udc6b\ud835\udc57\ud835\udc57\ud835\udc58\ud835\udc58\u2112\ud835\udc5d\ud835\udc5d\ud835\udc5d\ud835\udc5d\ud835\udc56\ud835\udc56\ud835\udc5d\ud835\udc5d\ufffd\ud835\udc6b\ud835\udc6b\ud835\udc57\ud835\udc57\ud835\udc58\ud835\udc58\u2032\u2112\ud835\udc61\ud835\udc61\ud835\udc5d\ud835\udc5d\ud835\udc56\ud835\udc56G. Flow\ufffd\ud835\udc65\ud835\udc65\ud835\udc57\ud835\udc57,\ufffd\ud835\udc66\ud835\udc66\ud835\udc57\ud835\udc57,\ud835\udc60\ud835\udc60\ud835\udc57\ud835\udc57,\ud835\udf03\ud835\udf03\ud835\udc57\ud835\udc57\ud835\udc3c\ud835\udc3c\ud835\udc56\ud835\udc56\ud835\udc3c\ud835\udc3c\ud835\udc57\ud835\udc57\fIn the remainder of this section, we assume that the images have been undistorted using the camera\ncalibration data. We convert them to grayscale for simplicity and simply normalize them individually\nusing their mean and standard deviation [42]. As will be discussed in Section 4.1, depth maps\nand camera parameters can all be obtained using off-the-shelf SfM algorithms [34]. As depth\nmeasurements are often missing around 3D object boundaries\u2014especially when computed SfM\nalgorithms\u2014image regions for which we do not have depth measurements are masked and discarded\nduring training.\n\nFeature map generation. We \ufb01rst use a fully convolutional network to generate a rich feature map\no from an image I, which can be used to extract keypoint locations as well as their attributes, i.e.,\nscale and orientation. We do this for two reasons. First, it has been shown that using such a mid-level\nrepresentation to estimate multiple quantities helps increase the predictive power of deep nets [21].\nSecond, it allows for larger batch sizes, that is, using more images simultaneously, which is key to\ntraining a robust detector.\nIn practice, we use a simple ResNet [15] layout with three blocks. Each block contains 5 \u21e5 5\nconvolutional \ufb01lters followed by batch normalization [17], leaky-ReLU activations, and another set\nof 5 \u21e5 5 convolutions. All convolutions are zero-padded to have the same output size as the input,\nand have 16 output channels. In our experiments, this has proved more successful that more recent\narchitectures relying on strided convolutions and pixel shuf\ufb02ing [11].\n\nScale-invariant keypoint detection. To detect scale-invariant keypoints we propose a novel ap-\nproach to scale-space detection that relies on the feature map o. To generate a scale-space response,\nwe resize it N times, at uniform intervals between 1/R and R, where N = 5 and R = p2 in our\nexperiments. These are convolved with N independent 5 \u21e5 5 \ufb01lters size, which results in N score\nmaps hn for 1 \uf8ff n < N, one for each scale. To increase the saliency of keypoints, we perform\na differentiable form of non-maximum suppression by applying a softmax operator over 15\u21e515\nwindows in a convolutional manner, which results in N sharper score maps, \u02c6hn\n1\uf8ffn<N. Since the\nnon-maximum suppression results are scale-dependent, we resize each \u02c6hn back to the original image\nsize, which yields \u00afhn\n1\uf8ffn<N. Finally, we merge all the \u00afhn into a \ufb01nal scale-space score map, S, with\na softmax-like operation. We de\ufb01ne it as\n\nwhere  is the Hadamard product.\nFrom this scale-invariant map we choose the top K pixels as keypoints, and further apply a local\nsoftargmax [8] for sub-pixel accuracy. While selecting the the top K keypoints is not differentiable,\nthis does not stop gradients from back-propagating through the selected points. Furthermore, the\nsub-pixel re\ufb01nement through softargmax also makes it possible for gradients to \ufb02ow through with\nrespect to keypoint coordinates.To predict the scale at each keypoint, we simply apply a softargmax\noperation over the scale dimension of \u00afhn. A simpler alternative would have been to directly regress\nthe scale once a keypoint has been detected. However, this turned out to be less effective in practice.\n\nOrientation estimation. To learn orientations we follow the approach of [48, 50], but on the shared\nfeature representation o instead of the image. We apply a single 5\u21e5 5 convolution on o which outputs\ntwo values for each pixel. They are taken to be the sine and cosine of the orientation and and used to\ncompute a dense orientation map \u2713 using the arctan function.\n\nDescriptor extraction. As discussed above, we extract from the score map S the K highest scoring\nfeature points and their image locations. With the scale map s and orientation map \u2713, this gives us K\nquadruplets of the form pk = {x, y, s, \u2713}k, for which we want to compute descriptors.\nTo this end, we consider image patches around the selected keypoint locations. We crop them from\nthe normalized images and resize them to 32 \u21e5 32. To preserve differentiability, we use the bilinear\nsampling scheme of [18] for cropping. Our descriptor network comprises three 3 \u21e5 3 convolutional\n\ufb01lters with a stride of 2 and 64, 128, and 256 channels respectively. Each one is followed by batch\nnormalization and a ReLU activation. After the convolutional layers, we have a fully-connected\n512-channel layer, followed by batch normalization, ReLU, and a \ufb01nal fully-connected layer to\nreduce the dimensionality to M=256. The descriptors are l2 normalized and we denote them as D.\n\n4\n\nS =Xn\n\n\u00afhn  softmaxn\u00afhn ,\n\n(1)\n\n\f3.2 Learning LF-Net\nAs shown in Fig. 1, we formulate the learning problem in terms of a two-branch architecture which\ntakes as input two images of the same scene, Ii and Ij, i 6= j, along with their respective depth maps\nand the camera intrinsics and extrinsics, which can be obtained from conventional SfM methods.\nGiven this data, we can warp the score maps to determine ground truth correspondences between\nimages. One distinctive characteristic of our setup is that branch j holds the components which break\ndifferentiability, and is thus never back-propagated, in contrast to a traditional Siamese architecture.\nTo do this in a mathematically sound way, we take inspiration from Q-learning [27] and use the\nparameters of the previous iteration of the network for this branch.\nWe formulate our training objective as a combination of two types of loss functions: image-level and\npatch-level. Keypoint detection requires image-level operations and also affects where patches are\nextracted, thus we use both image-level and patch-level losses. For the descriptor network, we use\nonly patch-level losses as they operate independently for each patch once keypoints are selected.\n\nImage-level loss. We can warp a score map with rigid-body transforms [12], using the projective\ncamera model. We call this the SE(3) module w, which in addition to the score maps takes as input\nthe camera pose P, calibration matrix K and depth map Z, for both images\u2014note that we omit the\nlatter three for brevity. We propose to select K keypoints from the warped score map for Ij with\nstandard, non-differentiable non-maximum suppression, and generate a clean score map by placing\nGaussian kernels with standard deviation  = 0.5 at those locations. We denote this operation g.\nNote that while it is non-differentiable, it only takes place on branch j, and thus has no effect in the\noptimization. Mathematically, we write\n\nHere, as mentioned before, occluded image regions are not used for optimization.\n\nLim(Si, Sj) = |Si  g(w(Sj))|2 .\n\n(2)\n\nPatch-wise loss. With existing methods [3, 25, 36], the pool of pair-wise relationships is prede\ufb01ned\nbefore training, assuming a detector is given. More importantly, forming these pairs from two\ndisconnected sets of keypoints will produce too many outliers for the training to ever converge.\nFinally, we want the gradients to \ufb02ow back to the keypoint detector network, so that we are able to\nlearn keypoints that are good for matching.\nWe propose to solve this problem by leveraging the ground truth camera motion and depth to form\nsparse patch correspondences on the \ufb02y, by warping the detected keypoints. Note that we are only\nable to do this as we warp over branch j and back-propagate through branch i.\nMore speci\ufb01cally, once K keypoints are selected from Ii, we warp their spatial coordinates to Ij,\nsimilarly as we do for the score maps to compute the image-level loss, but in the opposite direction.\nNote that we form the keypoint with scale and orientation from branch j, as they are not as sensitive\nas the location, and we empirically found that it helps the optimization. We then extract descriptors\nat these corresponding regions pk\nj . If a keypoint falls on occluded regions after warping,\nwe drop it from the optimisation process. With these corresponding regions and their associated\ndecriptors Dk\nj we form Lpair which is used to train the detector network, i.e., the keypoint,\norientation, and scale components. Mathematically we write\n|Dk\n\nLpair(Dk\n\ni and \u02c6Dk\n\ni  \u02c6Dk\n\nj|2 .\n\ni and \u02c6pk\n\ni , \u02c6Dk\n\n(3)\n\nj ) =Xk\n\nSimilarly, in addition to the descriptors, we also enforce geometrical consistency over the orientation\nof the detected and warped points. We thus write\n\nwhere \u02c6s and \u02c6\u2713 respectively denotes the warped scale and orientation of a keypoint, by using the\nrelative camera pose between the two images, and ori and scale are weights.\n\nTriplet loss for descriptors. To learn the descriptor, we also need to consider non-corresponding\npairs of patches. Similar to [3], we form a triplet loss to learn the ideal embedding space for the\n\n5\n\nLgeom(sk\n\ni ,\u2713 k\n\ni , \u02c6sk\n\ni , \u02c6\u2713k\n\nj ) = oriXk\n\ni  \u02c6\u2713k\n|\u2713k\n\nj |2 + scaleXk\n\n|sk\ni  \u02c6sk\n\nj|2 ,\n\n(4)\n\n\fi  \u02c6Dk\n\ni  \u02c6Dk0\n\n(5)\n\nj ) =Xk\n\nLtri(Dk\n\ni , \u02c6Dk\n\nj , \u02c6Dk0\n\nmax\u21e30,|Dk\n\nIn summary, the loss function that is used to learn each\n\npatches. However, for the positive pair we use the ground-truth geometry to \ufb01nd a match, as described\nabove. For the negative\u2014non-matching\u2014pairs, we employ a progressive mining strategy to obtain\nthe most informative patches possible. Speci\ufb01cally, we sort the negatives for each sample by loss in\ndecreasing order and sample randomly over the top M, where M = max(5, 64e 0.6k\n1000 ), where k is the\ncurrent iteration, i.e., we start with a pool of the 64 hardest samples and reduce it as the networks\nconverge, up to a minimum of 5. Sampling informative patches is critical to learn discriminative\ndescriptors, and random sampling will provide too many easy negative samples.\nWith the matching and non-matching pairs, we form the triplet loss as:\nj|2 | Dk\n\nj |2 + C\u2318 .\nwhere k0 6= k, i.e., it can be any non-corresponding sample, and C=1 is the margin.\nLoss function for each sub-network.\nsub-network is the following:\n\u2022 Detector loss: Ldet = Lim + pairLpair + Lgeom\n\u2022 Descriptor loss: Ldesc = Ltri\n3.3 Technical details, otimization, and inference\nTo make the optimization more stable, we \ufb02ip the images on each branch and merge the gradients\nbefore updating. We emphasize here that with our loss, the gradients for the patch-wise loss can\nsafely back-propagate through branch i, including the top K selection, to the image-level networks.\nLikewise, the softargmax operator used for keypoint extraction allows the optimization to differentiate\nthe patch-wise loss with respect to the location of the keypoints.\nNote that for inference we keep a single copy of the network, i.e., the architecture of Fig. 1-(a), and\nsimply run the differentiable part of the framework, branch i. Although differentiability is no longer\na concern, we still rely, for simplicity, on the spatial SoftMax for non-maximum supression and the\nsoftargmax and spatial transformers for patch sampling. Even so, our implementation can extract 512\nkeypoints from QVGA frames (320\u21e5240) at 62 fps and from VGA frames (640\u21e5480) at 25 fps (42\nand 20 respectively for 1024 keypoints), on a Titan X PASCAL. Please refer to the supplementary\nmaterial for a thorough comparison of computational costs.\nWhile training we extract 512 keypoints, as larger numbers become problematic due to memory\nconstraints. This also allows us to maintain a batch with multiple image pairs (6), which helps\nconvergence. Note that at test time we can choose as many keypoints as desired. As datasets\nwith natural images are composed of mostly upright images and are thus rather biased in terms of\norientation, we perform data augmentation by randomly rotating the input patches by \u00b1180o, and\ntransform the camera\u2019s roll angle accordingly. We also perform scale augmentation by resizing the\ninput patches by 1/p2 to p2, and transforming the focal length accordingly. This allows us to\ntrain models comparable to traditional keypoint extraction pipelines, i.e., with built-in invariance to\nrotation and scaling. However, in practice, many of the images in our indoors and outdoors examples\nare upright, and the best-performing models are obtained by disabling these augmentations as well\nas the orientation ans scale estimation completely. This is the approach followed by learned SLAM\nfront-ends such as [11]. We consider both strategies in the next section.\nFor optimization, we use ADAM [20] with a learning rate of 103. To balance the loss function for\nthe detector network we use pair = 0.01, and ori = scale = 0.1. Our implementation is written\nin TensorFlow and is publicly available.1\n\n4 Experiments\n\n4.1 Datasets\nWe consider both indoors and outdoors images as their characteristics drastically differ, as shown\nin Fig. 2. For indoors data we rely on ScanNet [10], an RGB-D dataset with over 2.5M images,\n\n1https://github.com/vcg-uvic/lf-net-release\n\n6\n\n\fFigure 2: Samples from our indoors and outdoors datasets. Image regions without depth measure-\nments, due to occlusions or sensor shortcomings, are drawn in red, and are simply excluded from the\noptimization. Note the remaining artefacts in the depth maps for outdoors images.\n\nincluding accurate camera poses from SfM reconstructions. These sequences show of\ufb01ce settings\nwith specularities and very signi\ufb01cant blurring artefacts, and the depth maps are incomplete due to\nsensing failures, specially around 3D object boundaries. The dataset provides training, validation,\nand test splits that we use accordingly. As this dataset is very large, we only use roughly half of the\navailable sequences for training and validation, but test on the entire set of 312 sequences, pre-selected\nby the authors, with the exception of LIFT. For LIFT we use a random subset of 43 sequences as the\nauthors\u2019 implementation is too slow. To prevent selecting pairs of images that do not share any \ufb01eld\nof view, we sample images 15 frames away, guaranteeing enough scene overlap. At test time, we\nconsider multiple values for the frame difference to evaluate increasing baselines.\nFor outdoors data we use 25 photo-tourism image collections of popular landmarks collected by [16,\n39]. We run COLMAP [34] to obtain dense 3D reconstructions, including dense but noisy and\ninaccurate depth maps for every image. We post-process the depth maps by projecting each image\npixel to 3D space at the estimated depth, and mark it as invalid if the closest 3D point from the\nreconstruction is further than a threshold. The resulting depth maps are still noisy, but many occluded\npixels are \ufb01ltered out as shown in Fig. 2. To guarantee a reasonable degree of overlap for each\nimage pair we perform a visibility check using the SfM points visible over both images. We consider\nbounding boxes twice the size of those containing these points to extract image regions roughly\ncorresponding, while ignoring very small ones. We use 14 sequences for training and validation,\nspliting the images into training and validation subsets by with a 70:30 ratio, and sample up to 50k\npairs from each different scene. For testing we use the remaining 11 sequences, which were not\nused for training or validation, and sample up to 1k pairs from each set. We use square patches size\n256 \u21e5 256 for training, for either data type.\n4.2 Baselines and metrics\n\nWe consider the following full local feature pipelines: SIFT [23], SURF [6] ORB [32], A-KAZE [2],\nLIFT [48], and SuperPoint [11], using the authors\u2019 release for the learned variants, LIFT and Super-\nPoint, and OpenCV for the rest. For ScanNet, we test on 320\u21e5240 images, which is commensurate\nwith the patches cropped while training. We do the same for the baselines, as their performance\nseems to be better than at higher resolutions, probably due to the low-texture nature of the images.\nFor the outdoors dataset, we resize the images so that the largest dimensions is 640 pixels, as they are\nricher in texture, and all methods work better at this resolution. Similarly, we extract 1024 keypoints\nfor outdoors images, but limit them to 512 for Scannet, as the latter contains very little texture.\nTo evaluate the entire local feature pipeline performance, we use the matching score [24], which\nis de\ufb01ned as the ratio of estimated correspondences that are correct according to the ground-truth\ngeometry, after obtaining them through nearest neighbour matching with the descriptors. As our\ndata exhibits complex geometry, and to emphasize accurate localization of keypoints, similar to [31]\nwe use a 5-pixel threshold instead of the overlap measure used in [24]. For results under different\nthresholds please refer to the supplementary material.\n\n4.3 Results on outdoors data\n\nFor this experiment we provide results independently for each sequence, in addition to the average.\nDue to the nature of the data, results vary from sequence to sequence. We provide quantitative results\n\n7\n\n\fTable 1: Matching score for the outdoors dataset. Best results are marked in bold.\n\nSequence\n\nSIFT SURF A-KAZE ORB LIFT SuperPoint\n\nLF-Net\n\nw/rot-scl w/o rot-scl\n\n\u2018british_museum\u2019\n\u2018\ufb02orence_cathedral_side\u2019\n\u2018lincoln_memorial_statue\u2019\n\u2018london_bridge\u2019\n\u2018milan_cathedral\u2019\n\u2018mount_rushmore\u2019\n\u2018piazza_san_marco\u2019\n\u2018reichstag\u2019\n\u2018sagrada_familia\u2019\n\u2018st_pauls_cathedral\u2019\n\u2018united_states_capitol\u2019\nAverage\n\n.265\n.181\n.193\n.177\n.188\n.225\n.115\n.212\n.199\n.149\n.118\n.184\n\n.288\n.158\n.204\n.170\n.221\n.241\n.115\n.209\n.175\n.160\n.103\n.186\n\n.287\n.116\n.167\n.168\n.194\n.210\n.106\n.175\n.140\n.150\n.086\n.164\n\n.055\n.027\n.037\n.057\n.021\n.041\n.026\n.097\n.031\n.026\n.028\n.041\n\n.318\n.204\n.220\n.250\n.237\n.300\n.145\n.246\n.205\n.177\n.134\n.221\n\n.468\n.359\n.384\n.468\n.401\n.512\n.253\n.414\n.295\n.319\n.220\n.372\n\n.456\n.285\n.288\n.342\n.423\n.379\n.233\n.379\n.311\n.266\n.173\n.321\n\n.560\n.362\n.357\n.452\n.520\n.543\n.287\n.466\n.341\n.347\n.232\n.406\n\nTable 2: Matching score for the indoors dataset. Best results are marked in bold.\n\nFrame difference SIFT SURF A-KAZE ORB LIFT SuperPoint\n\nLF-Net\n\n(w/rot-scl)\n\n(w/o rot-scl)\n\n10\n20\n30\n60\nAverage\n\n.320\n.264\n.226\n.152\n.241\n\n.464\n.357\n.290\n.179\n.323\n\n.465\n.337\n.260\n.145\n.302\n\n.223\n.172\n.141\n.089\n.156\n\n.389\n.283\n.247\n.147\n.267\n\n.688\n.599\n.525\n.358\n.542\n\n.607\n.497\n.419\n.276\n.450\n\n.688\n.574\n.483\n.300\n.511\n\nin Table 1 and qualitative examples in Fig. 3. As previously noted in Section 3.3, most images are\nupright and at similar scales, so that the best results are obtained simply bypassing scale and rotation\nestimation. In order to compare with SuperPoint, we train our models in this setup (\u2018w/o rot-scl\u2019).\nLF-Net gives best performance and outperforms SuperPoint by 9% relative. In order to compare with\ntraditional pipelines with explicit rotation and scale estimation, we also train our models with the\naugmentations described in section Section 3.3 (\u2018w/ rot-scl\u2019). Out of these baselines, our approach\noutperforms the closest competitor, LIFT, by 45% relative. We provide qualitative examples in Fig. 3.\n\n4.4 Results on indoors data\n\nThis dataset contains video sequences. To evaluate performance over different baselines, we sample\nimage pairs at different frame difference values: 10, 20, 30, and 60. At 10 the images are very\nsimilar, whereas at 60 there is a signi\ufb01cant degree of camera motion\u2014note that our method is trained\nexclusively at a 15-frame difference. Results are shown in Table 2. As for the outdoors case, the\nimages in this dataset are mostly upright, so we report results for models trained with and without\nexplicit rotation and scale detection. LF-Net achieves the same performance as SuperPoint on the\n10-frame difference case, but performs worse for larger frame differences\u20146% relative on average.\nWe believe this to be due to the fact that in of\ufb01ce-like indoor settings with little texture such as the\nones common in this dataset, 3D object boundaries are often the most informative features, and they\nmay be in many cases excluded from LF-Net training due to depth inaccuracies from the Kinect\nsensor. Note however that the self-supervised training strategy used by SuperPoint can also be applied\nto LF-Net to boost performance. Finally, among pipelines with explicit rotation and scale estimation,\nLF-Net outperforms the closest competitor, SURF, by 39% relative.\n\n4.5 Ablation study\n\nAs demonstrated in [48], it is crucial to train the different components jointly when learning a feature\nextraction pipeline. As an ablation study, we consider the case where pair = 0, i.e., we do not\ntrain the detector with the patch-wise loss, effectively separating the training of the detector and the\n\n8\n\n\f(a) SIFT\n\n(b) SURF\n\n(c) A-KAZE\n\n(d) LF-Net (ours)\n\nFigure 3: Qualitative matching results, with correct matches drawn in green.\n\ndescriptor. For this experiment we consider models trained with rotation and scale augmentations.\nWhile they still produce state-of-the-art results w.r.t. the rotation-sensitive baselines, they tend to\nconverge earlier, and training them jointly increases average matching score by 7% relative for the\noutdoors dataset (.299 to .321) and 1% relative for the indoors dataset (.445 to .450). Again, we\nbelieve that the small performance increase on indoors data is due to the inherent limits of the device\nused to capture the depth estimates. For more detailed results please refer to the supplementary\nappendix.\n\n4.6 Additional results\n\nMore experimental results that could not \ufb01t in the paper due to spatial constraints are available as\na supplementary appendix: a generalization study, training and testing on very different datasets; a\nperformance evaluation for different pixel thresholds, to evaluate the precision of the detected features;\nresults over the \u2018hpatches\u2019 dataset [4]; a study to benchmark performance under orientation changes;\ndetailed results for the ablation experiments of Section 4.5; and computational cost estimates.\n\n5 Conclusions\n\nWe have proposed LF-Net, a novel deep architecture to learn local features. It embeds the entire\nfeature extraction pipeline, and can be trained end-to-end with just a collection of images. To allow\ntraining from scratch without hand-crafted priors, we devise a two-branch setup and create virtual\ntarget responses iteratively. We run this non-differentiable process in one branch while optimizing\nover the other, which we keep differentiable, and show they converge to an optimal solution. Our\nmethod outperforms the state of the art by a large margin, on both indoor and outdoor datasets, at 60\nfps for QVGA images.\n\nAcknowledgments\nThis work was partially supported by the Natural Sciences and Engineering Research Council of\nCanada (NSERC) Discovery Grant \u201cDeep Visual Geometry Machines\u201d (RGPIN-2018-03788), and\nby systems supplied by Compute Canada.\n\nReferences\n[1] P. Alcantarilla, P. Fern\u00e1ndez, A. Bartoli, and A. J. Davidson. KAZE Features. In ECCV, 2012. 2\n\n9\n\n\f2008. 2\n\n2\n\n5\n\n[2] P. F. Alcantarilla, J. Nuevo, and A. Bartoli. Fast Explicit Diffusion for Accelerated Features in Nonlinear\n\nScale Spaces. In BMVC, 2013. 7\n\n[3] V. Balntas, E. Johns, L. Tang, and K. Mikolajczyk. PN-Net: Conjoined Triple Deep Network for Learning\n\nLocal Image Descriptors. In arXiv Preprint, 2016. 2, 5\n\n[4] V. Balntas, K. Lenc, A. Vedaldi, and K. Mikolajczyk. Hpatches: A benchmark and evaluation of handcrafted\n\nand learned local descriptors. In CVPR, 2017. 9, 13\n\n[5] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool. SURF: Speeded Up Robust Features. CVIU, 10(3):346\u2013359,\n\n[6] H. Bay, T. Tuytelaars, and L. Van Gool. SURF: Speeded Up Robust Features. In ECCV, 2006. 7\n[7] M. Brown, G. Hua, and S. Winder. Discriminative Learning of Local Image Descriptors. PAMI, 2011. 3\n[8] O. Chapelle and M. Wu. Gradient Descent Optimization of Smoothed Information Retrieval Metrics.\n\nInformation Retrieval, 13(3):216\u2013235, 2009. 4\n\n[9] C. Choy, J. Gwak, S. Savarese, and M. Chandraker. Universe Correspondence Network. In NIPS, 2016. 1,\n\n[10] A. Dai, A. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nie\u00dfner. Scannet: Richly-Annotated 3D\n\nReconstructions of Indoor Scenes. In CVPR, 2017. 6\n\n[11] D. Detone, T. Malisiewicz, and A. Rabinovich. Superpoint: Self-Supervised Interest Point Detection and\n\nDescription. CVPR Workshop on Deep Learning for Visual SLAM, 2018. 1, 3, 4, 6, 7\n\n[12] J. Engel, T. Sch\u00f6ps, and D. Cremers. LSD-SLAM: Large-Scale Direct Monocular SLAM. In ECCV, 2014.\n\n[13] X. Han, T. Leung, Y. Jia, R. Sukthankar, and A. C. Berg. MatchNet: Unifying Feature and Metric Learning\n\nfor Patch-Based Matching. In CVPR, 2015. 2\n\n[14] R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press,\n\n[15] K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In CVPR, pages\n\n[16] J. Heinly, J. Schoenberger, E. Dunn, and J.-M. Frahm. Reconstructing the World in Six Days. In CVPR,\n\n770\u2013778, 2016. 4\n\n2000. 2\n\n2015. 7\n\n[17] S. Ioffe and C. Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal\n\n2004. 7\n\n[18] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. Spatial Transformer Networks. In NIPS,\n\n[19] M. Keller, Z. Chen, F. Maffra, P. Schmuck, and M. Chli. Learning Deep Descriptors with Scale-Aware\n\nCovariate Shift. In ICML, 2015. 4\n\npages 2017\u20132025, 2015. 4\n\nTriplet Networks. In CVPR, 2018. 2\n\n[20] D. Kingma and J. Ba. Adam: A Method for Stochastic Optimisation. In ICLR, 2015. 6\n[21] I. Kokkinos. Ubernet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level\n\nVision Using Diverse Datasets and Limited Memory. In CVPR, 2017. 4\n\n[22] K. Lenc and A. Vedaldi. Learning Covariant Feature Detectors. In ECCV, 2016. 2\n[23] D. Lowe. Distinctive Image Features from Scale-Invariant Keypoints. IJCV, 20(2), 2004. 1, 2, 7\n[24] K. Mikolajczyk and C. Schmid. A Performance Evaluation of Local Descriptors. PAMI, 27(10):1615\u20131630,\n\n[25] A. Mishchuk, D. Mishkin, F. Radenovic, and J. Matas. Working Hard to Know Your Neighbor\u2019s Margins:\n\nLocal Descriptor Learning Loss. In NIPS, 2017. 2, 5\n\n[26] D. Mishkin, F. Radenovic, and J. Matas. Repeatability Is Not Enough: Learning Af\ufb01ne Regions via\n\nDiscriminability. In ECCV, 2018. 2\n\n[27] V. Mnih, K. Kavukcuoglu, D. Silver, A. Rusu, J. Veness, M. Bellemare, A. Graves, M. Riedmiller,\nA. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran,\nD. Wierstra, S. Legg, and D. Hassabis. Human-Level Control through Deep Reinforcement Learning.\nNature, 518(7540):529\u2013533, February 2015. 5\n\n[28] D. Mukherjee, Q. M. J. Wu, and G. Wang. A Comparative Experimental Study of Image Feature Detectors\n\nand Descriptors. MVA, 26(4):443\u2013466, 2015. 2\n\n[29] R. Mur-artal, J. Montiel, and J. Tard\u00f3s. Orb-Slam: A Versatile and Accurate Monocular Slam System.\n\nIEEE Transactions on Robotics, 31(5):1147\u20131163, 2015. 2\n\n[30] E. Rosten and T. Drummond. Machine Learning for High-Speed Corner Detection. In ECCV, 2006. 2\n[31] E. Rosten, R. Porter, and T. Drummond. Faster and Better: A Machine Learning Approach to Corner\n\nDetection. PAMI, 32:105\u2013119, 2010. 2, 7\n\n[32] E. Rublee, V. Rabaud, K. Konolidge, and G. Bradski. ORB: An Ef\ufb01cient Alternative to SIFT or SURF. In\n\nICCV, 2011. 2, 7\n\n[33] N. Savinov, A. Seki, L. Ladicky, T. Sattler, and M. Pollefeys. Quad-Networks: Unsupervised Learning to\n\nRank for Interest Point Detection. CVPR, 2017. 1, 2\n\n[34] J. Sch\u00f6nberger and J. Frahm. Structure-From-Motion Revisited. In CVPR, 2016. 1, 4, 7\n[35] J. Sch\u00f6nberger, H. Hardmeier, T. Sattler, and M. Pollefeys. Comparative Evaluation of Hand-Crafted and\n\nLearned Local Features. In CVPR, 2017. 2\n\n[36] E. Simo-serra, E. Trulls, L. Ferraz, I. Kokkinos, P. Fua, and F. moreno-noguer. Discriminative Learning of\n\nDeep Convolutional Feature Point Descriptors. In ICCV, 2015. 2, 3, 5\n\n[37] K. Simonyan, A. Vedaldi, and A. Zisserman. Learning Local Feature Descriptors Using Convex Optimisa-\n\ntion. PAMI, 2014. 2\n\n10\n\n\f[38] C. Strecha, A. Bronstein, M. Bronstein, and P. Fua. LDAHash: Improved Matching with Smaller\n\nDescriptors. PAMI, 34(1), January 2012. 2\n\n[39] B. Thomee, D. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L. Li. YFCC100M:\n\nthe New Data in Multimedia Research. In CACM, 2016. 7\n\n[40] Y. Tian, B. Fan, and F. Wu. L2-Net: Deep Learning of Discriminative Patch Descriptor in Euclidean Space.\n\n[41] E. Tola, V. Lepetit, and P. Fua. Daisy: An Ef\ufb01cient Dense Descriptor Applied to Wide Baseline Stereo.\n\nIn CVPR, 2017. 2\n\nPAMI, 32(5):815\u2013830, 2010. 2\n\n[42] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Improved Texture Networks: Maximizing Quality and Diversity\n\nin Feed-Forward Stylization and Texture Synthesis. In CVPR, 2017. 4\n\n[43] B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, E. Ilg, A. Dosovitskiy, and T. Brox. Demon: Depth and\n\nMotion Network for Learning Monocular Stereo. In CVPR, 2017. 1, 2\n\n[44] Y. Verdie, K. M. Yi, P. Fua, and V. Lepetit. TILDE: A Temporally Invariant Learned DEtector. In CVPR,\n\n[45] S. Vijayanarasimhan, S. Ricco, C. Schmid, R. Sukthankar, and K. Fragkiadaki. Sfm-Net: Learning of\n\nStructure and Motion from Video. arXiv Preprint, 2017. 1, 2\n\n[46] X. Wei, Y. Zhang, Y. Gong, and N. Zheng. Kernelized Subspace Pooling for Deep Local Descriptors. In\n\n2015. 2, 3\n\nCVPR, 2018. 2\n\n2, 3, 4, 7, 8\n\nIn CVPR, 2018. 1\n\n2016. 2, 3, 4\n\nIn CVPR, 2015. 2\n\n[47] C. Wu. Towards Linear-Time Incremental Structure from Motion. In 3DV, 2013. 2\n[48] K. M. Yi, E. Trulls, V. Lepetit, and P. Fua. LIFT: Learned Invariant Feature Transform. In ECCV, 2016. 1,\n\n[49] K. M. Yi, E. Trulls, Y. Ono, V. Lepetit, M. Salzmann, and P. Fua. Learning to Find Good Correspondences.\n\n[50] K. M. Yi, Y. Verdie, P. Fua, and V. Lepetit. Learning to Assign Orientations to Feature Points. In CVPR,\n\n[51] S. Zagoruyko and N. Komodakis. Learning to Compare Image Patches via Convolutional Neural Networks.\n\n[52] A. R. Zamir, T. Wekel, P. Agrawal, J. Malik, and S. Savarese. Generic 3D Representation via Pose\n\nEstimation and Matching. In ECCV, 2016. 1, 2\n\n[53] T. Zhou, M. Brown, N. Snavely, and D. Lowe. Unsupervised Learning of Depth and Ego-Motion from\n\nVideo. In CVPR, 2017. 1, 2\n\n11\n\n\f", "award": [], "sourceid": 3068, "authors": [{"given_name": "Yuki", "family_name": "Ono", "institution": "SONY"}, {"given_name": "Eduard", "family_name": "Trulls", "institution": "EPFL"}, {"given_name": "Pascal", "family_name": "Fua", "institution": "EPFL, Switzerland"}, {"given_name": "Kwang Moo", "family_name": "Yi", "institution": "University of Victoria"}]}