{"title": "Fast and Flexible Multi-Task Classification using Conditional Neural Adaptive Processes", "book": "Advances in Neural Information Processing Systems", "page_first": 7959, "page_last": 7970, "abstract": "The goal of this paper is to design image classification systems that, after an initial multi-task training phase, can automatically adapt to new tasks encountered at test time. We introduce a conditional neural process based approach to the multi-task classification setting for this purpose, and establish connections to the meta- and few-shot learning literature. The resulting approach, called CNAPs, comprises a classifier whose parameters are modulated by an adaptation network that takes the current task's dataset as input. We demonstrate that CNAPs achieves state-of-the-art results on the challenging Meta-Dataset benchmark indicating high-quality transfer-learning. We show that the approach is robust, avoiding both over-fitting in low-shot regimes and under-fitting in high-shot regimes. Timing experiments reveal that CNAPs is computationally efficient at test-time as it does not involve gradient based adaptation. Finally, we show that trained models are immediately deployable to continual learning and active learning where they can outperform existing approaches that do not leverage transfer learning.", "full_text": "Fast and Flexible Multi-Task Classi\ufb01cation Using\n\nConditional Neural Adaptive Processes\n\nJames Requeima\u2217\n\nUniversity of Cambridge\n\nInvenia Labs\n\njrr41@cam.ac.uk\n\nJonathan Gordon\u2217\n\nUniversity of Cambridge\n\njg801@cam.ac.uk\n\nJohn Bronskill\u2217\n\nUniversity of Cambridge\n\njfb54@cam.ac.uk\n\nSebastian Nowozin\n\nGoogle Research Berlin\nnowozin@google.com\n\nRichard E. Turner\n\nUniversity of Cambridge\n\nMicrosoft Research\nret26@cam.ac.uk\n\nAbstract\n\nThe goal of this paper is to design image classi\ufb01cation systems that, after an initial\nmulti-task training phase, can automatically adapt to new tasks encountered at test\ntime. We introduce a conditional neural process based approach to the multi-task\nclassi\ufb01cation setting for this purpose, and establish connections to the meta-learning\nand few-shot learning literature. The resulting approach, called CNAPS, comprises\na classi\ufb01er whose parameters are modulated by an adaptation network that takes the\ncurrent task\u2019s dataset as input. We demonstrate that CNAPS achieves state-of-the-\nart results on the challenging META-DATASET benchmark indicating high-quality\ntransfer-learning. We show that the approach is robust, avoiding both over-\ufb01tting\nin low-shot regimes and under-\ufb01tting in high-shot regimes. Timing experiments\nreveal that CNAPS is computationally ef\ufb01cient at test-time as it does not involve\ngradient based adaptation. Finally, we show that trained models are immediately\ndeployable to continual learning and active learning where they can outperform\nexisting approaches that do not leverage transfer learning.\n\n1\n\nIntroduction\n\nWe consider the development of general purpose image classi\ufb01cation systems that can handle tasks\nfrom a broad range of data distributions, in both the low and high data regimes, without the need for\ncostly retraining when new tasks are encountered. We argue that such systems require mechanisms\nthat adapt to each task, and that these mechanisms should themselves be learned from a diversity of\ndatasets and tasks at training time. This general approach relates to methods for meta-learning [1, 2]\nand few-shot learning [3]. However, existing work in this area typically considers homogeneous task\ndistributions at train and test-time that therefore require only minimal adaptation. To handle the more\nchallenging case of different task distributions we design a fully adaptive system, requiring speci\ufb01c\ndesign choices in the model and training procedure.\nCurrent approaches to meta-learning and few-shot learning for classi\ufb01cation are characterized by two\nfundamental trade-offs. (i) The number of parameters that are adapted to each task. One approach\nadapts only the top, or head, of the classi\ufb01er leaving the feature extractor \ufb01xed [4, 5]. While useful in\nsimple settings, this approach is prone to under-\ufb01tting when the task distribution is heterogeneous\n[6]. Alternatively, we can adapt all parameters in the feature extractor [7, 8] thereby increasing\n\n\u2217Authors contributed equally\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\f(a)\n\n(b)\n\nFigure 1: (a) Probabilistic graphical model detailing the CNP [13] framework. (b) Computational diagram\ndepicting the CNAPS model class. Red boxes imply parameters in the model architecture supplied by adaptation\nnetworks. Blue shaded boxes depict the feature extractor and the gold box depicts the linear classi\ufb01er.\n\n\ufb01tting capacity, but incurring a computation cost and opening the door to over-\ufb01tting in the low-shot\nregime. What is needed is a middle ground which strikes a balance between model capacity and\nreliability of the adaptation. (ii) The adaptation mechanism. Many approaches use gradient-based\nadaptation [7, 9]. While this approach can incorporate training data in a very \ufb02exible way, it is\ncomputationally inef\ufb01cient at test-time, may require expertise to tune the optimization procedure,\nand is again prone to over-\ufb01tting. Conversely, function approximators can be used to directly map\ntraining data to the desired parameters (we refer to this as amortization) [5, 10]. This yields \ufb01xed-cost\nadaptation mechanisms, and enables greater sharing across training tasks. However, it may under-\ufb01t\nif the function approximation is not suf\ufb01ciently \ufb02exible. On the other hand, high-capacity function\napproximators require a large number of training tasks to be learned.\nWe introduce a modelling class that is well-positioned with respect to these two trade-offs for the\nmulti-task classi\ufb01cation setting called Conditional Neural Adaptive Processes (CNAPS).2 CNAPS\ndirectly model the desired predictive distribution [11, 12], thereby introducing a conditional neural\nprocesses (CNPs) [13] approach to the multi-task classi\ufb01cation setting. CNAPS handles varying\nway classi\ufb01cation tasks and introduces a parametrization and training procedure enabling the model\nto learn to adapt the feature representation for classi\ufb01cation of diverse tasks at test time. CNAPS\nutilize i) a classi\ufb01cation model with shared global parameters and a small number of task-speci\ufb01c\nparameters. We demonstrate that by identifying a small set of key parameters, the model can balance\nthe trade-off between \ufb02exibility and robustness. ii) A rich adaptation neural network with a novel\nauto-regressive parameterization that avoids under-\ufb01tting while proving easy to train in practice with\nexisting datasets [6]. In Section 5 we evaluate CNAPS. Recently, Trianta\ufb01llou et al. [6] proposed\nMETA-DATASET, a few-shot classi\ufb01cation benchmark that addresses the issue of homogeneous train\nand test-time tasks and more closely resembles real-world few-shot multi-task learning. Many of the\napproaches that achieved excellent performance on simple benchmarks struggle with this collection\nof diverse tasks. In contrast, we show that CNAPS achieve state-of-the-art performance on the\nMETA-DATASET benchmark, often by comfortable margins and at a fraction of the time required by\ncompeting methods. Finally, we showcase the versatility of the model class by demonstrating that\nCNAPS can be applied \u201cout of the box\u201d to continual learning and active learning.\n\n2 Model Design\n\nWe consider a setup where a large number of training tasks are available, each composed of a set of\ninputs x and labels y. The data for task \u03c4 includes a context set D\u03c4 = {(x\u03c4\nn=1, with inputs\nand outputs observed, and a target set {(x\u03c4\u2217m , y\u03c4\u2217m )}M\u03c4\nm=1 for which we wish to make predictions (y\u03c4\u2217\nare only observed during training). CNPs [13] construct predictive distributions given x\u2217 as:\n\nn)}N\u03c4\n\nn, y\u03c4\n\np (y\u2217|x\u2217, \u03b8, D\u03c4 ) = p (y\u2217|x\u2217, \u03b8, \u03c8\u03c4 = \u03c8\u03c6 (D\u03c4 )) .\n\n(1)\nHere \u03b8 are global classi\ufb01er parameters shared across tasks. \u03c8\u03c4 are local task-speci\ufb01c parameters,\nproduced by a function \u03c8\u03c6(\u00b7) that acts on D\u03c4 . \u03c8\u03c6(\u00b7) has another set of global parameters \u03c6 called\nadaptation network parameters. \u03b8 and \u03c6 are the learnable parameters in the model (see Figure 1a).\n\n2Source code available at https://github.com/cambridge-mlg/cnaps.\n\n2\n\nx\u03c4\u2217my\u03c4\u2217m\u03c8\u03c6(D\u03c4)D\u03c4\u03b8m=1,...\u03c4=1,...SoftmaxOutput\ud835\udc5d\ud835\udc9a\u2217\ud835\udc99\u2217,\ud835\udf3d,\ud835\udf4d\ud835\udf0f)\ud835\udf4d\ud835\udc53\ud835\udf0f\ud835\udf4d\ud835\udc64\ud835\udf0f\ud835\udc64(\u22c5;\ud835\udf4d\ud835\udc53)\ud835\udc53\ud835\udf03(\ud835\udc99\u2217;\ud835\udf4d\ud835\udc53\ud835\udf0f)\ud835\udc99\u2217\ud835\udc53\ud835\udf03(\u22c5;\ud835\udf4d\ud835\udc53)\ud835\udf4d\ud835\udc53(\u22c5;\ud835\udf19\ud835\udc53)\ud835\udf4d\ud835\udc64(\u22c5;\ud835\udf19\ud835\udc64){\ud835\udc99\ud835\udf0f}{\ud835\udc9a\ud835\udf0f}\ud835\udc53\ud835\udf03(\u22c5;\ud835\udf4d\ud835\udc53)\ud835\udf3d{\ud835\udc53\ud835\udf03(\ud835\udc99\ud835\udf0f;\ud835\udf4d\ud835\udc53\ud835\udf0f)}\ud835\udf4d\ud835\udc53\ud835\udf0f\ud835\udf3dAdaptation NetworksClassification Model\f(a) A FiLM layer.\n\n(b) A ResNet basic block with FiLM layers.\n\nFigure 2: (Left) A FiLM layer operating on convolutional feature maps indexed by channel ch. (Right) How a\nFiLM layer is used within a basic Residual network block [14].\n\nCNAPS is a model class that specializes the CNP framework for the multi-task classi\ufb01cation setting.\nThe model-class is characterized by a number of design choices, made speci\ufb01cally for the multi-task\nimage classi\ufb01cation setting. CNAPS employ global parameters \u03b8 that are trained of\ufb02ine to capture\nhigh-level features, facilitating transfer and multi-task learning. Whereas CNPs de\ufb01ne \u03c8\u03c4 to be a\n\ufb01xed dimensional vector used as an input to the model, CNAPS instead let \u03c8\u03c4 be speci\ufb01c parameters\nof the model itself. This increases the \ufb02exibility of the classi\ufb01er, enabling it to model a broader range\nof input / output distributions. We discuss our choices (and associated trade-offs) for these parameters\nbelow. Finally, CNAPS employ a novel auto-regressive parameterization of \u03c8\u03c6(\u00b7) that signi\ufb01cantly\nimproves performance. An overview of CNAPS and its key components is illustrated in Figure 1b.\n\n2.1 Speci\ufb01cation of the classi\ufb01er: global \u03b8 and task-speci\ufb01c parameters \u03c8\u03c4\n\nWe begin by specifying the classi\ufb01er\u2019s global parameters \u03b8 followed by how these are adapted by the\nlocal parameters \u03c8\u03c4 .\nGlobal Classi\ufb01er Parameters. The global classi\ufb01er parameters will parameterize a feature extractor\nf\u03b8(x) whose output is fed into a linear classi\ufb01er, described below. A natural choice for f\u03b8(\u00b7) in the\nimage setting is a convolutional neural network, e.g., a ResNet [14]. In what follows, we assume that\nthe global parameters \u03b8 are \ufb01xed and known. In Section 3 we discuss the training of \u03b8.\nTask-Speci\ufb01c Classi\ufb01er Parameters: Linear Classi\ufb01cation Weights. The \ufb01nal classi\ufb01cation layer\nmust be task-speci\ufb01c as each task involves distinguishing a potentially unique set of classes. We\nuse a task speci\ufb01c af\ufb01ne transformation of the feature extractor output, followed by a softmax. The\nw \u2208 Rdf\u00d7C\u03c4 (suppressing the biases to simplify notation), where\ntask-speci\ufb01c weights are denoted \u03c8\u03c4\ndf is the dimension of the feature extractor output f\u03b8(x) and C \u03c4 is the number of classes in task \u03c4.\nTask-Speci\ufb01c Classi\ufb01er Parameters: Feature Extractor Parameters. A suf\ufb01ciently \ufb02exible\nmodel must have capacity to adapt its feature representation f\u03b8(\u00b7) as well as the classi\ufb01cation\nlayer (e.g. compare the optimal features required for ImageNet versus Omiglot). We therefore\nf , and denote f\u03b8(\u00b7) the unadapted feature\nintroduce a set of local feature extractor parameters \u03c8\u03c4\nextractor, and f\u03b8(\u00b7; \u03c8\u03c4\nIt is critical in few-shot multi-task learning to adapt the feature extractor in a parameter-ef\ufb01cient\nmanner. Unconstrained adaptation of all the feature extractor parameters (e.g. by \ufb01ne-tuning [9])\ngives \ufb02exibility, but it is also slow and prone to over-\ufb01tting [6]. Instead, we employ linear modulation\nof the convolutional feature maps as proposed by Perez et al. [15], which adapts the feature extractor\nthrough a relatively small number of task speci\ufb01c parameters.\nA Feature-wise Linear Modulation (FiLM) layer [15] scales and shifts the ith unadapted feature map\nfi in the feature extractor FiLM(fi; \u03b3\u03c4\ni and\ni . Figure 2a illustrates a FiLM layer operating on a convolutional layer, and Figure 2b illustrates\n\u03b2\u03c4\nhow a FiLM layer can be added to a standard Residual network block [14]. A key advantage of\nFiLM layers is that they enable expressive feature adaptation while adding only a small number of\nparameters [15]. For example, in our implementation we use a ResNet18 with FiLM layers after\ni }) constitute\nevery convolutional layer. The set of task speci\ufb01c FiLM parameters (\u03c8\u03c4\nfewer than 0.7% of the parameters in the model. Despite this, as we show in Section 5, they allow the\nmodel to adapt to a broad class of datasets.\n\nf ) the feature extractor adapted to task \u03c4.\n\ni using two task speci\ufb01c parameters, \u03b3\u03c4\n\nf = {\u03b3\u03c4\n\ni , \u03b2\u03c4\n\ni , \u03b2\u03c4\n\ni ) = \u03b3\u03c4\n\ni fi + \u03b2\u03c4\n\n3\n\nFiLM\ud835\udc87\ud835\udc56\ud835\udefe\ud835\udc56,1\ud835\udefd\ud835\udc56,1+\ud835\udefe\ud835\udc56,\ud835\udc50\u210e\ud835\udefd\ud835\udc56,\ud835\udc50\u210e\u20263x3BNFiLM \ud835\udc87\ud835\udc4f1ReLU3x3BNFiLM \ud835\udc87\ud835\udc4f2+ReLUblock \ud835\udc4f\ud835\udc4f1\ud835\udc4f1\ud835\udc4f2\ud835\udc4f2\fFigure 3: Implementation of functional representation of the class-speci\ufb01c parameters \u03c8w. In this parameteriza-\ntion, \u03c8c\n\nw are the linear classi\ufb01cation parameters for class c, and \u03c6w are the learnable parameters.\n\nf , \u03c8\u03c4\n\n2.2 Computing the local parameters via adaptation networks\nThe previous sections have speci\ufb01ed the form of the classi\ufb01er p (y\u2217|x\u2217, \u03b8, \u03c8\u03c4 ) in terms of the global\nw}. The local parameters could now be learned\nand task speci\ufb01c parameters, \u03b8 and \u03c8\u03c4 = {\u03c8\u03c4\nseparately for every task \u03c4 via optimization. While in practice this is feasible for small numbers\nof tasks (see e.g., [16, 17]), this approach is computationally demanding, requires expert oversight\n(e.g. for tuning early stopping), and can over-\ufb01t in the low-data regime.\nInstead, CNAPS uses a function, such as a neural network, that takes the context set D\u03c4 as an input\nand returns the task-speci\ufb01c parameters, \u03c8\u03c4 = \u03c8\u03c6 (D\u03c4 ). The adaptation network has parameters\n\u03c6 that will be trained on multiple tasks to learn how to produce local parameters that result in\ngood generalisation, a form of meta-learning. Sacri\ufb01cing some of the \ufb02exibility of the optimisation\napproach, this method is comparatively cheap computationally (only involving a forward pass through\nthe adaptation network), automatic (with no need for expert oversight), and employs explicit parameter\nsharing (via \u03c6) across the training tasks.\nAdaptation Network: Linear Classi\ufb01er Weights. CNAPS represents the linear classi\ufb01er weights\nw = \u03c8w(D\u03c4 ; \u03c6w, \u03c8f , \u03b8), denoted \u03c8w(D\u03c4 ) for brevity.\n\u03c8\u03c4\nThere are three challenges with this approach: \ufb01rst, the dimensionality of the weights depends on the\ntask (\u03c8\u03c4\nw is a matrix with a column for each class, see Figure 3) and thus the network must output\nparameters of different dimensionalities; second, the number of datapoints in D\u03c4 will also depend\non the task and so the network must be able to take inputs of variable cardinality; third, we would\nlike the model to support continual learning. To handle the \ufb01rst two challenges we follow Gordon\net al. [5]. First, each column of the weight matrix is generated independently from the context points\nfrom that class \u03c8\u03c4\nC)], an approach which scales to arbitrary numbers\nof classes. Second, we employ a permutation invariant architecture [18, 19] for \u03c8w(\u00b7) to handle the\nvariable input cardinality (see Appendix E for details). Third, as permutation invariant architectures\ncan be incrementally updated [20], continual learning is supported (as discussed in Section 5).\nIntuitively, the classi\ufb01er weights should be determined by the representation of the data points\nemerging from the adapted feature extractor. We therefore input the adapted feature representation\nof the data points into the network, rather than the raw data points (hence the dependency of \u03c8w on\n\u03c8f and \u03b8). To summarize, \u03c8w(\u00b7) is a function on sets that accepts as input a set of adapted feature\nrepresentations from D\u03c4\n\nw as a parameterized function of the form \u03c8\u03c4\n\nc , and outputs the cth column of the linear classi\ufb01cation matrix, i.e.,\nc ; \u03c6w, \u03c8f , \u03b8) = \u03c8w ({f\u03b8 (xm; \u03c8f )|xm \u2208 D\u03c4 , ym = c}; \u03c6w) .\n\nw = [\u03c8w (D\u03c4\n\n. . . , \u03c8w (D\u03c4\n\n\u03c8w (D\u03c4\n\n1 ) ,\n\n(2)\n\nHere \u03c6w are learnable parameters of \u03c8w(\u00b7). See Figure 3 for an illustration.\nAdaptation Network: Feature Extractor Parameters. CNAPS represents the task-speci\ufb01c feature\nextractor parameters \u03c8\u03c4\nf , comprising the parameters of the FiLM layers \u03b3\u03c4 and \u03b2\u03c4 in our imple-\nmentation, as a parameterized function of the context-set D\u03c4 . Thus, \u03c8f (\u00b7; \u03c6f , \u03b8) is a collection of\nfunctions (one for each FiLM layer) with parameters \u03c6f , many of which are shared across functions.\nWe denote the function generating the parameters for the ith FiLM layer \u03c8i\nOur experiments (Section 5) show that this mapping requires careful parameterization. We propose a\nnovel parameterization that improves performance in complex settings with diverse datasets. Our\nimplementation contains two components: a task-speci\ufb01c representation that provides context about\nthe task to all layers of the feature extractor (denoted z\u03c4\nG), and an auto-regressive component that\nprovides information to deeper layers in the feature extractor concerning how shallower layers have\nadapted to the task (denoted zi\nG is computed\nfor every task \u03c4 by passing the inputs x\u03c4\n\nn through a global set encoder g with parameters in \u03c6f .\n\nf (\u00b7) network is zi = (z\u03c4\n\nAR). The input to the \u03c8i\n\nf (\u00b7) for brevity.\n\nAR). z\u03c4\n\nG, zi\n\n4\n\nSplit\ud835\udc53\ud835\udf03(\ud835\udc99;\ud835\udf4d\ud835\udc53\ud835\udf0f)by class label \ud835\udc66{\ud835\udc53\ud835\udf03(\ud835\udc99;\ud835\udf4d\ud835\udc53\ud835\udf0f)}{\ud835\udc9a}\ud835\udf4d\ud835\udc64(\u00b7;\ud835\udf19\ud835\udc64,\ud835\udf4d\ud835\udc53\ud835\udf0f,\ud835\udf3d){\ud835\udc53\ud835\udf03(\ud835\udc65\ud835\udc58\ud835\udc50;\ud835\udf4d\ud835\udc53\ud835\udf0f)}\ud835\udc58=1\ud835\udc58\ud835\udc50(i.e. \ud835\udc58\ud835\udc50train examples from each class \ud835\udc50)\ud835\udc53\ud835\udf03(\ud835\udc99\u2217;\ud835\udf4d\ud835\udc53\ud835\udf0f)Linear Classifier \ud835\udc64Softmax\ud835\udc5d\ud835\udc9a\u2217\ud835\udc99\u2217,\ud835\udf3d,\ud835\udf4d\ud835\udf0f)\ud835\udc641\ud835\udc64\ud835\udc50\ud835\udc64\ud835\udc36\u2026\ud835\udc4f\ud835\udc50\ud835\udc4f1\ud835\udc4f\ud835\udc36\u2026\u2026\u2026Shared network for each class \ud835\udc50in CMean Pooling\ud835\udf19\ud835\udc4f\ud835\udf19\ud835\udc64\ud835\udc67\ud835\udc36\ud835\udf13\ud835\udc64\ud835\udc37\ud835\udc56\ud835\udf0f\u2254\ud835\udc64\ud835\udc56;\ud835\udc4f\ud835\udc56\fFigure 4: Implementation of the feature-extractor: an independently learned set encoder g provides a \ufb01xed\ncontext that is concatenated to the (processed) activations of x from the previous ResNet block. The inputs\nf (\u00b7), which outputs the FiLM parameters for layer i. Green arrows correspond\nzi = (z\u03c4\nAR is computed by\nto propagation of auto-regressive representations. Note that the auto-regressive component zi\nprocessing the adapted activations {f i\n\nf )} of the previous convolutional block.\n\n\u03b8(x; \u03c8\u03c4\n\nG, zi\n\nAR) are then fed to \u03c8i\n\nFigure 5: Adaptation network \u03c6f . R\u03b3ibj ch and R\u03b2ibj ch denote a vector of regularization weights that are\nlearned with an l2 penalty.\n\nTo adapt the lth layer in the feature extractor, it is useful for the system to have access to the\nrepresentation of task-relevant inputs from layer l \u2212 1. While zG could in principle encode how layer\nl \u2212 1 has adapted, we opt to provide this information directly to the adaptation network adapting layer\nl by passing the adapted activations from layer l\u2212 1. The auto-regressive component zi\nAR is computed\nby processing the adapted activations of the previous convolutional block with a layer-speci\ufb01c set\nencoder (except for the \ufb01rst residual block, whose auto-regressive component is given by the un-\nadapted initial pre-processing stage in the ResNet). Both the global and all layer-speci\ufb01c set-encoders\nare implemented as permutation invariant functions [18, 19] (see Appendix E for details). The full\nf (\u00b7) networks is illustrated in\nparameterization is illustrated in Figure 4, and the architecture of \u03c8i\nFigure 5.\n\n3 Model Training\n\nThe previous section has speci\ufb01ed the model (see Figure 1b for a schematic). We now describe how\nto train the global classi\ufb01er parameters \u03b8 and the adaptation network parameters \u03c6 = {\u03c6f , \u03c6w}.\nTraining the global classi\ufb01er parameters \u03b8. A natural approach to training the model (originally\nemployed by CNPs [13]) would be to maximize the likelihood of the training data jointly over \u03b8 and\n\u03c6. However, experiments (detailed in Appendix D.3) showed that it is crucially important to adopt a\ntwo stage process instead. In the \ufb01rst stage, \u03b8 are trained on a large dataset (e.g., the training set of\nImageNet [21, 6]) in a full-way classi\ufb01cation procedure, mirroring standard pre-training. Second, \u03b8\nare \ufb01xed and \u03c6 are trained using episodic training over all meta-training datasets in the multi-task\nsetting. We hypothesize that two-stage training is important for two reasons: (i) during the second\nstage, \u03c6f are trained to adapt f\u03b8(\u00b7) to tasks \u03c4 by outputting \u03c8\u03c4\nf . As \u03b8 has far more capacity than \u03c8\u03c4\nf ,\nif they are trained in the context of all tasks, there is no need for \u03c8\u03c4\nf to adapt the feature extractor,\nresulting in little-to-no training signal for \u03c6f and poor generalisation. (ii) Allowing \u03b8 to adapt during\n\n5\n\n\ud835\udc53\ud835\udf03\ud835\udc99\u2217Pre\ud835\udc99\ud835\udc99\ud835\udc53\ud835\udf03(\ud835\udc99\u2217;\ud835\udf4d\ud835\udc53\ud835\udf0f)\ud835\udc53\ud835\udf03(\ud835\udc99;\ud835\udf4d\ud835\udc53\ud835\udf0f)Post\ud835\udf4d\ud835\udc87(\ud835\udc37\ud835\udf0f)\ud835\udc54Set Encoder\ud835\udc9b\ud835\udc3a\ud835\udf0fblock2\ud835\udefe1,\ud835\udefd1block 1layer 1\ud835\udf13\ud835\udc871\ud835\udc9bAR1\ud835\udf13\ud835\udc871SetEncoderblock2\ud835\udefe2,\ud835\udefd2block 1layer 2\ud835\udf13\ud835\udc872\ud835\udc9bAR2\ud835\udf13\ud835\udc872SetEncoderblock2\ud835\udefe3,\ud835\udefd3block 1layer 3\ud835\udf13\ud835\udc873\ud835\udc9bAR3\ud835\udf13\ud835\udc873SetEncoderblock2\ud835\udefe4,\ud835\udefd4block 1layer 4\ud835\udf13\ud835\udc874\ud835\udc9bAR4\ud835\udf13\ud835\udc874SetEncoder\ud835\udc53\ud835\udf3d1(\ud835\udc99)\ud835\udc53\ud835\udf3d2(\ud835\udc99;\ud835\udf4d\ud835\udc53\ud835\udf0f)\ud835\udc53\ud835\udf3d3(\ud835\udc99;\ud835\udf4d\ud835\udc53\ud835\udf0f)\ud835\udc53\ud835\udf3d4(\ud835\udc99;\ud835\udf4d\ud835\udc53\ud835\udf0f)\ud835\udc81\ud835\udc3aConcatenate\ud835\udf19\ud835\udc87\ud835\udc56\ud835\udc9b\ud835\udc34\ud835\udc45\ud835\udc56(\ud835\udc592penalty)\ud835\udf19\ud835\udc53\ud835\udf37\ud835\udc4f1\ud835\udf37\ud835\udc56\ud835\udc4f1\ud835\udf19\ud835\udc53\ud835\udf38\ud835\udc4f1(\ud835\udc592penalty)\ud835\udf38\ud835\udc56\ud835\udc4f11(\ud835\udc592penalty)\ud835\udf19\ud835\udc53\ud835\udf37\ud835\udc4f2\ud835\udf37\ud835\udc56\ud835\udc4f2\ud835\udf19\ud835\udc53\ud835\udf38\ud835\udc4f2(\ud835\udc592penalty)\ud835\udf38\ud835\udc56\ud835\udc4f21\fFigure 6: Model design space. The y-axis repre-\nsents the number of task-speci\ufb01c parameters |\u03c8\u03c4|.\nIncreasing |\u03c8\u03c4| increases model \ufb02exibility, but also\nthe propensity to over-\ufb01t. The x-axis represents the\ncomplexity of the mechanism used to adapt the task-\nspeci\ufb01c parameters to training data \u03c8(D\u03c4 ). On the\nright are amortized approaches (i.e. using \ufb01xed func-\ntions). On the left is gradient-based adaptation. Mixed\napproaches lie between. Computational ef\ufb01ciency in-\ncreases to the right. Flexibility increases to the left,\nbut with it over-\ufb01tting and need for hand tuning.\n\nthe second phase violates the principle of \u201ctrain as you test\u201d, i.e., when test tasks are encountered,\n\u03b8 will be \ufb01xed, so it is important to simulate this scenario during training. Finally, \ufb01xing \u03b8 during\nmeta-training is desireable as it results in a dramatic decrease in training time.\n\nTraining the adaptation network parameters \u03c6. Following the work of Garnelo et al. [13], we\ntrain \u03c6 with maximum likelihood. An unbiased stochastic estimator of the log-likelihood is:\n\n(cid:88)\n\nm,\u03c4\n\n\u02c6L (\u03c6) =\n\n1\n\nM T\n\nlog p (y\u2217\u03c4\n\nm |x\u2217\u03c4\n\nm , \u03c8\u03c6 (D\u03c4 ) , \u03b8) ,\n\n(3)\n\nm , x\u2217\u03c4\n\nm , D\u03c4} \u223c \u02c6P , with \u02c6P representing the data distribution (e.g., sampling tasks and\nwhere {y\u2217\u03c4\nsplitting them into disjoint context (D\u03c4 ) and target data {(x\u2217\u03c4\nm=1). Maximum likelihood\ntraining therefore naturally uses episodic context / target splits often used in meta-learning. In our\nexperiments we use the protocol de\ufb01ned by Trianta\ufb01llou et al. [6] and META-DATASET for this\nsampling procedure. Algorithm A.1 details computation of the stochastic estimator for a single task.\n\nm )}Mt\n\nm , y\u2217\u03c4\n\n4 Related Work\n\nOur work frames multi-task classi\ufb01cation as directly modelling the predictive distribution\np(y\u2217|x\u2217, \u03c8(D\u03c4 )). The perspective allows previous work [7, 5, 15, 22, 16, 17, 23, 4, 6, 24, 9, 25, 26]\nto be organised in terms of i) the choice of the parameterization of the classi\ufb01er (and in particular the\nnature of the local parameters), and ii) the function used to compute the local parameters from the\ntraining data. This space is illustrated in Figure 6, and further elaborated upon in Appendix B.\nOne of the inspirations for our work is conditional neural processes (CNPs) [13]. CNPs directly model\nthe predictive distribution p(y\u2217|x\u2217, \u03c8(D\u03c4 )) and train the parameters using maximum likelihood.\nWhereas previous work on CNPs has focused on homogeneous regression and classi\ufb01cation datasets\nand fairly simple models, here we study multiple heterogeneous classi\ufb01cation datasets and use a\nmore complex model to handle this scenario. In particular, whereas the original CNP approach to\nclassi\ufb01cation required pre-specifying the number of classes in advance, CNAPS handles varying way\nclassi\ufb01cation tasks, which is required for e.g. the meta-dataset benchmark. Further, CNAPS employs\na parameter-sharing hierarchy that parameterizes the feature extractor. This contrasts to the original\nCNP approach that shared all parameters across tasks, and use latent inputs to the decoder to adapt to\nnew tasks. Finally, CNAPS employs a meta-training procedure geared towards learning to adapt\nto diverse tasks. Similarly, our work can be viewed as a deterministic limit of ML-PIP [5] which\nemploys a distributional treatment of the local-parameters \u03c8.\nA model with design choices closely related to CNAPS is TADAM [27]. TADAM employs a\nsimilar set of local parameters, allowing for adaptation of both the feature extractor and classi\ufb01cation\nlayer. However, it uses a far simpler adaptation network (lacking auto-regressive structure) and\nan expensive and ad-hoc training procedure. Moreover, TADAM was applied to simple few-shot\nlearning benchmarks (e.g. CIFAR100 and mini-ImageNet) and sees little gain from feature extractor\nadaptation. In contrast, we see a large bene\ufb01t from adapting the feature extractor. This may in part\nre\ufb02ect the differences in the two models, but we observe that feature extractor adaptation has the\nlargest impact when used to adapt to different datasets and that two stage training is required to see\nthis. Further differences are our usage of the CNP framework and the \ufb02exible deployment of CNAPS\nto continual learning and active learning (see Section 5).\n\n6\n\nAdaptation Mechanism \ud835\udf4d(\ud835\udc37\ud835\udf0f)Faster at Test-Time# Task-specific Parameters \ud835\udf4d\ud835\udf0fAllClassifierandFeatureAdaptersClassifierOnlyMulti-step GradientFew-step GradientSemi-AmortizedAmortizedFinetune [9]ResidualAdapters [16, 17]LEO [23],Proto-MAML [6]CNAPS,TADAM [27]VERSA [5],Proto Nets [4],Matching Nets [24]MAML [7]Meta-LSTM [22]Model FlexibilityCAVIA [25]Disc. k-shot [26]\f5 Experiments and Results\n\nThe experiments target three key questions: (i) Can CNAPS improve performance in multi-task\nfew-shot learning? (ii) Does the use of an adaptation network bene\ufb01t computational-ef\ufb01ciency and\ndata-ef\ufb01ciency? (iii) Can CNAPS be deployed directly to complex learning scenarios like continual\nlearning and active learning? The experiments use the following modelling choices (see Appendix E\nfor full details). While CNAPS can utilize any feature extractor, a ResNet18 [14] is used throughout to\nenable fair comparison with Trianta\ufb01llou et al. [6]. To ensure that each task is handled independently,\nbatch normalization statistics [28] are learned (and \ufb01xed) during the pre-training phase for \u03b8. Actual\nbatch statistics of the test data are never used during meta-training or testing.\n\nFew Shot Classi\ufb01cation. The \ufb01rst experiment tackles a demanding few-shot classi\ufb01cation chal-\nlenge called META-DATASET [6]. META-DATASET is composed of ten (eight train, two test) image\nclassi\ufb01cation datasets. The challenge constructs few-shot learning tasks by drawing from the follow-\ning distribution. First, one of the datasets is sampled uniformly; second, the \u201cway\u201d and \u201cshot\u201d are\nsampled randomly according to a \ufb01xed procedure; third, the classes and context / target instances are\nsampled. Where a hierarchical structure exists in the data (ILSVRC or OMNIGLOT), task-sampling\nrespects the hierarchy. In the meta-test phase, the identity of the original dataset is not revealed\nand the tasks must be treated independently (i.e. no information can be transferred between them).\nNotably, the meta-training set comprises a disjoint and dissimilar set of classes from those used for\nmeta-test. Full details are available in Appendix C.1 and [6].\nTrianta\ufb01llou et al. [6] consider two stage training: an initial stage that trains a feature extractor in a\nstandard classi\ufb01cation setting, and a meta-training stage of all parameters in an episodic regime. For\nthe meta-training stage, they consider two settings: meta-training only on the META-DATASET version\nof ILSVRC, and on all meta-training data. We focus on the latter as CNAPS rely on training data\nfrom a variety of training tasks to learn to adapt, but provide results for the former in Appendix D.1.\nWe pre-train \u03b8 on the meta-training set of the META-DATASET version of ILSVRC, and meta-train\n\u03c6 in an episodic fashion using all meta-training data. We compare CNAPS to models considered\nby Trianta\ufb01llou et al. [6], including their proposed method (Proto-MAML) in Table 1. We meta-test\nCNAPS on three additional held-out datasets: MNIST [29], CIFAR10 [30], and CIFAR100 [30]. As\nan ablation study, we compare a version of CNAPS that does not make use of the auto-regressive\ncomponent zAR, and a version that uses no feature extractor adaptation. In our analysis of Table 1, we\ndistinguish between two types of generalization: (i) unseen tasks (classes) in meta-training datasets,\nand (ii) unseen datasets.\n\nUnseen tasks: CNAPS achieve signi\ufb01cant improvements over existing methods on seven of the\neight datasets. The exception is the TEXTURES dataset, which has only seven test classes and\naccuracy is highly sensitive to the train / validation / test class split. The ablation study demonstrates\nthat removing zAR from the feature extractor adaptation degrades accuracy in most cases, and that\nremoving all feature extractor adaptation results in drastic reductions in accuracy.\n\nUnseen datasets: CNAPS-models outperform all competitive models with the exception of FINE-\nTUNE on the TRAFFIC SIGNS dataset. Removing zAR from the feature extractor decreases accuracy\nand removing the feature extractor adaptation entirely signi\ufb01cantly impairs performance. The degra-\ndation is particularly pronounced when the held out dataset differs substantially from the dataset used\nto pretrain \u03b8, e.g. for MNIST.\nNote that the superior results when using the auto-regressive component can not be attributed to\nincreased network capacity alone. In Appendix D.4 we demonstrate that CNAPS yields superior\nclassi\ufb01cation accuracy when compared to parallel residual adapters [17] even though CNAPS requires\nsigni\ufb01cantly less network capacity in order to adapt the feature extractor to a given task.\n\nAdditional results: Results when meta-training only on the META-DATASET version of ILSVRC\nare given in Table D.3. In Appendix D.2, we visualize the task encodings and parameters, demon-\nstrating that the model is able to learn meaningful task and dataset level representations and parame-\nterizations. The results support the hypothesis that learning to adapt key parts of the network is more\nrobust and achieves signi\ufb01cantly better performance than existing approaches.\n\n7\n\n\fFinetune\n43.1 \u00b1 1.1\n71.1 \u00b1 1.4\n72.0 \u00b1 1.1\n59.8 \u00b1 1.2\n69.1 \u00b1 0.9\n47.0 \u00b1 1.2\n38.2 \u00b1 1.0\n85.3 \u00b1 0.7\n66.7 \u00b1 1.2\n35.2 \u00b1 1.1\n\nMatchingNet\n36.1 \u00b1 1.0\n78.3 \u00b1 1.0\n69.2 \u00b1 1.0\n56.4 \u00b1 1.0\n61.8 \u00b1 0.7\n60.8 \u00b1 1.0\n33.7 \u00b1 1.0\n81.9 \u00b1 0.7\n55.6 \u00b1 1.1\n28.8 \u00b1 1.0\n\nProtoNet\n44.5 \u00b1 1.1\n79.6 \u00b1 1.1\n71.1 \u00b1 0.9\n67.0 \u00b1 1.0\n65.2 \u00b1 0.8\n64.9 \u00b1 0.9\n40.3 \u00b1 1.1\n86.9 \u00b1 0.7\n46.5 \u00b1 1.0\n39.9 \u00b1 1.1\n\nfo-MAML\n32.4 \u00b1 1.0\n71.9 \u00b1 1.2\n52.8 \u00b1 0.9\n47.2 \u00b1 1.1\n56.7 \u00b1 0.7\n50.5 \u00b1 1.2\n21.0 \u00b1 1.0\n70.9 \u00b1 1.0\n34.2 \u00b1 1.3\n24.1 \u00b1 1.1\n\nProto-MAML\n47.9 \u00b1 1.1\n82.9 \u00b1 0.9\n74.2 \u00b1 0.8\n70.0 \u00b1 1.0\n67.9 \u00b1 0.8\n66.6 \u00b1 0.9\n42.0 \u00b1 1.1\n88.5 \u00b1 0.7\n52.3 \u00b1 1.1\n41.3 \u00b1 1.0\n\nDataset\n\nILSVRC [21]\nOmniglot [31]\nAircraft [32]\nBirds [33]\nTextures [34]\nQuick Draw [35]\nFungi [36]\nVGG Flower [37]\nTraf\ufb01c Signs [38]\nMSCOCO [39]\nMNIST [29]\nCIFAR10 [30]\nCIFAR100 [30]\n\nCNAPS\n(no \u03c8f )\n43.8 \u00b1 1.0\n60.1 \u00b1 1.3\n53.0 \u00b1 0.9\n55.7 \u00b1 1.0\n60.5 \u00b1 0.8\n58.1 \u00b1 1.0\n28.6 \u00b1 0.9\n75.3 \u00b1 0.7\n55.0 \u00b1 0.9\n41.2 \u00b1 1.0\n76.0 \u00b1 0.8\n61.5 \u00b1 0.7\n44.8 \u00b1 1.0\n\nCNAPS\n(no zAR)\n51.3 \u00b1 1.0\n88.0 \u00b1 0.7\n76.8 \u00b1 0.8\n71.4 \u00b1 0.9\n62.5 \u00b1 0.7\n71.9 \u00b1 0.8\n46.0 \u00b1 1.1\n89.2 \u00b1 0.5\n60.1 \u00b1 0.9\n42.0 \u00b1 1.0\n88.6 \u00b1 0.5\n60.0 \u00b1 0.8\n48.1 \u00b1 1.0\n\nCNAPS\n52.3 \u00b1 1.0\n88.4 \u00b1 0.7\n80.5 \u00b1 0.6\n72.2 \u00b1 0.9\n58.3 \u00b1 0.7\n72.5 \u00b1 0.8\n47.4 \u00b1 1.0\n86.0 \u00b1 0.5\n60.2 \u00b1 0.9\n42.6 \u00b1 1.1\n92.7 \u00b1 0.4\n61.5 \u00b1 0.7\n50.1 \u00b1 1.0\n\nTable 1: Few-shot classi\ufb01cation results on META-DATASET [6] using models trained on all training datasets. All\n\ufb01gures are percentages and the \u00b1 sign indicates the 95% con\ufb01dence interval over tasks. Bold text indicates the\nscores within the con\ufb01dence interval of the highest score. Tasks from datasets below the dashed line were not\nused for training. Competing methods\u2019 results from [6].\n\nFigure 7: Comparing CNAPS to gradient based feature extractor adaptation: accuracy on 5-way classi\ufb01cation\ntasks from withheld datasets as a function of processing time. Dot size re\ufb02ects shot number (1 to 25 shots).\n\nFiLM Parameter Learning Performance: Speed-Accuracy Trade-off. CNAPS generate FiLM\nlayer parameters for each task \u03c4 at test time using the adaptation network \u03c8f (D\u03c4 ). It is also possible\nto learn the FiLM parameters via gradient descent (see [16, 17]). Here we compare CNAPS to this\napproach. Figure 7 shows plots of 5-way classi\ufb01cation accuracy versus time for four held out data\nsets as the number of shots was varied. For gradient descent, we used a \ufb01xed learning rate of 0.001\nand took 25 steps for each point. The overall time required to produce the plot was 1274 and 7214\nseconds for CNAPS and gradient approaches, respectively, on a NVIDIA Tesla P100-PCIE-16GB\nGPU. CNAPS is at least 5 times faster at test time than gradient-based optimization requiring only a\nsingle forward pass through the network while gradient based approaches require multiple forward\nand backward passes. Further, the accuracy achieved with adaptation networks is signi\ufb01cantly higher\nfor fewer shots as it protects against over-\ufb01tting. For large numbers of shots, gradient descent catches\nup, albeit slowly.\n\nComplex Learning Scenarios: Continual Learning.\nIn continual learning [40] new tasks appear\nover time and existing tasks may change. The goal is to adapt accordingly, but without retaining old\ndata which is challenging for arti\ufb01cial systems. To demonstrate the the versatility CNAPS we show\nthat, although it has not been explicitly trained for continual learning, we are able to apply the same\nmodel trained for the few-shot classi\ufb01cation experiments (without the auto-regressive component) to\nstandard continual learning benchmarks on held out datasets: Split MNIST [41] and Split CIFAR100\n[42]. We modify the model to compute running averages for the representations of both \u03c8\u03c4\nw and \u03c8\u03c4\nf\n(see Appendix F for further details), in this way it performs incremental updates using the new data\nand the old model, and does not need to access old data. Figure 8 (left) shows the accumulated multi-\nand single-head [42] test accuracy averaged over 30 runs (further results and more detailed \ufb01gures are\nin Appendix G). Figure 8 (right) shows average results at the \ufb01nal task comparing to SI [41], EWC\n[43], VCL [44], and Riemannian Walk [42].\nFigure 8 demonstrates that CNAPS naturally resists catastrophic forgetting [43] and compares\nfavourably to competing methods, despite the fact that it was not exposed to these datasets during\ntraining, observes orders of magnitude fewer examples, and was not trained explicitly to perform\ncontinual learning. CNAPS performs similarly to, or better than, the state-of-the-art Riemannian\nWalk method which departs from the pure continual learning setting by maintaining a small number\nof training samples across tasks. Conversely, CNAPS has the advantage of being exposed to a larger\n\n8\n\n20100300Time (seconds)3040506070Accuracy (%)COCO20100300Time (seconds)405060708090Traffic Signs20100300Time (seconds)405060708090100MNIST20100300Time (seconds)304050607080CIFAR 10CNAPsGradient Descent\fMethod\nSI [41]\nEWC [43]\nVCL [44]\n\nRWalk [42]\nCNAPS\n\nMNIST\n\nCIFAR100\n\n-\n\nSingle\n22.8\n23.1\n\nSingle Multi\n73.2\n57.6\n55.8\n72.8\n\nMulti\n99.3\n99.3\n98.5\n\u00b1 0.4\n99.3\n34.0\n37.2\n98.9\n\u00b1 0.2 \u00b1 0.9 \u00b1 0.5 \u00b1 0.6\n\n-\n\n74.2\n76.0\n\n-\n\n82.5\n80.9\n\nFigure 8: Continual learning classi\ufb01cation results on Split MNIST and Split CIFAR100 using a model trained\non all training datasets. (Left) The plots show accumulated accuracy averaged over 30 runs for both single-\nand multi-head scenarios. (Right) Average accuracy at \ufb01nal task computed over 30 experiments (all \ufb01gures are\npercentages). Errors are one standard deviation. Additional results from [42, 45].\n\nFigure 9: Accuracy vs active learning iterations for held-out classes / languages. (Top) CNAPS and (bottom)\nprototypical networks. Error shading is one standard error. CNAPS achieves better accuracy than prototypical\nnetworks and improvements over random acquisition, whereas prototypical networks do not.\n\nrange of datasets and can therefore leverage task transfer. We emphasize that this is not meant to be\nan \u201capples-to-apples\u201d comparison, but rather, the goal is to demonstrate the out-of-the-box versatility\nand strong performance of CNAPS in new domains and learning scenarios.\nComplex Learning Scenarios: Active Learning. Active learning [46, 47] requires accurate data-\nef\ufb01cient learning that returns well-calibrated uncertainty estimates. Figure 9 compares the perfor-\nmance of CNAPS and prototypical networks using two standard active learning acquisition functions\n(variation ratios and predictive entropy [46]) against random acquisition on the FLOWERS dataset and\nthree representative held-out languages from OMNIGLOT (performance on all languages is presented\nin Appendix H). Figure 9 and Appendix H show that CNAPS achieves higher accuracy on average\nthan prototypical networks. Moreover, CNAPS achieves signi\ufb01cant improvements over random\nacquisition, whereas prototypical networks do not. These tests indicates that CNAPS is more accurate\nand suggest that CNAPS has better calibrated uncertainty estimates than prototypical networks.\n\n6 Conclusions\n\nThis paper has introduced CNAPS, an automatic, fast and \ufb02exible modelling approach for multi-\ntask classi\ufb01cation. We have demonstrated that CNAPS achieve state-of-the-art performance on the\nMETA-DATASET challenge, and can be deployed \u201cout-of-the-box\u201d to diverse learning scenarios such\nas continual and active learning where they are competitive with the state-of-the-art. Future avenues\nof research are to consider the exploration of the design space by introducing gradients and function\napproximation to the adaptation mechanisms, as well as generalizing the approach to distributional\nextensions of CNAPS [48, 49].\n\n9\n\n1234592.595.097.5100.0Accuracy (%)MNIST Multi-headRWalk 10k-shotCNAPs 1-shotCNAPs 10-shotCNAPs 100-shot1234560708090100MNIST Single-head12345678910Tasks4050607080Accuracy (%)CIFAR100 Multi-head12345678910Tasks20406080CIFAR100 Single-head657075C-NAPsVGG Flower758085Avesta606570Kannada6570MalayalamVar RatPred EntRand0102030Acquisitions60657075Proto-Net0102030Acquisitions7580850102030Acquisitions6065700102030Acquisitions6570\fAcknowledgments\n\nThe authors would like to thank Ambrish Rawat for helpful discussions and David Duvenaud, Wessel\nBruinsma, Will Tebbutt Adri\u00e0 Garriga Alonso, Eric Nalisnick, and Lyndon White for the insightful\ncomments and feedback. Richard E. Turner is supported by Google, Amazon, Improbable and\nEPSRC grants EP/M0269571 and EP/L000776/1.\n\nReferences\n[1] J\u00fcrgen Schmidhuber. Evolutionary principles in self-referential learning. PhD thesis, Technische\n\nUniversit\u00e4t M\u00fcnchen, 1987.\n\n[2] Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media,\n\n2012.\n\n[3] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept\n\nlearning through probabilistic program induction. Science, 350(6266):1332\u20131338, 2015.\n\n[4] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In\n\nAdvances in Neural Information Processing Systems, pages 4080\u20134090, 2017.\n\n[5] Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner.\nMeta-learning probabilistic inference for prediction. In International Conference on Learning\nRepresentations, 2019. URL https://openreview.net/forum?id=HkxStoC5F7.\n\n[6] Eleni Trianta\ufb01llou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin,\nCarles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset:\nA dataset of datasets for learning to learn from few examples. arXiv preprint arXiv:1903.03096,\n2019.\n\n[7] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta-\ntion of deep networks. In International Conference on Machine Learning, pages 1126\u20131135,\n2017.\n\n[8] Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint\n\narXiv:1803.02999, 2018.\n\n[9] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in\ndeep neural networks? In Advances in neural information processing systems, pages 3320\u20133328,\n2014.\n\n[10] Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan Yuille. Few-shot image recognition by predicting\n\nparameters from activations. arXiv preprint arXiv:1706.03466, 2017.\n\n[11] Seymour Geisser. On the prediction of observables: a selective update. Technical report,\n\nUniversity of Minnesota, 1983.\n\n[12] Seymour Geisser. Predictive inference. Routledge, 2017.\n\n[13] Marta Garnelo, Dan Rosenbaum, Chris J Maddison, Tiago Ramalho, David Saxton, Murray\nShanahan, Yee Whye Teh, Danilo J Rezende, and SM Eslami. Conditional neural processes.\narXiv preprint arXiv:1807.01613, 2018.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image\nrecognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,\npages 770\u2013778, 2016.\n\n[15] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM:\nVisual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on\nArti\ufb01cial Intelligence, 2018.\n\n[16] Sylvestre-Alvise Rebuf\ufb01, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains\nwith residual adapters. In Advances in Neural Information Processing Systems, pages 506\u2013516,\n2017.\n\n10\n\n\f[17] Sylvestre-Alvise Rebuf\ufb01, Hakan Bilen, and Andrea Vedaldi. Ef\ufb01cient parametrization of multi-\ndomain deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and\nPattern Recognition, pages 8119\u20138127, 2018.\n\n[18] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov,\nand Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems,\npages 3394\u20133404, 2017.\n\n[19] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point\nsets for 3d classi\ufb01cation and segmentation. Proc. Computer Vision and Pattern Recognition\n(CVPR), IEEE, 1(2):4, 2017.\n\n[20] Manasi Vartak, Arvind Thiagarajan, Conrado Miranda, Jeshua Bratman, and Hugo\nLarochelle. A meta-learning perspective on cold-start recommendations for items.\nIn\nI. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and\nR. Garnett, editors, Advances in Neural Information Processing Systems 30, pages\n6904\u20136914. Curran Associates, Inc., 2017. URL http://papers.nips.cc/paper/\n7266-a-meta-learning-perspective-on-cold-start-recommendations-for-items.\npdf.\n\n[21] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng\nHuang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual\nrecognition challenge. International journal of computer vision, 115(3):211\u2013252, 2015.\n\n[22] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In Proceed-\n\nings of the International Conference on Learning Representations (ICLR), 2017.\n\n[23] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon\nOsindero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint\narXiv:1807.05960, 2018.\n\n[24] Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al. Matching networks for one\nshot learning. In Advances in Neural Information Processing Systems, pages 3630\u20133638, 2016.\n\n[25] Luisa M Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson.\n\nCAML: Fast context adaptation via meta-learning. arXiv preprint arXiv:1810.03642, 2018.\n\n[26] Matthias Bauer, Mateo Rojas-Carulla, Jakub Bart\u0142omiej \u00b4Swi \u02dbatkowski, Bernhard Sch\u00f6lkopf, and\nRichard E Turner. Discriminative k-shot learning using probabilistic models. arXiv preprint\narXiv:1706.00326, 2017.\n\n[27] Boris N Oreshkin, Alexandre Lacoste, and Pau Rodriguez. TADAM: Task dependent adaptive\n\nmetric for improved few-shot learning. arXiv preprint arXiv:1805.10123, 2018.\n\n[28] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training\nby reducing internal covariate shift. In International Conference on Machine Learning, pages\n448\u2013456, 2015.\n\n[29] Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. AT&T Labs\n\n[Online]. Available: http://yann. lecun. com/exdb/mnist, 2:18, 2010.\n\n[30] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images.\n\nTechnical report, Citeseer, 2009.\n\n[31] Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning\nof simple visual concepts. In Proceedings of the Annual Meeting of the Cognitive Science\nSociety, volume 33, 2011.\n\n[32] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-\n\ngrained visual classi\ufb01cation of aircraft. arXiv preprint arXiv:1306.5151, 2013.\n\n[33] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The\n\ncaltech-ucsd birds-200-2011 dataset. 2011.\n\n11\n\n\f[34] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.\nDescribing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision\nand Pattern Recognition, pages 3606\u20133613, 2014.\n\n[35] David Ha and Douglas Eck. A neural representation of sketch drawings. arXiv preprint\n\narXiv:1704.03477, 2017.\n\n[36] Brigit Schroeder and Yin Cui. Fgvcx fungi classi\ufb01cation challenge at fgvc5. https://www.\n\nkaggle.com/c/fungi-challenge-fgvc-2018, 2018.\n\n[37] Maria-Elena Nilsback and Andrew Zisserman. Automated \ufb02ower classi\ufb01cation over a large\nnumber of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image\nProcessing, pages 722\u2013729. IEEE, 2008.\n\n[38] Sebastian Houben, Johannes Stallkamp, Jan Salmen, Marc Schlipsing, and Christian Igel.\nDetection of traf\ufb01c signs in real-world images: The german traf\ufb01c sign detection benchmark. In\nThe 2013 international joint conference on neural networks (IJCNN), pages 1\u20138. IEEE, 2013.\n\n[39] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr\nDoll\u00e1r, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European\nconference on computer vision, pages 740\u2013755. Springer, 2014.\n\n[40] Mark B Ring. Child: A \ufb01rst step towards continual learning. Machine Learning, 28(1):77\u2013104,\n\n1997.\n\n[41] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic\nintelligence. In Proceedings of the 34th International Conference on Machine Learning-Volume\n70, pages 3987\u20133995. JMLR. org, 2017.\n\n[42] Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian\nwalk for incremental learning: Understanding forgetting and intransigence. In Proceedings of\nthe European Conference on Computer Vision (ECCV), pages 532\u2013547, 2018.\n\n[43] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins,\nAndrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.\nOvercoming catastrophic forgetting in neural networks. Proceedings of the national academy of\nsciences, 114(13):3521\u20133526, 2017.\n\n[44] Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual\n\nlearning. arXiv preprint arXiv:1710.10628, 2017.\n\n[45] Siddharth Swaroop, Cuong V Nguyen, Thang D Bui, and Richard E Turner. Improving and\n\nunderstanding variational continual learning. arXiv preprint arXiv:1905.02099, 2019.\n\n[46] David A Cohn, Zoubin Ghahramani, and Michael I Jordan. Active learning with statistical\n\nmodels. Journal of arti\ufb01cial intelligence research, 4:129\u2013145, 1996.\n\n[47] Burr Settles. Active learning. Synthesis Lectures on Arti\ufb01cial Intelligence and Machine\n\nLearning, 6(1):1\u2013114, 2012.\n\n[48] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami,\n\nand Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622, 2018.\n\n[49] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum,\nOriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on\nLearning Representations, 2019. URL https://openreview.net/forum?id=SkE6PjC9KX.\n\n12\n\n\f", "award": [], "sourceid": 4367, "authors": [{"given_name": "James", "family_name": "Requeima", "institution": "University of Cambridge / Invenia Labs"}, {"given_name": "Jonathan", "family_name": "Gordon", "institution": "University of Cambridge"}, {"given_name": "John", "family_name": "Bronskill", "institution": "University of Cambridge"}, {"given_name": "Sebastian", "family_name": "Nowozin", "institution": "Google Research Berlin"}, {"given_name": "Richard", "family_name": "Turner", "institution": "Cambridge"}]}