{"title": "DATA: Differentiable ArchiTecture Approximation", "book": "Advances in Neural Information Processing Systems", "page_first": 876, "page_last": 886, "abstract": "Neural architecture search (NAS) is inherently subject to the gap of architectures during searching and validating. To bridge this gap, we develop Differentiable ArchiTecture Approximation (DATA) with an Ensemble Gumbel-Softmax (EGS) estimator to automatically approximate architectures during searching and validating in a differentiable manner. Technically, the EGS estimator consists of a group of Gumbel-Softmax estimators, which is capable of converting probability vectors to binary codes and passing gradients from binary codes to probability vectors. Benefiting from such modeling, in searching, architecture parameters and network weights in the NAS model can be jointly optimized with the standard back-propagation, yielding an end-to-end learning mechanism for searching deep models in a large enough search space. Conclusively, during validating, a high-performance architecture that approaches to the learned one during searching is readily built. Extensive experiments on a variety of popular datasets strongly evidence that our method is capable of discovering high-performance architectures for image classification, language modeling and semantic segmentation, while guaranteeing the requisite efficiency during searching.", "full_text": "DATA: Differentiable ArchiTecture Approximation\n\nJianlong Chang1,2,3\n\nXinbang Zhang1,2\n\nShiming Xiang1,2\n\nYiwen Guo4,5\nChunhong Pan1\n\nGaofeng Meng1\n\n1NLPR, Institute of Automation, Chinese Academy of Sciences\n\n2School of Arti\ufb01cial Intelligence, University of Chinese Academy of Sciences\n3Samsung Research China - Beijing, 4Intel Labs China, 5Bytedance AI Lab\n{jianlong.chang, xinbang.zhang, gfmeng, smxiang, chpan}@nlpr.ia.ac.cn\n\nguoyiwen.ai@bytedance.com\n\nAbstract\n\nNeural architecture search (NAS) is inherently subject to the gap of architectures\nduring searching and validating. To bridge this gap, we develop Differentiable\nArchiTecture Approximation (DATA) with an Ensemble Gumbel-Softmax (EGS)\nestimator to automatically approximate architectures during searching and vali-\ndating in a differentiable manner. Technically, the EGS estimator consists of a\ngroup of Gumbel-Softmax estimators, which is capable of converting probability\nvectors to binary codes and passing gradients from binary codes to probability\nvectors. Bene\ufb01ting from such modeling, in searching, architecture parameters\nand network weights in the NAS model can be jointly optimized with the stan-\ndard back-propagation, yielding an end-to-end learning mechanism for searching\ndeep models in a large enough search space. Conclusively, during validating, a\nhigh-performance architecture that approaches to the learned one during searching\nis readily built. Extensive experiments on a variety of popular datasets strongly\nevidence that our method is capable of discovering high-performance architectures\nfor image classi\ufb01cation, language modeling and semantic segmentation, while\nguaranteeing the requisite ef\ufb01ciency during searching.\n\n1\n\nIntroduction\n\nIn the era of deep learning, how to design proper network architectures for speci\ufb01c problems is\na crucial but challenging task. However, designing architecture with state-of-the-art performance\ntypically requires substantial efforts from human experts. In order to eliminate such exhausting\nengineering, many neural architecture search (NAS) methods have been devoted to accomplishing\nthe task automatically [14, 27, 55], i.e., evolution-based NAS [13, 18, 26, 41, 43, 44, 45, 47],\nreinforcement learning-based NAS [2, 3, 21, 42, 56, 59, 60], and gradient-based NAS [11, 34, 35, 46,\n53], which has achieved signi\ufb01cant successes in a multitude of \ufb01elds, including image classi\ufb01cation [4,\n12, 21, 30, 31, 34, 44, 53, 60], semantic segmentation [8, 32] and object detection [9, 15, 50, 52, 60].\nAlthough the achievements in the literature are brilliant, these methods are still hard to effectively\nbridge the gap between architectures during searching and validating. That is, feasible paths in a\nlearned architecture are dependent on each other and become deeply coupled during searching. In\nvalidating, however, the inherited architectures from searching always decouple the dependent paths\nrudely, such as DARTS [34] and SNAS [53] that choose only one path in validating. As a result, the\neffectiveness of the searched architectures are unclear although they could surpass the random ones.\nIn order to eliminate the limitation, Differentiable ArchiTecture Approximation (DATA) is proposed\nto elegantly minimize the gap of architectures during searching and validating. For this purpose, we\ndevelop the Ensemble Gumbel-Softmax (EGS) estimator, an ensemble of a group of Gumbel-Softmax\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\festimators, which is in a position to sample an architecture that approaches the one during searching\nas close as possible, while maintaining the differentiability of a promising NAS pipeline for requisite\nef\ufb01ciency. That is, our EGS estimator suf\ufb01ces to not only decouple the relationship between different\npaths in learned architectures but also pass gradient seamlessly, yielding an end-to-end mechanism of\nsearching deep models in a large enough search space.\nTo sum up, the main contributions of this work are:\n\n\u2022 By generalizing the Gumbel-Softmax estimator, we develop the EGS estimator, which\nprovides a successful attempt to effectively and ef\ufb01ciently perform structural decisions like\npolicy gradient in the reinforcement-learning, with higher ef\ufb01ciency.\n\n\u2022 With the EGS estimator, the DATA model can seamlessly bridge the gap of architectures\nbetween searching and validating, and be learned with the standard back-propagation,\nyielding an end-to-end mechanism of searching deep models in a large enough search space.\n\u2022 Extensive experiments strongly demonstrate that our DATA model consistently outperforms\ncurrent NAS models in searching high-performance convolutional and recurrent architectures\nfor image classi\ufb01cation, semantic segmentation, and language modeling.\n\n2 Differentiable architecture search\n\nBefore introducing our approach, we \ufb01rst brie\ufb02y review NAS. Without loss of generality, the\narchitecture search space A can be naturally represented by directed acyclic graphs (DAG) each\nconsisting of an ordered sequence of nodes. For a speci\ufb01c architecture, it always corresponds\nto a graph \u03b1 \u2208 A, represented as N (\u03b1, w) with network weights w. Intrinsically, the goal in\nNAS is to \ufb01nd a graph \u03b1\u2217 \u2208 A that minimizes the validation loss Lval(N (\u03b1\u2217, w\u2217)), where the\nnetwork weights w\u2217 associated with the architecture \u03b1\u2217 are obtained by minimizing the training loss\nw\u2217 = arg minw Ltrain(N (\u03b1\u2217, w)), i.e.,\n\u03b1\u2208A Lval(N (\u03b1, w\u2217)),\n\nLtrain(N (\u03b1\u2217, w)).\n\ns.t. w\u2217 = arg min\n\nmin\n\n(1)\n\nw\n\nmin\n\ns.t. w\u2217 = arg minw Ltrain(N (\u03b1\u2217, w))\n\n\u03b1\u2208A Lval(N (\u03b1, w\u2217)),\nThis implies that the essence of NAS is to solve a bi-level optimization problem, which is hard to\noptimize because of the nested relationship between architecture parameters \u03b1 and network weights\nw. To handle this issue, we parameterize architectures with binary codes, and devote to jointly\nlearning architectures and network weights in a differentiable way.\n\n2.1 Parameterizing architectures with binary codes\nFor simplicity, we denote all DAGs with n ordered nodes as A = {e(i,j)|1 \u2264 i < j \u2264 n}, where\ne(i,j) indicates a directed edge from the i-th node to the j-th node. Corresponding to each directed\nedge e(i,j), there are a set of candidate primitive operations O = {o1,\u00b7\u00b7\u00b7 , oK}, such as convolution,\npooling, identity, and zero. With these operations, the output at the j-th node can be formulated as\n\nx(j) =\n\no(i,j)(x(i))\n\n(2)\n\n(cid:88)\n\ni<j\n\nK(cid:88)\n\nwhere x(i) denotes the input from the i-th node, and o(i,j)(\u00b7) is a function applied to x(i) which can\nbe decomposed into a superposition of primitive operations in O, i.e.,\n\no(i,j)(x(i)) =\n\nA(i,j)\n\nk\n\n\u00b7 ok(x(i)), s.t. A(i,j)\n\nk\n\n\u2208 {0, 1}, 1 \u2264 k \u2264 K,\n\n(3)\n\nk=1\n\nwhere ok(\u00b7) is the k-th candidate primitive operation in O, and A(i,j)\nsigni\ufb01es a binary weight to\nindicate whether the operation ok(\u00b7) is utilized on the edge e(i,j). For a network, by such de\ufb01nition,\nthere is one and only one architecture code A \u2208 {0, 1}n\u00d7n\u00d7K that corresponds to it, which implies\nthat we can learn the code A to approximate the optimal architecture in A.\n\nk\n\n2\n\n\fFigure 1: A conceptual visualization for the searching process within our model. (a) First, an\narchitecture (i.e., directed acyclic graph) consisting of four ordered nodes is prede\ufb01ned. (b) During\nthe forward propagation, with three candidate primitive operations (i.e., green, orange and cyan\nlines), the binary function f (\u00b7) is employed to generate a network in a differentiable manner. During\nthe backward propagation, the standard back-propagation algorithm is utilized to simultaneously\ncalculate the gradients of the both architecture parameters and network weights. (c) Finally, the\ndetails of the cell can be generated with the binary function f (\u00b7) and utilized to handle speci\ufb01c tasks.\n\n2.2 From probability vectors to binary codes\n\nBene\ufb01ting from the uniqueness property of our architecture code A, the task of learning an architec-\nture can therefore be converted to learning the optimal binary code A. However, it is a fussy NP-hard\nproblem, and is stubborn to learn directly. To overcome the obstacle, we introduce a binary function\nf (\u00b7) to approach the optimal binary codes with probability vectors, which can be easily obtained in\ndeep models. Formally, the categorical choice in Eq. (3) can be rewritten as follows\n\n\u02dco(i,j)(x(i)) =\n\nf\n\nP (i,j)\n\nk\n\n(cid:88)K\n\nk=1\nP (i,j)\n\n(cid:16)\n\n(cid:17) \u2208 {0, 1}, 1 \u2264 k \u2264 K,\n\n= 1, P (i,j)\n\n\u2265 0, f\n\nP (i,j)\n\nk\n\nk=1\n\ns.t.\nis the k-th element in the probability vector P(i,j) \u2208 RK and denotes the probability of\nwhere P (i,j)\nchoosing the k-th operation on the edge e(i,j), and f (\u00b7) represents a binary function that suf\ufb01ces to\nmap a probability vector to a binary code and pass gradients in a continuous manner. Speci\ufb01cally,\nf (\u00b7) is chosen to be a monotonically increasing function in our method, i.e.,\n\nk\n\nk\n\nk\n\nf (P (i,j)\n\n) \u2264 f (P (i,j)\n\n, 1 \u2264 k1, k2 \u2264 K.\nBy substituting A(i,j)\ninstead as the variable to be optimized, we\nhave successfully achieved a continuous relaxation. Bene\ufb01ting from the \ufb02exibility of our formulation,\nfurthermore, the optimization of NAS in Eq. (1) can be seamlessly jointed together, i.e.,\n\nk2\nk2\n) and considering P (i,j)\n\nk with f (P (i,j)\n\n\u2264 P (i,j)\n\n) if P (i,j)\n\n(5)\n\nk1\n\nk1\n\nk\n\nk\n\nK(cid:88)\n\n(cid:16)\n\n(cid:17) \u00b7 ok(x(i)),\n\n(4)\n\n(6)\n\nE\u03b1\u223cP[Ltrain(N (\u03b1, w))],\n\nmin\nw,\u03b1\n\nwhere \u03b1 \u223c P signi\ufb01es that an architecture \u03b1 and P \u2208 Rn\u00d7n\u00d7K are one-to-one correspondence.\nWith the objective in Eq. (6), the main process of optimizing it is to minimize the expected perfor-\nmance of architectures associated with K probability vectors P \u2208 Rn\u00d7n\u00d7K. That is, the network \u03b1\nis \ufb01rst generated from the binary function f (\u00b7) and P. Afterward, the loss on the training dataset can\nbe calculated by forward propagation. Relying on this loss, the gradients of the network architecture\nparameter P and the network parameter w are yielded to modify these parameters better. Because of\nthe differentiability, our model can be trained end-to-end by the standard back-propagation algorithm.\nIn the end, the network architecture \u03b1 is identi\ufb01ed by P, and the network parameter w is estimated by\nretraining on the training set. A conceptual visualization of such a process is illustrated in Figure 1.\n\n3 Architecture approximation - Ensemble Gumbel-Softmax (EGS)\n\nAlthough the reformulation presented in Section 2.2 makes the search space continuous, how to\nde\ufb01ne the binary function f (\u00b7) as desired to map each probability to a binary code needs to be sorted\n\n3\n\n12341234(a)(b)(c)12341234samplinggradient\fFigure 2: A visualized comparison between Gumbel-Softmax (left) and ensemble Gumbel-Softmax\n(right, M = 2). For a probability vector p = [0.5, 0.5], Gumbel-Softmax solely pertains to sample\nonly two binary codes with the same probability, i.e., P ([1, 0]) = P ([0, 1]) = 0.5. In contrast, our\nensemble Gumbel-Softmax is capable of sampling more diversi\ufb01ed binary codes, i.e., [1, 0], [1, 1]\nand [0, 1]. Furthermore, the probabilities of sampling these binary codes are logical. Typically, it\nis conceptually intuitive that the probability of sampling [1, 1] is larger than the probabilities of\nsampling the others since the probabilities in p = [0.5, 0.5] are equal to each other.\n\nout. For a coarse f (\u00b7), it may aggravate the gap between architectures during searching and validating,\nsuch as DARTS [34] and SNAS [53] that strictly limit the binary codes as one-hot vectors. As for\na re\ufb01ned f (\u00b7), we introduce an Ensemble Gumbel-Softmax (EGS) estimator to optimize the NAS\nproblem with a principled approximation. As such, our model can be directly optimized with the\nback-propagation algorithm in an end-to-end way, and bridges the gap between architectures during\nsearching and validating as close as possible, yielding an ef\ufb01cient and effective searching mechanism.\n\n3.1 Gumbel-Softmax (GS) - to one-hot vectors\n\nA natural formulation for representing discrete variable is to use the categorical distribution. However,\npartially due to the inability to back-propagate information through samples, it seems rarely applied\nin deep learning. In this work, we resort to the Gumbel-Max trick [17] for enabling back-propagation\nand and representing the process of taking decision as sampling from a categorical distribution, in\norder to perform NAS in a principled way. Speci\ufb01cally, given a probability vector p = [p1,\u00b7\u00b7\u00b7 , pK]\nand a discrete random variable with P (L = k) \u221d pk, we sample from the discrete variable L by\nintroducing the Gumbel random variables. To be more speci\ufb01c, we let\n\nL = arg max\n\nk\u2208{1,\u00b7\u00b7\u00b7 ,K} log pk + Gk,\n\n(7)\n\nwhere {Gk}k\u2264K is a sequence of the standard Gumbel random variables, and they are typically\nsampled from the Gumbel distribution G = \u2212 log(\u2212 log(X)) with X \u223c U [0, 1]. An obstacle to\ndirectly using such approach in our problem is that the argmax operation is not really continuous.\nOne straightforward way of dealing with this problem is to replace the argmax operation with a\nsoftmax [25, 37]. Formally, the Gumbel-Softmax (GS) estimator can be expressed as\n\n(cid:80)K\n\n\u02c6Lk =\n\nexp ((log pk + Gk) /\u03c4 )\nk=1 exp ((log pk + Gk) /\u03c4 )\n\n, 1 \u2264 k \u2264 K,\n\n(8)\n\nwhere \u02c6Lk indicates the probability that pk is the maximal entry in p, and \u03c4 is a temperature. When\n\u03c4 \u2192 0, [ \u02c6L1,\u00b7\u00b7\u00b7 , \u02c6LK] converges to an one-hot vector, and in the other extreme it will become a\ndiscrete uniform distribution with \u03c4 \u2192 +\u221e.\nFrom the expression in Eq. (8), we see that GS estimator pertains solely to deal with the problems\nthat only one category requires to be determined, i.e., the outputs are one-hot vectors not any binary\ncode. In NAS, however, an optimal architecture may require multiple operations on one edge,\nconsidering the practical signi\ufb01cance [19, 49]. For instance, the residual module y = F (x) + I(x)\nin ResNets [19] consists of two operations with a learnable mapping F (\u00b7) and the identity I(\u00b7). That\nis, choosing different operations in O may not be mutually exclusive but compatible. One direct\nway of handling this limitation is to map all possible operation combinations to 2K-dimensional\nvectors, where K is the number of candidate operations in O. However, it seems dif\ufb01cult to search\narchitectures ef\ufb01ciently when there are many candidate operations, i.e., K is really large.\n\n4\n\nGumbel-softmaxEnsemble Gumbel-softmax\f3.2 Ensemble Gumbel-Softmax (EGS) - to any binary code\n\nIn order to address the aforementioned limitation in the traditional GS estimator, we propose an\nEnsemble Gumbel-Softmax (EGS) estimator to model the binary function f (\u00b7) formulated in Eq. (4),\nwhich is capable of choosing diversiform numbers of operations on different edges. To this end,\narchitectures are equally recoded into a group of one-hot vectors that can be sampled from probability\nvectors with the GS estimator. Because of the equivalency, in turn, any architecture is sampled by\ncompositing the results from the GS estimator.\nFor clarity of exposition, the recoding of A(i,j) \u2208 {0, 1}K is described only, where A(i,j) implies\nthe chosen operations on an edge e(i,j). Naturally, such a K-dimensional vector A(i,j) \u2208 {0, 1}K\ncan be recoded into a superposition of K one-hot vectors, i.e.,\n\nK(cid:88)\n\nk=1\n\nA(i,j) =\n\nvk \u00b7 a(i,j)\n\nk\n\n, vk \u2208 {0, 1}, 1 \u2264 k \u2264 K,\n\n(9)\n\nk\n\n\u2208 RK is a K-dimensional one-hot vector that uniquely corresponds to the operation\nwhere a(i,j)\nok \u2208 O, vk = 1 implies that the operation ok is chosen on edge e(i,j), and vk = 0 otherwise.\nBene\ufb01ting from the equivalency, any architecture code can be represented with a group of one-hot\nvectors, and one-hot vectors also can be sampled from probability vectors with the GS estimator.\nIntrinsically, such straightforward process can be considered as the inverse operation of the binary\nfunction f (\u00b7). That is, the problem of modeling the binary function f (\u00b7) can be recast as to \ufb01nd the\ninversion of such process.\nInspired from the above relationship between architecture codes, one-hot vectors and probability\nvectors, we model the binary function f (\u00b7) by introducing the inversion of this relationship. In\nFigure 2, a visualized comparison between the GS and EGS estimators intuitively shows that our\nEGS estimator is more excellent than the GS estimator, in terms of both sampling capability and\nrationality in practice. Given a probability vector, the EGS estimator, an ensemble of multiple GS\nestimators is profound for sampling any binary code, i.e.,\nDe\ufb01nition 1. For a K-dimensional probability vector p = [p1,\u00b7\u00b7\u00b7 , pK] \u2208 RK and M one-hot\nvectors {z(1),\u00b7\u00b7\u00b7 , z(M )} sampled from p with the GS estimator, the K-dimensional binary code\nb = [b1,\u00b7\u00b7\u00b7 , bK] \u2208 {0, 1}K sampled with EGS is\nz(i)\nk\n\n, 1 \u2264 k \u2264 K\n\n(cid:17)\n\n(cid:16)\n\nbk = max\n1\u2264i\u2264M\n\nwhere M is sampling times, bk is the k-th element in b, and z(i)\n\nk indicates the k-th element in z(i).\n\n3.3 Understanding EGS\n\nTo reveal the serviceability and sampling capability of the developed EGS estimator, according to\nDe\ufb01nition 1, two basic propositions are given in the following.\nProposition 1. For arbitrary probability vector p = [p1,\u00b7\u00b7\u00b7 , pK] and sampling times M, the\nK-dimensional binary code b \u2208 {0, 1}K sampled with EGS always meets\n\nP (bk1 = 1) \u2264 P (bk2 = 1) \u21d4 pk1 \u2264 pk2, 1 \u2264 k1, k2 \u2264 K,\n\nwhere P (bk = 1) is the probability of bk = 1, and P (bk1 = 1) = P (bk2 = 1) \u21d4 pk1 = pk2.\nProposition 1 means that the binary codes sampled with the EGS estimator strictly depend on the\nprobabilities at the corresponding locations. That is, EGS always tends to be a monotonically\nincreasing function in terms of probability, and suf\ufb01ces to act as the binary function f (\u00b7).\nProposition 2. For arbitrary probability vector p = [p1,\u00b7\u00b7\u00b7 , pK] and number of sampling times M,\n\n(cid:1) \u00d7(cid:0)2M \u2212 1(cid:1) different binary codes, which includes\n\nthe EGS estimator is capable of sampling(cid:0)K\n\nthe whole binary codes with up to M ones and at least 1 one.\n\nM\n\nProposition 2 indicates that the sampling capability of the EGS estimator increases exponentially\nwith M. In practice, larger M is always employed to deal with more complex tasks for effect, and\nsmaller one can be utilized to search more lightweight networks for ef\ufb01ciency.\n\n5\n\n\fTable 1: Comparison with state-of-the-art image classi\ufb01ers on CIFAR-10 (lower test error is better).\n\nArchitecture\n\nDenseNet-BC [22]\nPNAS [31]\nHierarchical evolution [33]\nAmoebaNet-A [44]\nAmoebaNet-B + cutout [44]\nNASNet-A + cutout [60]\nENAS + cutout [42]\nDARTS (1-th order) + cutout [34]\nDARTS (2-th order) + cutout [34]\nSNAS + mild + cutout [53]\nSNAS + moderate + cutout [53]\nSNAS + aggressive + cutout [53]\nRandom search baseline + cutout\nDATA (M = 4) + cutout\nDATA (M = 7) + cutout\n\nTest Error\n\nParams\n\n(%)\n3.46\n3.41\n3.75\n3.34\n2.55\n2.65\n2.89\n3.00\n2.76\n2.98\n2.85\n3.10\n3.29\n2.70\n2.59\n\n(M)\n25.6\n3.2\n15.7\n3.2\n2.8\n3.3\n4.6\n3.3\n3.3\n2.9\n2.8\n2.3\n3.2\n3.2\n3.4\n\nSearch Cost\n(GPU days)\n\nOps\n\nSearch\n\n-\n225\n300\n3150\n3150\n2000\n0.5\n1.5\n4\n1.5\n1.5\n1.5\n4\n1\n1\n\n-\n8\n6\n19\n19\n13\n6\n7\n7\n-\n-\n-\n7\n7\n7\n\nmanual\nSMBO\nevolution\nevolution\nevolution\n\nRL\nRL\n\ngradient-based\ngradient-based\ngradient-based\ngradient-based\ngradient-based\n\nrandom\n\ngradient-based\ngradient-based\n\nTable 2: Comparison with classi\ufb01ers on ImageNet in the mobile setting (lower test error is better).\n\nArchitecture\n\nInception-v1 [48]\nMobileNet [20]\nShuf\ufb02eNet-v2 2\u00d7 [36]\nPNAS [31]\nAmoebaNet-A [44]\nAmoebaNet-B [44]\nAmoebaNet-C [44]\nNASNet-A [60]\nNASNet-B [60]\nNASNet-C [60]\nDARTS (on CIFAR-10) [34]\nSNAS (mild constraint) [53]\nGDAS [18]\nDATA (M = 4)\nDATA (M = 7)\n\nTest Error (%)\nTop 5\nTop 1\n30.2\n10.1\n29.4\n10.5\n25.1\n25.8\n25.5\n26.0\n24.3\n26.0\n27.2\n27.5\n26.7\n27.3\n26.0\n25.5\n24.9\n\n-\n8.1\n8.0\n8.5\n7.6\n8.4\n8.7\n9.0\n8.7\n9.2\n8.5\n8.3\n8.0\n\nParams\n\nFLOPs\n\n(M)\n6.6\n4.2\n\u223c5\n5.1\n5.1\n5.3\n6.4\n5.3\n5.3\n4.9\n4.7\n4.3\n5.3\n4.9\n5.0\n\n(M)\n1448\n569\n591\n588\n555\n555\n570\n564\n488\n558\n574\n522\n581\n568\n588\n\nSearch Cost\n(GPU days)\n\n-\n-\n-\n\n\u223c225\n3150\n3150\n3150\n2000\n2000\n2000\n\n4\n1.5\n0.21\n1\n1\n\nSearch\n\nmanual\nmanual\nmanual\nSMBO\nevolution\nevolution\nevolution\n\nRL\nRL\nRL\n\ngradient-based\ngradient-based\ngradient-based\ngradient-based\ngradient-based\n\nSynthetically, the EGS estimator is capable of improving NAS by tactfully bridging the gap of\narchitectures between searching and validating, while maintaining the differentiability of the NAS\npipeline. In practice, the smaller gap guarantees the effectiveness of the inherited architectures from\nsearching to validating, and the differentiability implies that an end-to-end mechanism of searching\narchitectures is achieved for the requisite ef\ufb01ciency.\n\n4 Experiments\n\nIn this section, we systematically carry out extensive experiments to verify the capability of our\nmodel in discovering high-performance architectures. For each task, the experiments consist of two\nstages, following with the previous work [34, 53]. First, the cell architectures are searched based on\nour EGS estimator and the best cells are found according to their validation performance. Second, the\ntransferability of the best cells learned on CIFAR-10 [28] and Penn Tree Bank (PTB) [51] are investi-\ngated by using them on large datasets, i.e., classi\ufb01cation on ImageNet [10] and language modeling on\nWikiText-2 (WT2) [39], respectively. As a greatly improved work of DARTS, speci\ufb01cally, the experi-\nmental settings always inherit from it, except some special settings in each experiment. Speci\ufb01cally,\nthe core code of DATA is released at https://github.com/XinbangZhang/DATA-NAS.\n\nImage classi\ufb01cation\n\n4.1\nArchitecture search on CIFAR-10 In our experiments, the candidate primitive set O includes eight\ntypical operations, i.e., 3 \u00d7 3 and 5 \u00d7 5 separable convolutions, 3 \u00d7 3 and 5 \u00d7 5 dilated separable\nconvolutions, 3 \u00d7 3 max pooling, 3 \u00d7 3 average pooling, identity, and zero. In order to preserve\n\n6\n\n\fTable 3: Comparison with state-of-the-art language models on PTB (lower perplexity is better).\n\nArchitecture\n\nVariational RHN [57]\nLSTM [40]\nLSTM + skip connections [38]\nLSTM + 15 softmax experts [54]\nDARTS (\ufb01rst order) [34]\nDARTS (second order) [34]\nENAS [42]\nRandom search baseline\nDATA (M = 4)\nDATA (M = 7)\n\nPerplexity\ntest\n65.4\n58.8\n58.3\n56.0\n57.6\n55.7\n63.1\n59.4\n56.2\n55.3\n\nvalid\n67.9\n60.7\n60.9\n58.1\n60.2\n58.1\n68.3\n61.8\n58.3\n57.1\n\nParams Search Cost\n(M)\n(GPU days)\n23\n24\n24\n22\n23\n23\n24\n23\n23\n23\n\n-\n-\n-\n-\n0.5\n1\n0.5\n2\n0.5\n0.5\n\nOps\n\n-\n-\n-\n-\n4\n4\n4\n4\n4\n4\n\nSearch\n\nmanual\nmanual\nmanual\nmanual\n\ngradient-based\ngradient-based\n\nRL\n\nrandom\n\ngradient-based\ngradient-based\n\ntheir spatial resolution, all operations are of stride one, and the convolutional feature maps are\npadded if necessary. In EGS, the sampling time M is set to 4 and 7 for a rich search space. During\nsearching, the ReLU-Conv-BN order is utilized in the whole convolution operations, and every\nseparable convolution is always applied twice.\nThe settings of nodes in our convolutional cell are also following the previous work [60, 44, 31, 34].\nSpeci\ufb01cally, every cell consists of n = 7 nodes, among which the output node is de\ufb01ned as the\ndepthwise concatenation of all the intermediate nodes. The larger networks are always built by\nstacking multiple cells together. In the k-th cell, the \ufb01rst and second nodes are set equally to the\noutputs in the (k \u2212 2)-th and (k \u2212 1)-th cells respectively, with 1 \u00d7 1 convolution as necessary.\nFurthermore, the reduction cell with the reduce architecture is utilized at the 1/3 and 2/3 of the total\ndepth of the network. The rest of cells are the normal cell with the normal architecture. Speci\ufb01cally,\nthe searched cells are reported in the supplementary material.\nArchitecture validation on CIFAR-10 To evaluate the searched architecture, a large network of 20\ncells is trained from scratch for 600 epoches with batch size 96 and report its performance on the test\nset. For fair comparison, we set cutout with size 16, path dropout of probability 0.2 and auxiliary\ntowers with weight 0.4 following exiting works [60, 44, 31, 34]. We report the mean of 5 independent\nruns for our full model, with different initializations.\nTable 1 gives the searched architectures and classi\ufb01cation results on CIFAR-10, which shows that\nDATA achieves comparable results with the state-of-the-art with less computation resources. Such a\ngood performance veri\ufb01es that DATA can effectively and ef\ufb01ciently search worthy architectures for\nclassi\ufb01cation. In DATA, furthermore, higher accuracy is yielded when M = 7 compared with M = 4.\nThis scenario is in accordance with our motivation that more richer search spaces is bene\ufb01cial for\nsearching more better architectures. Speci\ufb01cally, the search progresses of different models are showed\nin Figure 3 (a), which shows that DATA requires less training epochs than SNAS (100 vs 150). A\npossible reason is that only child architectures are sampled and optimized in DATA, which is faster\nthan optimizing the whole network in SNAS.\nTransferability validation on ImageNet On ImageNet, the mobile setting where the input image\nsize is 224\u00d7224 and the number of multiply-add operations of the model is restricted to be under\n600M. An architecture of 14 cells is trained for 250 epoches with batch size 128, weight decay\n3\u00d710\u22125 and poly learning rate scheduler with initial learning rate 0.1. During training, label\nsmoothing [49] and auxiliary loss [29] are used. Other hyperparameters follow [34] and the mean of\n5 independent runs is used for validation.\nIn Table 2, we report the quantitative results on ImageNet. Note that the cell searched on CIFAR-10\ncan be smoothly employed to deal with the large-scale classi\ufb01cation task. Compared with other\ngradient-based NAS methods, furthermore, greater margins are yielded on ImageNet. A possible\nreason is that more complex architectures can be searched in DATA because of the larger search space.\nConsequently, such more complex architectures handle more complex task on ImageNet better.\n\n4.2 Language modeling\n\nArchitecture search on PTB In the language modeling task, DATA is used to search suitable\nactivation function between nodes. Following the setting in [60, 42, 34], \ufb01ve popular functions\nincluding sigmoid, tanh, relu, identity, and zero, are considered in the candidate primitive set O. In\n\n7\n\n\fTable 4: Comparison with state-of-the-art language models on WT2 (lower perplexity rate is better).\n\nArchitecture\n\nLSTM + augmented loss [23]\nLSTM + cache pointer [16]\nLSTM [40]\nLSTM + skip connections [38]\nLSTM + 15 softmax experts [54]\nDARTS (searched on PTB) [34]\nENAS (searched on PTB) [42]\nDATA (M = 4)\nDATA (M = 7)\n\nPerplexity\n\nParams\n\n-\n\nvalid\n91.5\n69.1\n69.1\n66.0\n69.5\n72.4\n67.3\n66.5\n\ntest\n87.0\n68.9\n66.0\n65.9\n63.3\n66.9\n70.4\n64.6\n64.2\n\n(M)\n28\n-\n33\n24\n33\n33\n33\n33\n33\n\nSearch Cost\n(GPU days)\n\n-\n-\n-\n-\n-\n1\n0.5\n1\n1\n\nSearch\n\nmanual\nmanual\nmanual\nmanual\nmanual\n\ngradient-based\n\nRL\n\ngradient-based\ngradient-based\n\nTable 5: Sensitivity to number of sampling times on CIFAR-10 (lower test error is better).\n9\n2.45\n3.97\n\nSampling Times (M)\nTest Error (%)\nParams (M)\n\n2.72\n3.41\n\n6\n\n2.60\n3.49\n\n8\n\n2.50\n3.79\n\n4\n\n2.70\n3.24\n\n2\n\n2.95\n2.68\n\n1\n\n2.94\n2.54\n\n3\n\n2.78\n2.71\n\n5\n\n7\n\n2.59\n3.44\n\nthe recurrent cell, there are n = 12 nodes and the sampling time M is set to 4 and 7 in ensemble\nGumbel-Softmax for a rich search space. Similar to ENAS [42] and DARTS [34], in cells, the very\n\ufb01rst intermediate node is obtained by linearly transforming the two input nodes, adding up the results\nand then passing through the tanh function, and the rest of activation functions are learned with DATA\nand enhanced with the highway [58]. The batch normalization [24] in each node to prevent gradient\nexplosion in searching, and disable it during validation. In addition, the recurrent network consists of\nonly a single cell, i.e., any repetitive pattern is not assumed in the recurrent architecture.\nArchitecture validation on PTB In this experiment, a single-layer recurrent network consisting\nof searched cells is trained with 1600 epoches, and batch size 64 using averaged SGD. Both of the\nembedding and the hidden sizes are set to 850 to ensure our model size is comparable with other\nbaselines. Other hyper-parameters are set following [34]. For a fair comparison, any additional\nenhancement is not utilized, such as \ufb01ne-tuned at the end of the optimization.\nTable 3 lists the results in this experiment. From the table, we observe that DATA also is in a position\nto search recurrent architectures effectively. It empirically shows that the back-propagation algorithm\ncan guide DATA to hit a preferable recurrent architecture, while maintaining the requisite ef\ufb01ciency.\nSimilar to the conclusion in the experiments on CIFAR-10, lower perplexity is achieved when the\nlarger M is used, which veri\ufb01es that a large search space is also valuable for recurrent architectures .\nTransferability validation on WT2 Different from the settings on PTB, on WT2, we apply emb-\ndding hidden sizes 700, weight decay 5\u00d710\u22127, and hidden-node variational dropout 0.15. Other\nhyperparameters remain the same in the experiment on PTB. In Table 4, the results on WT2 indicate\nthat the transferability is also retentive on recurrent architectures. Conclusively, the consistent results\nin the above experiments on ImageNet and WT2 strongly demonstrate the transferability on both\nconvolutional and recurrent architectures.\n\n4.3 Ablation study\n\nSensitivity to number of sampling times We perform experiments on CIFAR-10 to analyze the\nsensitivities to the number of sampling time M. Table 5 gives the results in this experiment. From\nthis table, it can be observed that larger M indicates higher performance, while more parameters will\nbe introduced as M increases. This is in accordance with the statement in Proposition 2, i.e., more\ncapable networks might be found with larger M to get higher performance.\nPerformance on semantic segmentation We validate the capability of DATA on a more complex\ntask, i.e., semantic segmentation on the PASCAL VOC-2012. In this experiment, we apply the\nnetwork structure searched on CIFAR-10 as feature extractor and combine it with the head adopted in\nDeeplab v3 [7]. Compared with NASNet [60] and DARTS [34], DATA achieves better performances\nwith the larger margins, especially when M is large. The results in Table 6 verify that DATA have\nmore prominent superiority on more complex tasks, not just toy tasks on the tiny datasets, because of\na large search space that is proportional to the sampling time M.\n\n8\n\n\fTable 6: Semantic segmentation on the PASCAL VOC-2012 (higher mIOU is better).\n\nArchitecture\nmIOU(%)\nParams (M)\n\nNASNet [60]\n\nDARTS [34]\n\nDATA (M = 1)\n\nDATA (M = 4)\n\nDATA (M = 7)\n\n73.7\n12.4\n\n73.2\n11.8\n\n73.4\n10.8\n\n74.1\n11.7\n\n75.6\n12.7\n\nIn\ufb02uence of initializations\nTo investigate the stability of the EGS estimator, we compare the performance of DATA under\ninitializations. Speci\ufb01cally. in Table 7 and Figure 3 (b), the stds of DATA and the variances of\nDATA with different sampling time (M) are reported, which indicate that DATA (M=1,std=0.08)\nachieves lower stds than DARTS (k=1,std=0.14). Furthermore, Figure 3 (c) shows the variances of\narchitectures in search process for a comprehensive analysis.\nContribution of Ensemble Gumbel-Softmax\nTo explore the contribution of the EGS estimator on DATA, we study the gaps of of the validation\nerrors between search networks and child networks on CIFAR-10. For different models, Table 8\nreports the comparison of the validation errors at the end of search and after architecture derivation\nwithout \ufb01ne-tuning. The results show that DATA (gap=1.87, err=9.21) is superior to maintain\nbetter performance compared with SNAS (gap=2.13, err=9.33) and especially DARTS (gap=33.03,\nerr=45.34). This demonstrates that the EGS estimator endows DATA the capability of bridging the\ngap of architectures during searching and validating.\n\n5 Conclusion\n\nWe present DATA to bridge the gap of architectures during searching and validating in a differentiable\nmanner. For this purpose, the EGS estimator that consists of an ensemble of a group of Gumbel-\nSoftmax estimators is developed, which is in a position to sample an architecture that approaches to\nthe one during searching as close as possible, while guaranteeing the requisite ef\ufb01ciency. By searching\nwith the standard back-propagation, DATA is able to outperform the state-of-the-art architecture\nsearch methods on various tasks, with remarkably better ef\ufb01ciency.\nFuture work may include searching the whole networks with our EGS estimator and injecting the EGS\nestimator into deep models to handle other machine learning tasks. For the \ufb01rst work, the sampling\ncapability of the EGS estimator guarantees the practicability of searching any networks, but how to\nfurther improve the ef\ufb01ciency remains to be solved. For the second work, the differentiability of the\nEGS estimator indicates that it can be utilized anywhere in networks, i.e., an interesting direction is\nto recast the clustering process into our ensemble Gumbel-Softmax. By aggregating inputs in each\ncluster, conclusively, a general pooling for both deep networks and deep graph networks [1, 5, 6] can\nbe developed to deal with Euclidean and non-Euclidean structured data uniformly.\n\nAcknowledgments\n\nThis research was supported by Major Project for New Generation of AI Grant No. 2018AAA0100402,\nand the National Natural Science Foundation of China under Grants 91646207, 61976208, 61773377,\nand 61573352. We would like to thank Lele Yu, Jie Gu, Cheng Da, and Yukang Chen for their\ninvaluable contributions in shaping the early stage of this work.\n\n9\n\nTable1:ValidationerroronCIFAR-10.ModelSearchChildGapTable2:ValidationerroronCIFAR-10.ModelSearchChildGapTable3:ValidationerroronCIFAR-10.ModelSearchChildGapTable4:ValidationerroronCIFAR-10.ModelSearchChildGapTable5:ValidationerroronCIFAR-10.ModelSearchChildGapTable6:ValidationerroronCIFAR-10.ModelSearchChildGapTable7:NumberofoperationsonCIFAR-10.ModelError(%)Params(M)DARTS(k=1)3.00\u00b10.143.30DARTS(k=2)3.10\u00b10.124.00DARTS(k=3)2.95\u00b10.135.20SNAS2.85\u00b10.022.80DATA(M=1)2.94\u00b10.092.54DATA(M=4)2.70\u00b10.103.24DATA(M=7)2.59\u00b10.093.44Table8:ValidationerroronCIFAR-10.ModelSearchChildGapDARTS12.3345.3433.01SNAS11.469.332.13DATA(M=7)11.089.211.87105010015020.040.060.080.05010015020.040.060.080.0Epochsvalidationaccuracy(a):Searchprogress.ENASDARTSSNASDATA1245797.097.297.4SamplingTimes(M)validationaccuracy(b):Differentsamplings.2040608010096.897.097.297.4Epochsvalidationaccuracy(c):Differentinitializations.Figure 3: Ablation study.\fReferences\n[1] Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vin\u00edcius Flores Zambaldi,\nMateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, \u00c7aglar G\u00fcl\u00e7ehre,\nFrancis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey Allen, Charles\nNash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew Botvinick,\nOriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph\nnetworks. CoRR, abs/1806.01261, 2018.\n\n[2] Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, and Samy Bengio. Neural combinatorial\n\noptimization with reinforcement learning. CoRR, abs/1611.09940, 2016.\n\n[3] Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V. Le. Neural optimizer search with reinforcement\n\n[4] Francesco Paolo Casale, Jonathan Gordon, and Nicol\u00f3 Fusi. Probabilistic neural architecture search. CoRR,\n\nlearning. In ICML, pages 459\u2013468, 2017.\n\nabs/1902.05116, 2019.\n\n[5] Jianlong Chang, Jie Gu, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Structure-\n\naware convolutional neural networks. In NeurIPS, pages 11\u201320, 2018.\n\n[6] Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Qi Zhang, Shiming Xiang, and Chunhong Pan. Local-\n\naggregation graph networks. IEEE T-PAMI, 2019.\n\n[7] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution\n\nfor semantic image segmentation. CoRR, abs/1706.05587, 2017.\n\n[8] Liang-Chieh Chen, Maxwell D. Collins, Yukun Zhu, George Papandreou, Barret Zoph, Florian Schroff,\nHartwig Adam, and Jonathon Shlens. Searching for ef\ufb01cient multi-scale architectures for dense image\nprediction. In NeurIPS, 2018.\n\n[9] Yukang Chen, Tong Yang, Xiangyu Zhang, Gaofeng Meng, Chunhong Pan, and Jian Sun. Detnas: Neural\n\narchitecture search on object detection. CoRR, abs/1903.10979, 2019.\n\n[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical\n\n[11] Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. In CVPR, pages\n\nimage database. In CVPR, 2009.\n\n1761\u20131770, 2019.\n\n[12] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Simple and ef\ufb01cient architecture search for\n\nconvolutional neural networks. In ICLR Workshop Track Proceedings, 2018.\n\n[13] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Ef\ufb01cient multi-objective neural architecture search\n\nvia lamarckian evolution. 2018.\n\n[14] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of\n\nMachine Learning Research, 20:55:1\u201355:21, 2019.\n\n[15] Golnaz Ghiasi, Tsung-Yi Lin, Ruoming Pang, and Quoc V. Le. NAS-FPN: learning scalable feature\n\npyramid architecture for object detection. CoRR, abs/1904.07392, 2019.\n\n[16] Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with a continuous\n\ncache. CoRR, abs/1612.04426, 2016.\n\n[17] Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: a series of\n\nlectures. Number 33. US Govt. Print. Of\ufb01ce, 1954.\n\n[18] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single\n\npath one-shot neural architecture search with uniform sampling. CoRR, abs/1904.00420, 2019.\n\n[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.\n\nIn CVPR, 2016.\n\n[20] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand,\nMarco Andreetto, and Hartwig Adam. Mobilenets: Ef\ufb01cient convolutional neural networks for mobile\nvision applications. CoRR, abs/1704.04861, 2017.\n\n[21] Chi-Hung Hsu, Shu-Huan Chang, Da-Cheng Juan, Jia-Yu Pan, Yu-Ting Chen, Wei Wei, and Shih-\nChieh Chang. MONAS: multi-objective neural architecture search using reinforcement learning. CoRR,\nabs/1806.10332, 2018.\n\n[22] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected\n\nconvolutional networks. In CVPR, 2017.\n\n[23] Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classi\ufb01ers: A loss\n\nframework for language modeling. CoRR, abs/1611.01462, 2016.\n\n[24] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing\n\ninternal covariate shift. In ICML, pages 448\u2013456, 2015.\n\n[25] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. CoRR,\n\n[26] Purushotham Kamath, Abhishek Singh, and Debo Dutta. Neural architecture construction using en-\n\nabs/1611.01144, 2016.\n\nvelopenets. CoRR, abs/1803.06744, 2018.\n\n[27] Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnab\u00e1s P\u00f3czos, and Eric P. Xing. Neural\narchitecture search with bayesian optimisation and optimal transport. In NeurIPS, pages 2020\u20132029, 2018.\n[28] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Master\u2019s\n\nThesis, Department of Computer Science, University of Torono, 2009.\n\n10\n\n\f[29] Chen-Yu Lee, Saining Xie, Patrick W. Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised\n\n[30] Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. CoRR,\n\nnets. In AISTATS, 2015.\n\nabs/1902.07638, 2019.\n\n[31] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan L.\n\nYuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018.\n\n[32] Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L. Yuille, and Li Fei-\nFei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. CoRR,\nabs/1901.02985, 2019.\n\n[33] Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hierarchical\n\nrepresentations for ef\ufb01cient architecture search. CoRR, abs/1711.00436, 2017.\n\n[34] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. CoRR,\n\nabs/1806.09055, 2018.\n\nNeurIPS, pages 7827\u20137838, 2018.\n\n[35] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. In\n\n[36] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shuf\ufb02enet V2: practical guidelines for\n\nef\ufb01cient CNN architecture design. In ECCV, 2018.\n\n[37] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation\n\nof discrete random variables. CoRR, abs/1611.00712, 2016.\n\n[38] G\u00e1bor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models.\n\n[39] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models.\n\nCoRR, abs/1707.05589, 2017.\n\nCoRR, abs/1609.07843, 2016.\n\nmodels. CoRR, abs/1708.02182, 2017.\n\n[40] Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language\n\n[41] Risto Miikkulainen, Jason Zhi Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala\nRaju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, and Babak Hodjat. Evolving deep neural\nnetworks. CoRR, abs/1703.00548, 2017.\n\n[42] Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Ef\ufb01cient neural architecture search\n\nvia parameter sharing. In ICML, 2018.\n\n[43] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le,\n\nand Alexey Kurakin. Large-scale evolution of image classi\ufb01ers. In ICML, 2017.\n\n[44] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classi\ufb01er\n\narchitecture search. CoRR, abs/1802.01548, 2018.\n\n[45] Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the\n\nsearch phase of neural architecture search. CoRR, abs/1902.08142, 2019.\n\n[46] Richard Shin, Charles Packer, and Dawn Song. Differentiable neural network architecture search. 2018.\n[47] Kenneth O Stanley, Jeff Clune, Joel Lehman, and Risto Miikkulainen. Designing neural networks through\n\nneuroevolution. Nature Machine Intelligence, 1(1):24\u201335, 2019.\n\n[48] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru\n\nErhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.\n\n[49] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking\n\nthe inception architecture for computer vision. In CVPR, 2016.\n\n[50] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V. Le. Mnasnet: Platform-aware\n\nneural architecture search for mobile. CoRR, abs/1807.11626, 2018.\n\n[51] Ann Taylor, Mitchell Marcus, and Beatrice Santorini. The penn treebank: an overview. In Treebanks. 2003.\n[52] Du Tran, Jamie Ray, Zheng Shou, Shih-Fu Chang, and Manohar Paluri. Convnet architecture search for\n\nspatiotemporal feature learning. CoRR, abs/1708.05038, 2017.\n\n[53] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. CoRR,\n\nabs/1812.09926, 2018.\n\n[54] Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck:\n\nA high-rank RNN language model. CoRR, abs/1711.03953, 2017.\n\n[55] Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. Nas-bench-101:\n\nTowards reproducible neural architecture search. CoRR, abs/1902.09635, 2019.\n\n[56] Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network\n\n[57] Julian G. Zilly, Rupesh Kumar Srivastava, Jan Koutn\u00edk, and J\u00fcrgen Schmidhuber. Recurrent highway\n\n[58] Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutn\u00edk, and J\u00fcrgen Schmidhuber. Recurrent highway\n\n[59] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. CoRR,\n\narchitecture generation. In CVPR, 2018.\n\nnetworks. CoRR, abs/1607.03474, 2016.\n\nnetworks. In ICML, 2017.\n\nabs/1611.01578, 2016.\n\n[60] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for\n\nscalable image recognition. In CVPR, 2018.\n\n11\n\n\f", "award": [], "sourceid": 477, "authors": [{"given_name": "Jianlong", "family_name": "Chang", "institution": "National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences"}, {"given_name": "xinbang", "family_name": "zhang", "institution": "Institute of Automation,Chinese  Academy of Science"}, {"given_name": "Yiwen", "family_name": "Guo", "institution": "Bytedance AI Lab"}, {"given_name": "GAOFENG", "family_name": "MENG", "institution": "Institute of Automation, Chinese Academy of Sciences"}, {"given_name": "SHIMING", "family_name": "XIANG", "institution": "Chinese Academy of Sciences, China"}, {"given_name": "Chunhong", "family_name": "Pan", "institution": "Institute of Automation, Chinese Academy of Sciences"}]}