{"title": "Exploring Unexplored Tensor Network Decompositions for Convolutional Neural Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 5552, "page_last": 5562, "abstract": "Tensor decomposition methods are widely used for model compression and fast inference in convolutional neural networks (CNNs). Although many decompositions are conceivable, only CP decomposition and a few others have been applied in practice, and no extensive comparisons have been made between available methods. Previous studies have not determined how many decompositions are available, nor which of them is optimal. In this study, we first characterize a decomposition class specific to CNNs by adopting a flexible graphical notation. The class includes such well-known CNN modules as depthwise separable convolution layers and bottleneck layers, but also previously unknown modules with nonlinear activations. We also experimentally compare the tradeoff between prediction accuracy and time/space complexity for modules found by enumerating all possible decompositions, or by using a neural architecture search. We find some nonlinear decompositions outperform existing ones.", "full_text": "Einconv: Exploring Unexplored Tensor Network\n\nDecompositions for Convolutional Neural Networks\n\nKohei Hayashi\n\nPreferred Networks\n\nTaiki Yamaguchi\u02da\n\nThe University of Tokyo\n\nhayasick@preferred.jp\n\nyamaguchi@hep-th.phys.s.u-tokyo.ac.jp\n\nYohei Sugawara\n\nPreferred Networks\n\nsuga@preferred.jp\n\nShin-ichi Maeda\nPreferred Networks\n\nichi@preferred.jp\n\nAbstract\n\nTensor decomposition methods are widely used for model compression and fast in-\nference in convolutional neural networks (CNNs). Although many decompositions\nare conceivable, only CP decomposition and a few others have been applied in\npractice, and no extensive comparisons have been made between available methods.\nPrevious studies have not determined how many decompositions are available, nor\nwhich of them is optimal. In this study, we \ufb01rst characterize a decomposition class\nspeci\ufb01c to CNNs by adopting a \ufb02exible graphical notation. The class includes\nsuch well-known CNN modules as depthwise separable convolution layers and\nbottleneck layers, but also previously unknown modules with nonlinear activa-\ntions. We also experimentally compare the tradeoff between prediction accuracy\nand time/space complexity for modules found by enumerating all possible de-\ncompositions, or by using a neural architecture search. We \ufb01nd some nonlinear\ndecompositions outperform existing ones.\n\n1\n\nIntroduction\n\nConvolutional neural networks (CNNs) typically process spatial data such as images using multiple\nconvolutional layers [Goodfellow et al., 2016]. The high performance of CNNs is often offset by\ntheir heavy demands on memory and CPU/GPU, making them problematic to deploy on edge devices\nsuch as mobile phones [Howard et al., 2017].\n\nOne straightforward approach to reducing costs is the introduction of a low-dimensional linear\nstructure into the convolutional layers [Smith et al., 1997, Rigamonti et al., 2013, Tai et al., 2015,\nKim et al., 2015, Denton et al., 2014, Lebedev et al., 2014, Wang et al., 2018]. This typically is done\nthrough tensor decomposition, which represents the convolution \ufb01lter in sum-product form, reducing\nthe number of parameters to save memory space and reduce the calculation cost for forwarding paths.\n\nThe manner in which this cost reduction is achieved depends heavily on the structure of the tensor\ndecomposition. For example, if the target is a two-way tensor, i.e., a matrix, the only meaningful\ndecomposition is X \u201c UV, because others such as X \u201c ABC are reduced to that form but have\nmore parameters. However, for higher-order tensors, there are many possible ways to perform tensor\ndecomposition, of which only a few have been actively studied (e.g., see [Kolda and Bader, 2009]).\nSuch multi-purpose decompositions have been applied to CNNs but are not necessarily optimal for\nthem, because of tradeoffs between prediction accuracy and time/space complexity. The need to\n\n\u02daThis work was completed during an internship at Preferred Networks.\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fX\n\nh\n\nw\n\n\u02da\n\nc\n\n\u02da\n\nh1\n\nw1\n\ni\n\nj\n\nT\n\nc1\n\nX\n\n\u02da\n\n\u02da\n\nX\n\n\u02da\n\n\u02da\n\nX\n\n\u02da\n\n\u02da\n\n(a) Standard\n\n(b) Depthwise Separable\n\n(c) Bottleneck/Tucker-2\n\n(d) Inverted Bottleneck\n\nX\n\n\u02da\n\n\u02da\n\n\u02da\n\n\u02da\n\nX\n\n\u02da\n\n\u02da\n\nX\n\n\u02da\n\n\u02da\n\nX\n\n\u02da\n\n\u02da\n\n(e) Factoring\n\n(f) Flattened\n\n(g) CP\n\n(h) Low-rank Filter\n\nFigure 1: Visualizing linear structures in various convolutional layers, where X is input and T is a\nconvolution kernel. The \u201clegs\u201d h1, w1, c1 respectively represent the spatial height, spatial width, and\noutput channels. We will further explain these diagrams in Section 3.\n\nconsider many factors, including application domains, tasks, entire CNN architectures, and hardware\nlimitations, makes the emergence of new optimization techniques inevitable.\n\nIn this study, we investigate a hidden realm of tensor decompositions to identify maximally resource-\nef\ufb01cient convolutional layers. We \ufb01rst characterize a decomposition class speci\ufb01c to CNNs by\nadopting a \ufb02exible hypergraphical notation based on tensor networks [Penrose, 1971]. The class\ncan deal with nonlinear activations, and includes modern light-weight CNN layers such as the\nbottleneck layers used in ResNet [He et al., 2015], the depthwise separable layers used in Mobilenet\nV1 [Howard et al., 2017], the inverted bottleneck layers used in Mobilenet V2 [Sandler et al.,\n2018], and others, as shown in Figure 1. The notation permits us to handle convolutions in three\nor more dimensions straightforwardly.\nIn our experiments, we study the accuracy/complexity\ntradeoff by enumerating all possible decompositions for 2D and 3D image data sets. Furthermore,\nwe evaluate nonlinear extensions by combining neural architecture search with the LeNet and\nResNet architectures. The code implemented in Chainer [Tokui et al., 2019] is available at https:\n//github.com/pfnet-research/einconv.\n\nNotation We use the notation rns \u201c t1, . . . , nu, where n is a positive integer. Lower-case letters\ndenote scalars when in ordinary type, vectors when in bold (e.g., a, a). Upper-case letters denote\nmatrices when in bold, tensors when in bold script (e.g. A, A).\n\n2 Preliminaries\n\n2.1 Convolution in Neural Networks\n\nConsider a 2D image of height H P N, width W P N, and number of channels C P N, where a\nchannel is a feature (e.g., R, G, or B) possessed by each pixel. The image can be represented by a\nthree-way tensor X P RH\u02c6W \u02c6C . Typically, the convolution operation, applied to such a tensor, will\nchange the size and the number of channels. We assume that the size of the convolution \ufb01lter is odd.\nLet I, J P t1, 3, 5, . . . u be the \ufb01lter\u2019s height and width, P P N be the padding size, and S P N be the\nstride. Then, the output has height H 1 \u201c pH ` 2P \u00b4 Iq{S ` 1 and width W 1 \u201c pW ` 2P \u00b4 Jq{S ` 1.\nWhen we set the number of output channels to C 1 P N, the convolution layer yields an output\nZ P RH 1\u02c6W 1\u02c6C1\n\nin which each element is given as\n\ntijcc1 xh1\n\niw1\n\nj c,\n\n(1)\n\nzh1w1c1 \u201c \u00ff\niPrIs\n\n\u00ff\njPrJs\n\n\u00ff\ncPrCs\n\n2\n\n\fHere T P RI\u02c6J\u02c6C\u02c6C1\ni \u201c ph1 \u00b4 1qS ` i \u00b4 P\nj \u201c pw1 \u00b4 1qS ` j \u00b4 P are spatial indices used for convolution. For simplicity, we omit the\nand w1\nbias parameter. There are IJ CC 1 parameters, and the time complexity of (1) is OpIJ CH 1W 1C 1q.\n\nis a weight, which is termed as the I \u02c6 J kernel, and h1\n\nAlthough (1) is standard, there are several important special cases used to reduce computational\ncomplexity. The case when I \u201c J \u201c 1 is called 1 \u02c6 1 convolution [Lin et al., 2013, Szegedy et al.,\n2015]; it applies a linear transformation to the channels only, and does not affect the spatial directions.\nDepthwise convolution [Chollet, 2016] is arguably the opposite of 1 \u02c6 1 convolution: it works as\nthough the input and output channels (rather than the spatial dimensions) are one dimensional, i.e.,\n\nzh1w1c1 \u201c \u00ff\niPrIs\n\n\u00ff\njPrJs\n\ntijc1 xh1\n\niw1\n\nj c1 .\n\n(2)\n\n2.2 Tensor Decomposition in Convolution\n\nTo reduce computational complexity, Kim et al. [2015] applied Tucker-2 decomposition [Tucker,\n1966] to the kernel T , replacing the original kernel by T T2, where each element is given by\n\nijcc1 \u201c \u00ff\ntT2\n\u03b1PrAs\n\n\u00ff\n\u03b2PrBs\n\ngij\u03b1\u03b2uc\u03b1vc1\u03b2.\n\n(3)\n\nHere G P RI\u02c6J\u02c6A\u02c6B, U P RC\u02c6A, V P RC1\u02c6B are new parameters, and A, B P N are rank-\nlike hyperparameters. Note that convolution with the Tucker-2 kernel T T2 is equivalent to three\nconsecutive convolutions: 1 \u02c6 1 convolution with kernel U, I \u02c6 J convolution with kernel G, and\n1 \u02c6 1 convolution with kernel V. The hyperparameters A and B may be viewed as intermediate\nchannels during the three convolutions. Hence, when A, B are smaller than C, C 1, a cost reduction\nis expected, because the heavy I \u02c6 J convolution is now being taken with the A, B channel pair\ninstead of with C, C 1. The reduction ratios of the number of parameters and the inference cost for the\nTucker-2 decomposition compared to the original are both at least AB{CC 1.\n\nSimilarly, several authors [Denton et al., 2014, Lebedev et al., 2014] have employed CP decomposi-\ntion [Hitchcock, 1927], which reparametrizes the kernel as\n\nijcc1 \u201c \u00ff\ntCP\n\u03b3Pr\u0393s\n\n\u02dcui\u03b3 \u02dcvj\u03b3 \u02dcwc\u03b3 \u02dcsc1\u03b3 ,\n\n(4)\n\nwhere \u02dcU, \u02dcV, \u02dcW, \u02dcS are new parameters and \u0393 P N is a hyperparameter.\n\n3 The Einconv Layer\n\nWe have seen that both the convolution operation (1), (2) and the decompositions of the kernel\n(3), (4) are given as the sum-product of tensors with many indices. Although the indices may cause\nexpressions to appear cluttered, they play important roles. There are two classes of indices: those\nconnected to the output shape (h1, w1, c1) and those used for summation (i, j, c, \u03b1, \u03b2, \u03b3). Convolution\nand its decomposition are speci\ufb01ed by how the indices interact and are distributed into tensor variables.\nFor example, in Tucker-2 decomposition, the spatial, input channel, and output channel information\nG, U, V are separated through their respective indices pi, jq, c1, c. Moreover, they are joined by\ntwo-step connections: the input channel and spatial information are connected by \u03b1, and the output\nchannel and spatial information by \u03b2. Here, we can consider the summation indices to be paths that\ndeliver input information to the output.\n\nA hypergraph captures the index interaction in a clean manner. The basic idea is that tensors are\ndistinguished only by the indices they own and we consider them as vertices. Vertices are connected\nif the corresponding tensors share indices to be summed. (For notational simplicity, we will often\nrefer to a tensor by its indices alone, i.e., U \u201c puabcqaPrAs,bPrBs,cPrCs is equivalent to ta, b, cu.)\nAs an example, consider the decomposition of a kernel T . Let the outer indices O \u201c ti, j, c, c1u be\nthe indices of the shape of T , the inner indices I \u201c pr1, r2, . . . q be the indices used for summation,\nand inner dimensions R \u201c pR1, R2, . . . q P R|I| be the dimensions of I. Assume that M P N tensors\nare involved in the decomposition, and let V \u201c tv1, . . . , vM | vm P 2OYIu denote the set of these\ntensors, where 2A denotes the power set of a set A. Given V, each inner index r P I de\ufb01nes a\n\n3\n\n\fhyperedge er \u201c tv | r P v for v P Vu. Let E \u201c ten | n P O Y Iu denote the set of hyperedges. For\nexample, suppose I \u201c t\u03b1, \u03b2u and V \u201c tti, j, \u03b1, \u03b2u, tc, \u03b1u, tc1, \u03b2uu; then, the undirected weighted\nhypergraph pV, E, Rq is equivalent to Tucker-2 decomposition (3).\n\nThis idea is also applicable to the convolution operation by the introduction of dummy tensors\nthat absorb the index patterns used in convolution. Recall that in (1) the special index h1\ni indicates\nwhich vertical elements of the kernel and the input image are coupled in the convolution. Let\nP P t0, 1uH\u02c6H 1\u02c6I be a (dummy) binary tensor where each element is de\ufb01ned as phh1i \u201c 1 if h \u201c h1\ni\nand 0 otherwise, and let Q P t0, 1uW \u02c6W 1\u02c6J be the horizontal counterpart of P. Furthermore, let\nus modify the index sets to O \u201c th1, w1, c1u and I \u201c ph, w, i, j, cq, and the dimensions to R \u201c\npH, W, I, J, Cq. Then, vertices V \u201c tth, w, cu, ti, j, c, c1u, th, h1, iu, tw, w1, juu and hyperedges E\nthat are automatically de\ufb01ned by V exactly represent the convolution operation (1), where we ensure\nthat the tensor of th, h1, iu is \ufb01xed by P and the tensor of tw, w1, ju is \ufb01xed by Q.\n\nThe above mathematical explanation may sound too winding, but visualization will help greatly. Let\nus introduce several building blocks for the visualization. Let a circle (vertex) indicate a tensor, and a\nline (edge) connected to the circle indicate an index associated with that tensor. When an edge is\nconnected on only one side, it corresponds to an outer index of the tensor; otherwise, it corresponds\nto an inner index used for summation. The summation and elimination of inner indices is called\ncontraction. For example,\n\nA\n\nj\n\nB\n\ni\n\nk\n\n\u201c\n\nC\n\ni\n\nk\n\n\u00f0\u00f1 \u00ff\n\naijbjk \u201c cik.\n\nj\n\nA hyperedge that is connected to more than three vertices is depicted with a black dot:\n\nA\n\nj\n\nB\n\ni\n\nC\n\nk\n\n\u00f0\u00f1 \u00ff\n\naijbjcjk.\n\nj\n\n(5)\n\n(6)\n\nFinally, a node with symbol \u201c\u02da\u201d indicates a dummy tensor. In our context, this implicitly indicates\nthat some spatial convolution is involved:\n\nh1\n\n\u02da\n\nB\n\ni\n\nA\n\nh\n\n\u00f0\u00f1 \u00ff\n\nphh1iahbi\n\nh,i\n\n(7)\n\nThe use of a single hyperedge to represent the summed inner index is the graphical equivalent of\nthe Einstein summation convention in tensor algebra. Inspired by this equivalence and by NumPy\u2019s\neinsum function [Wiebe, 2011], we term a hypergraphically-representable convolution layer an\nEinconv layer.\n\n3.1 Examples\n\nIn Figure 1, we give several examples of hypergraphical notation. Many existing CNN modules can\nobviously be described as Einconv layers (but without nonlinear activation).\n\nSeparable and Low-rank Filters Although a kernel is usually square, i.e., I \u201c J , we often take\nthe convolution separately along the vertical and horizontal directions. In this case, the convolution\noperation is equivalent to the application of two \ufb01lters of sizes pI, 1q and p1, Jq. This can be\nconsidered the rank-1 approximation of the I \u02c6 J convolution. A separable \ufb01lter [Smith et al., 1997]\nis a technique to speed up convolution when the \ufb01lter is exactly of rank one. Rigamonti et al. [2013]\nextended this idea by approximating \ufb01lters as low-rank matrices for a single input channel, and Tai\net al. [2015] further extended it for multiple input channels (Figure 1h).\n\nFactored Convolution In the case of a large \ufb01lter size, factored convolution is commonly used to\nreplace the large \ufb01lter with multiple small-sized convolutions [Szegedy et al., 2016]. For example,\ntwo consecutive 3 \u02c6 3 convolutions are equivalent to one 5 \u02c6 5 convolution in which the \ufb01rst 3 \u02c6 3\n\ufb01lter has been enlarged by the second 3 \u02c6 3 \ufb01lter. Interestingly, the factorization of convolution is\nexactly represented in Einconv by adding two additional dummy tensors.\n\n4\n\n\fh\n\nw\n\nd\n\nX\n\nh1\n\ni\nw1\nj\n\nd1\n\nk\n\n\u02da\n\u02da\n\u02da\nc\n\nc1\n\nT\n\nX\n\n\u02da\n\u02da\n\u02da\n\n\u02da\n\u02da\n\u02da\n\nX\n\n(a) Standard\n\n(b) Depthwise Separable\n\n(c) (2+1)D\n\nFigure 2: Graphical visualizations of 3D convolutions.\n\nBottleneck Layers\nIn ResNet [He et al., 2015], the bottleneck module is used as a building\nblock: input channels are reduced before convolution, and then expanded afterwards. Finally, a\nskip connection is used, re-adding the original input. Figure 1c shows the module without the skip\nconnection. From the diagram, we see that the linear structure of the bottleneck is equivalent to\nTucker-2 decomposition.\n\nDepthwise Separable Convolution Mobilenet V1 [Howard et al., 2017] is a seminal light-weight\narchitecture. It employs depthwise separable convolution [Sifre and Mallat, 2014, Chollet, 2016] as a\nbuilding block; this is a combination of depthwise and 1 \u02c6 1 convolution (Figure 1b), and works well\nwith limited computational resources.\n\nInverted Bottleneck Layers Mobilenet V2 [Sandler et al., 2018], the second generation of Mo-\nbilenet, employs a building block called the inverted bottleneck module (Figure 1d). It is similar\nto the bottleneck module, but there are two differences. First, whereas in the bottleneck module,\nthe number of intermediate channels is smaller than the number of input or of output channels, in\nthe inverted bottleneck this relationship is reversed, and the intermediate channels are \u201cballooned\u201d.\nSecond, there are two intermediate channels in the bottleneck module, while the inverted bottleneck\nhas only one.\n\n3.2 Higher Order Convolution\n\nWe have, thus far, considered 2D convolution, but Einconv can naturally handle higher-order convolu-\ntion. For example, consider a 3D convolution. Let d, d1 be the input/output indices for depth, and k\nbe the index of \ufb01lter depth. Then, by adding d1 to O and d, k to I, we can construct a hypergraph\nfor 3D convolution. Figure 2 shows the hypergraphs for the the standard 3D convolution and for\ntwo light-weight convolutions: the depthwise separable convolution [K\u00f6p\u00fckl\u00fc et al., 2019], and\nthe (2+1)D convolution [Tran et al., 2018] which factorizes a full 3D convolution into 2D and 1D\nconvolutions.\n\n3.3 Reduction and Enumeration\n\nAlthough the hypergraphical notation is powerful, we need to be careful about its redundancy. For\nexample, consider a hypergraph pV, Eq where an inner index a P I is only used by the m-th tensor,\ni.e, a P vm and a R vn for n \u2030 m. Then, any tensors represented by pV, Eq, whatever their inner\ndimensions, are also represented by removing a from every element of V and the a-th hyperedge\nfrom E. Similarly, self loops do not increase the representability [Ye and Lim, 2018]. In terms\nof representability of the Einconv layer, there is no reason to choose redundant hypergraphs.2 We\ntherefore want to remove them ef\ufb01ciently.\n\nFor simplicity, let us consider the 2D convolution case, in which the results are straightforwardly\nextensible to higher-order cases. Let z denote the set difference operator and n denote the element-\nwise set difference operator, which is used to remove an index from all vertices, e.g., V n a \u201c\ntv1za, . . . , vM zau for index a P O Y I. For convenience, we de\ufb01ne a map \u03b8 : O Y I \u00d1 N that\n\n2It might be possible that some redundant Einconv layer outperforms equivalent nonredundant ones, because\n\nparametrization in\ufb02uences optimization. However, we focus here on representability alone.\n\n5\n\n\freturns the dimension of an index a P O Y I, e.g. \u03b8piq \u201c I. To discuss representability, we introduce\nthe following notation for the space of Einconv layers:\nDe\ufb01nition 1. Given vertices V \u201c tv1, . . . , vM u and inner dimensions R, let FV : R\u015aaPv1\n\u00a8 \u00a8 \u00a8 \u02c6 R\u015aaPvM\nTV pRq \u010e RI\u02c6J\u02c6C\u02c6C1\nR\u015aaPvm\n\n\u03b8paq \u02c6\nbe the contraction of M tensors along with V. In addition, let\nbe the space that FV covers, i.e., TV pRq \u201c tFV pU1, . . . , UM q | Um P\n\n\u03b8paq \u00d1 RI\u02c6J\u02c6C\u02c6C1\n\n\u03b8paq for m P rM su.\n\nNext, we show several suf\ufb01cient conditions for hypergraphs to be redundant.\nProposition 1 (Ye and Lim 2018, Proposition 3.5). Given inner dimensions R P R|I|, if Ra \u201c 1,\nTV pRq is equivalent to TVnap. . . , Ra\u00b41, Ra`1, . . . q.\nProposition 2. If vm \u010e vn for some m, n P rM s, TV pRq is equivalent to TVzvmpRq.\nProposition 3. If ea \u201c eb for a, b P I, TV pRq is equivalent to TVnap \u02dcRq where \u02dcR \u201c\np. . . , Ra\u00b41, Ra`1, . . . , Rb\u00b41, RaRb, Rb`1, . . . q.\nProposition 4. Assume the convolution is size-invariant, i.e., H \u201c H 1 and W \u201c W 1. Then, given\n\ufb01lter height and width I, J P t1, 3, 5, . . . u, the number of possible combinations that eventually\nachieve I \u02c6 J convolution is \u03c0p I\u00b41\n2 q, where \u03c0 : N \u00d1 N is the partition function of integers.\n(See [Sloane, 2019] for examples.)\n\n2 q\u03c0p J\u00b41\n\nProposition 1 says that, if the inner dimension of an inner index is one, we can eliminate it from\nthe hypergraph. Proposition 2 shows that, if the indices of a vertex form a subset of the indices of\nanother vertex (e.g. v1 \u201c ta, cu and v2 \u201c ta, b, cu), we can remove the \ufb01rst vertex. Proposition 3\nmeans that a \u201cdouble\u201d hyperedge on the dimensions A, B P N is reduced to a single hyperedge on\nthe dimension AB. Proposition 4 tells us the possible choices of \ufb01lter size. We defer the proofs to the\nSupplementary material. By combining the above propositions, we can obtain the following theorem:\n\nTheorem 1. If the number of inner indices and the \ufb01lter size is \ufb01nite, the set of nonredundant\nhypergraphs representing convolution (1) is \ufb01nite.\n\nTo enumerate nonredundant hypergraphs, we \ufb01rst use the condition of Proposition 2. Because of the\nvertex-subset constraint in Proposition 2, a valid vertex set must be a subset of the power set of all\nthe indices O Y I, and its size is at most 22|OYI|\n. After enumerating the vertex sets satisfying this\nconstraint, we eliminate some of them using the other propositions.3 We used this algorithm in the\nexperiments (Section 6).\n\n4 Nonlinear Extension\n\nTensor decomposition involves multiple linear operations, and each vertex can be seen as a linear\nlayer. For example, consider a linear map W : RC \u00d1 RC1\n. If W is written as a product of three\nmatrices W \u201c ABC, we can consider the linear map to be a composition of three linear layers:\nWpxq \u201c pA \u02dd B \u02dd Cqpxq for a vector input x P RC . This might lead one to conclude that, in addition\nto reducing computational complexity, tensor decomposition with many vertices also contributes\nto an increase in representability. However, because the rank of W is determined by the minimum\nrank of either A, B, or C, and the representability of a matrix is solely controlled by its rank, adding\nlinear layers does not improve representability. This problem arises in Einconv layers.\n\nA simple solution is to add nonlinear functions between linear layers. Although this is easy to\nimplement, enumeration is no longer possible, because the equivalence relation becomes non-trivial\nwith the introduction of nonlinearity, causing an in\ufb01nite number of candidates to exist. It is not\npossible to enumerate an in\ufb01nite number of candidates, thus an ef\ufb01cient neural architecture search\nalgorithm ( [Zoph and Le, 2016]) is needed. Many such algorithms have been proposed, based on\ngenetic algorithms (GAs) [Real et al., 2018], reinforcement learning [Zoph and Le, 2016], and other\nmethods [Zoph et al., 2018, Pham et al., 2018]. In this study, we employ GA because hypergraphs have\na discrete structure that is highly compatible with it. As multiobjective optimization problems need\nto be solved (e.g., number of parameters vs. prediction accuracy), we use the nondominated sorting\n\n3For more details, see the real code: https://github.com/pfnet-research/einconv/blob/master/\n\nenumerate_graph.py\n\n6\n\n\fgenetic algorithm II (NSGA2) [Deb et al., 2002], which is one of the most popular multiobjective\nGAs. In Section 6.2 we will demonstrate that we can \ufb01nd better Einconv layers by GA than by\nenumeration.\n\n5 Related Work\n\nTensor network notation, a graphical notation for linear tensor operations, was developed by the\nquantum many-body physics community (see tutorial by Bridgeman and Chubb [2017]). Our notation\nis basically a subset of this, except that ours allows hyperedges. Such hyperedges are convenient for\nrepresenting certain convolutions, such as depthwise convolution (see Figure 1b; the inclusion of the\nrightmost vertex indicates depthwise convolution). The reduction of redundant tensor networks was\nrecently studied by Ye and Lim [2018], and we extended the idea to include convolution (Section 3.3).\n\nThere are several studies that combine deep neural networks and tensor networks. Stoudenmire and\nSchwab [2016] studied shallow fully-connected neural networks, where the weight is decomposed\nusing the tensor train decomposition [Oseledets, 2011]. Novikov et al. [2015] took a similar approach\nto deep feed-forward networks, which was later extended to recurrent neural networks [He et al.,\n2017, Yang et al., 2017]. Cohen and Shashua [2016] addressed a CNN architecture that can be\nviewed as a huge tensor decomposition. They interpreted the entire forward process, including the\npooling operation, as a tensor decomposition; this differs from our approach of reformulating a single\nconvolutional layer. Another difference is their focus on a speci\ufb01c decomposition (hierarchical Tucker\ndecomposition [Hackbusch and K\u00fchn, 2009]); we do not impose any restrictions on decomposition\nforms.\n\n6 Experiments\n\nWe examined the performance tradeoffs of Einconv layers in image classi\ufb01cation tasks. We measured\ntime complexity by counting the FLOPs of the entire forwarding path, and space complexity by\ncounting the total number of parameters. All the experiments were conducted on NVIDIA P100 and\nV100 GPUs. The details of the training recipes are described in the Supplementary material.\n\n6.1 Enumeration\n\nFirst, we investigated the basic classes of Einconv for 2D and 3D convolutions. For 2D convolution\nwith a \ufb01lter size of 3 \u02c6 3, we enumerated the 901 nonredundant hypergraphs having at most two\ninner indices, where the inner dimensions were all \ufb01xed to 2. In addition to these, we compared\nbaseline Einconv layers that include nonlinear activations and/or more inner indices. We used the\nFashion-MNIST dataset [Xiao et al., 2017] to train the LeNet-5 network [LeCun et al., 1998]. The\nresult (Figure 3) shows that, in terms of FLOPs, two baselines (standard and CP) achieve Pareto\noptimality, but other nameless Einconv layers \ufb01ll the gap between those two.\n\nSimilarly, for a 3 \u02c6 3 \u02c6 3 \ufb01lter, we enumerated 3D Einconv having at most one inner index, of which\nthere were 492 instances in total. We used the 3D MNIST dataset [de la Iglesia Castro, 2016] with\narchitecture inspired by C3D [Tran et al., 2014]. The results (Figure 4) show that, in contrast to the\n2D case, the baselines dominated the Pareto frontier. This could be because we did not enumerate the\ncase with two inner indices due to its enormous size.4\n\n6.2 GA Search with Non-linear Activation\n\nNext, we evaluated the full potential of Einconv by combining it with a neural architecture search.\nIn contrast to the previous experiments, we used Einconv layers from a larger space, i.e., we\nallowed nonlinear activations (ReLUs), factoring-like multiple convolutions, and changes of the\ninner dimensions. We employed two architectures: LeNet-5 and ResNet-50. We trained LeNet-5\nwith the Fashion-MNIST dataset, and the ResNet-50 with the CIFAR-10 dataset. Note that, for\nResNet-50, a signi\ufb01cant number of Einconv instances could not be trained because the GPU memory\n\n4For 3D convolution, the number of tensor decompositions having two inner indices is more than ten thousand.\nTraining all of them would require 0.1 million CPU/GPU days, which was infeasible with our computational\nresources.\n\n7\n\n\f\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\ny\nc\na\nr\nu\nc\nc\nA\n\n \nt\ns\ne\nT\n\n0.88\n\n0.86\n\n0.84\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf standard\n\nflattend\n\ndep.sep.\n\nbottleneck\n\ncp\n\n104.8\n\n104.9\n\n105\n\n105.1\n\n105.2\n\n105.3\n\n106\n\n107\n\n108\n\nNumber of Parameters\n\nTotal FLOPs\n\nFigure 3: Enumeration of 2D Einconv for LeNet-5 trained with Fashion-MNIST. Black dots indicate\nunnamed tensor decompositions found by the enumeration.\n\n0.8\n\n0.6\n\n0.4\n\n0.2\n\ny\nc\na\nr\nu\nc\nc\nA\n\n \nt\ns\ne\nT\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf standard\n\ndep.sep.\n\ncp\n\n2p1\n\ntt\n\nht\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n106\n\n106.5\n\n107\n\n106.5\n\n107\n\n107.5\n\n108\n\n108.5\n\n109\n\n109.5\n\nNumber of Parameters\n\nTotal FLOPs\n\nFigure 4: Enumeration of 3D Einconv for C3D-like networks trained with 3D MNIST, where 2p1, tt,\nand ht mean (2+1)D convolution [Tran et al., 2018], tensor train decomposition [Oseledets, 2011],\nand hierarchical Tucker decomposition [Hackbusch and K\u00fchn, 2009], respectively.\n\nwas insuf\ufb01cient. For the GA search, we followed the strategy of AmoebaNet [Real et al., 2018]:\nwe did not use crossover operations, and siblings were produced only by mutation. Five mutation\noperations were prepared for changing the number of vertices/hyperedges and two for changing the\norder of contraction.5 We set test accuracy and the number of parameters as multiobjectives to be\noptimized by NSGA2.\n\nThe results of LeNet-5 (Figure 5) show the tradeoff between the multiobjectives. Within the clearly\nde\ufb01ned and relatively smooth Pareto frontier, nameless Einconv layers outperform the baselines.\nThe best accuracy achieved by Einconv was \u201e 0.92, which was better than that of the standard\nconvolution (\u201e 0.91). Although the results of ResNet-50 (Figure 6) show a relatively rugged Pareto\nfrontier, Einconv still achieves better tradeoffs than named baselines other than the standard and CP\nconvolutions.\n\n7 Conclusion and Discussion\n\nHerein, we studied hypergraphical structures in CNNs. We found that a variety of CNN layers may\nbe described hypergraphically, and that there exists an enormous number of variants never previously\nencountered. We found experimentally that the Einconv layers, the proposed generalized CNN layers,\nyielded excellent results.\n\nOne striking observation from the experiments is that certain existing decompositions, such as CP\ndecomposition, consistently achieved good accuracy/complexity tradeoffs. This empirical result is\nsomewhat unexpected; there is no theoretical reason that existing decompositions should outperform\n\n5See https://github.com/pfnet-research/einconv/blob/master/mutation.py for implementa-\n\ntion details.\n\n8\n\n\f0.92\n\n0.88\n\n0.84\n\n0.80\n\ny\nc\na\nr\nu\nc\nc\nA\n\n \nt\ns\ne\nT\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf \u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf \u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf standard\n\nflattend\n\ndep.sep.\n\nbottleneck\n\ncp\n\nfactoring\n\n104.6\n\n104.8\n\n105\n\n105.2\n\n105.4\n\n105.6\n\n106\n\n107\n\n108\n\n109\n\n1010\n\nNumber of Parameters\n\nTotal FLOPs\n\nFigure 5: GA search of 2D Einconv for LeNet-5 trained with Fashion-MNIST. Black dots indicate\nunnamed tensor decompositions found by the GA search.\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\ny\nc\na\nr\nu\nc\nc\nA\n\n \nt\ns\ne\nT\n\n0.90\n\n0.85\n\n0.80\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf standard\n\nflattend\n\ndep.sep.\n\nbottleneck\n\ncp\n\nfactoring\n\n107\n\n107.5\n\n108\n\n108.5\n\n108\n\n108.5\n\n109\n\n109.5\n\n1010\n\nNumber of Parameters\n\nTotal FLOPs\n\nFigure 6: GA search of 2D Einconv for ResNet-50 trained with CIFAR-10.\n\nthe new, unnamed ones. Developing a theory capable of explaining this phenomenon, or at least\nof characterizing the necessary conditions (e.g. symmetricity of decomposition) to achieve good\ntradeoffs would be a promising (but challenging) direction for future work.\n\nOne major limitation at present is the computational cost of searching. For example, the GA search\nfor ResNet-50 in Section 6.2 took 829 CPU/GPU days. This was mainly because of the long training\nperiods (approximately 10 CPU/GPU hours for each training), but also because the GA may not\nhave been leveraging the information on hypergraphs well. Although we incorporated some prior\nknowledge of hypergraphs such as the proximity regarding edge removing and vertex adding through\nmutation operations, simultaneous optimization of hypergraph structures and neural networks using\nsparse methods such as LASSO or Bayesian sparse models may be more promising.\n\nAcknowledgments\n\nWe thank our colleagues, especially Tommi Kerola, Mitsuru Kusumoto, Kazuki Matoya, Shotaro\nSano, Gentaro Watanabe, and Toshihiko Yanase, for helpful discussion, and Takuya Akiba for\nimplementing the prototype of an enumeration algorithm. We also thank Jacob Bridgeman for\nsharing an elegant TikZ style for drawing tensor network diagrams. We \ufb01nally thank the anonymous\n(meta-)reviewers for helpful comments and discussion.\n\nReferences\n\nJ. C. Bridgeman and C. T. Chubb. Hand-waving and interpretive dance: an introductory course on\n\ntensor networks. Journal of Physics A: Mathematical and Theoretical, 50(22):223001, 2017.\n\nF. Chollet. Xception: Deep learning with depthwise separable convolutions, corr abs/1610.02357.\n\nURL http://arxiv. org/abs/1610.02357, 2016.\n\n9\n\n\fN. Cohen and A. Shashua. Convolutional recti\ufb01er networks as generalized tensor decompositions. In\n\nInternational Conference on Machine Learning, pages 955\u2013963, 2016.\n\nD. de la Iglesia Castro. 3d mnist dataset. https://www.kaggle.com/daavoo/3d-mnist, 2016.\n\nK. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm:\n\nNsga-ii. IEEE transactions on evolutionary computation, 6(2):182\u2013197, 2002.\n\nE. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus. Exploiting linear structure within\nconvolutional networks for ef\ufb01cient evaluation. In Advances in neural information processing\nsystems, pages 1269\u20131277, 2014.\n\nI. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016.\n\nW. Hackbusch and S. K\u00fchn. A new scheme for the tensor representation. Journal of Fourier analysis\n\nand applications, 15(5):706\u2013722, 2009.\n\nK. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. eprint. arXiv\n\npreprint arXiv:0706.1234, 2015.\n\nZ. He, S. Gao, L. Xiao, D. Liu, H. He, and D. Barber. Wider and deeper, cheaper and faster:\nTensorized lstms for sequence learning. In Advances in neural information processing systems,\npages 1\u201311, 2017.\n\nF. L. Hitchcock. The expression of a tensor or a polyadic as a sum of products. Journal of Mathematics\n\nand Physics, 6(1-4):164\u2013189, 1927.\n\nA. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam.\nMobilenets: Ef\ufb01cient convolutional neural networks for mobile vision applications. arXiv preprint\narXiv:1704.04861, 2017.\n\nY.-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin. Compression of deep convolutional neural\n\nnetworks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530, 2015.\n\nT. G. Kolda and B. W. Bader. Tensor decompositions and applications. SIAM review, 51(3):455\u2013500,\n\n2009.\n\nO. K\u00f6p\u00fckl\u00fc, N. Kose, A. Gunduz, and G. Rigoll. Resource ef\ufb01cient 3d convolutional neural networks.\n\narXiv preprint arXiv:1904.02422, 2019.\n\nV. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky. Speeding-up convolutional neural\n\nnetworks using \ufb01ne-tuned cp-decomposition. arXiv preprint arXiv:1412.6553, 2014.\n\nY. LeCun, L. Bottou, Y. Bengio, P. Haffner, et al. Gradient-based learning applied to document\n\nrecognition. Proceedings of the IEEE, 86(11):2278\u20132324, 1998.\n\nM. Lin, Q. Chen, and S. Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013.\n\nA. Novikov, D. Podoprikhin, A. Osokin, and D. P. Vetrov. Tensorizing neural networks. In Advances\n\nin neural information processing systems, pages 442\u2013450, 2015.\n\nI. V. Oseledets. Tensor-train decomposition. SIAM Journal on Scienti\ufb01c Computing, 33(5):2295\u20132317,\n\n2011.\n\nR. Penrose. Applications of negative dimensional tensors. Combinatorial mathematics and its\n\napplications, 1:221\u2013244, 1971.\n\nH. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean. Ef\ufb01cient neural architecture search via\n\nparameter sharing. arXiv preprint arXiv:1802.03268, 2018.\n\nE. Real, A. Aggarwal, Y. Huang, and Q. V. Le. Regularized evolution for image classi\ufb01er architecture\n\nsearch. arXiv preprint arXiv:1802.01548, 2018.\n\nR. Rigamonti, A. Sironi, V. Lepetit, and P. Fua. Learning separable \ufb01lters. In Proceedings of the\n\nIEEE conference on computer vision and pattern recognition, pages 2754\u20132761, 2013.\n\n10\n\n\fM. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals\nand linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern\nRecognition, pages 4510\u20134520, 2018.\n\nL. Sifre and S. Mallat. Rigid-motion scattering for image classi\ufb01cation. PhD thesis, Ph. D. thesis, 1:\n\n3, 2014.\n\nN. J. A. Sloane. The On-Line Encyclopedia of Integer Sequences. A000041, 2019.\n\nS. W. Smith et al. The scientist and engineer\u2019s guide to digital signal processing. California Technical\n\nPub. San Diego, 1997.\n\nE. Stoudenmire and D. J. Schwab. Supervised learning with tensor networks. In Advances in Neural\n\nInformation Processing Systems, pages 4799\u20134807, 2016.\n\nC. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and\nA. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on\ncomputer vision and pattern recognition, pages 1\u20139, 2015.\n\nC. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture\nfor computer vision. In Proceedings of the IEEE conference on computer vision and pattern\nrecognition, pages 2818\u20132826, 2016.\n\nC. Tai, T. Xiao, Y. Zhang, X. Wang, et al. Convolutional neural networks with low-rank regularization.\n\narXiv preprint arXiv:1511.06067, 2015.\n\nS. Tokui, R. Okuta, T. Akiba, Y. Niitani, T. Ogawa, S. Saito, S. Suzuki, K. Uenishi, B. Vogel, and\nH. Yamazaki Vincent. Chainer: A deep learning framework for accelerating the research cycle.\nIn Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &\nData Mining, pages 2002\u20132011. ACM, 2019.\n\nD. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. \u201clearning spatiotemporal features with\n\n3d convolutional networks,\u201d. arXiv preprint arXiv:1412.0767, 1177, 2014.\n\nD. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri. A closer look at spatiotemporal\nconvolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision\nand Pattern Recognition, pages 6450\u20136459, 2018.\n\nL. R. Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3):279\u2013311,\n\n1966.\n\nW. Wang, Y. Sun, B. Eriksson, W. Wang, and V. Aggarwal. Wide compression: Tensor ring nets.\nIn Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages\n9329\u20139338, 2018.\n\nM. Wiebe.\n\nNumpy-discussion:\n\neinsum.\n\nhttps://mail.python.org/pipermail/\n\nnumpy-discussion/2011-January/054586.html, 2011.\n\nH. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine\n\nlearning algorithms, 2017.\n\nY. Yang, D. Krompass, and V. Tresp. Tensor-train recurrent neural networks for video classi\ufb01cation.\nIn Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages\n3891\u20133900. JMLR. org, 2017.\n\nK. Ye and L.-H. Lim. Tensor network ranks. arXiv preprint arXiv:1801.02662, 2018.\n\nB. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. arXiv preprint\n\narXiv:1611.01578, 2016.\n\nB. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. Learning transferable architectures for scalable image\nrecognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,\npages 8697\u20138710, 2018.\n\n11\n\n\f", "award": [], "sourceid": 2977, "authors": [{"given_name": "Kohei", "family_name": "Hayashi", "institution": "Preferred Networks"}, {"given_name": "Taiki", "family_name": "Yamaguchi", "institution": "The University of Tokyo"}, {"given_name": "Yohei", "family_name": "Sugawara", "institution": "Preferred Networks, Inc."}, {"given_name": "Shin-ichi", "family_name": "Maeda", "institution": "Preferred Networks"}]}