{"title": "Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 2558, "page_last": 2567, "abstract": "Real-world image recognition is often challenged by the variability of visual styles including object textures, lighting conditions, filter effects, etc. Although these variations have been deemed to be implicitly handled by more training data and deeper networks, recent advances in image style transfer suggest that it is also possible to explicitly manipulate the style information. Extending this idea to general visual recognition problems, we present Batch-Instance Normalization (BIN) to explicitly normalize unnecessary styles from images. Considering certain style features play an essential role in discriminative tasks, BIN learns to selectively normalize only disturbing styles while preserving useful styles. The proposed normalization module is easily incorporated into existing network architectures such as Residual Networks, and surprisingly improves the recognition performance in various scenarios. Furthermore, experiments verify that BIN effectively adapts to completely different tasks like object classification and style transfer, by controlling the trade-off between preserving and removing style variations. BIN can be implemented with only a few lines of code using popular deep learning frameworks.", "full_text": "Batch-Instance Normalization for Adaptively\n\nStyle-Invariant Neural Networks\n\nHyeonseob Nam\n\nLunit Inc.\n\nSeoul, South Korea\nhsnam@lunit.io\n\nHyo-Eun Kim\n\nLunit Inc.\n\nSeoul, South Korea\nhekim@lunit.io\n\nAbstract\n\nReal-world image recognition is often challenged by the variability of visual styles\nincluding object textures, lighting conditions, \ufb01lter effects, etc. Although these vari-\nations have been deemed to be implicitly handled by more training data and deeper\nnetworks, recent advances in image style transfer suggest that it is also possible to\nexplicitly manipulate the style information. Extending this idea to general visual\nrecognition problems, we present Batch-Instance Normalization (BIN) to explicitly\nnormalize unnecessary styles from images. Considering certain style features\nplay an essential role in discriminative tasks, BIN learns to selectively normalize\nonly disturbing styles while preserving useful styles. The proposed normalization\nmodule is easily incorporated into existing network architectures such as Residual\nNetworks, and surprisingly improves the recognition performance in various sce-\nnarios. Furthermore, experiments verify that BIN effectively adapts to completely\ndifferent tasks like object classi\ufb01cation and style transfer, by controlling the trade-\noff between preserving and removing style variations. BIN can be implemented\nwith only a few lines of code using popular deep learning frameworks.1\n\n1\n\nIntroduction\n\nInformation described by an image generally consists of spatial and style information, such as object\nshape and texture, respectively. While the spatial information usually represents the key contents of\nthe image, the style information often involves extraneous details that complicate recognition tasks.\nTypical examples include the discrepancies in object textures caused by different camera settings and\nlight conditions, which disturb object classi\ufb01cation. Despite the quantum leap of deep learning in\ncomputer vision, visual recognition in real-world applications still suffers from these style variations\ninherent in images.\nModerating the variability of image styles has been studied recently to enhance the quality of images\ngenerated from neural networks. Instance Normalization (IN) [22] is a representative approach which\nwas introduced to discard instance-speci\ufb01c contrast information from an image during style transfer.\nInspired by this, Huang et al. [10] provided a rational interpretation that IN performs a form of style\nnormalization, showing simply adjusting the feature statistics\u2014namely the mean and variance\u2014of\na generator network can control the style of the generated image. Due to this nature, IN has been\nwidely adopted as an alternative to Batch Normalization (BN) in style transfer [14, 3] and generative\nadversarial networks (GANs) [13, 27].\nIt is a reasonable assumption that IN would be bene\ufb01cial not only in generative tasks but also in\ndiscriminative tasks for addressing unnecessary style variations. However, directly applying IN to a\nclassi\ufb01cation problem degrades the performance [22], probably because styles often serve as useful\n\n1https://github.com/hyeonseob-nam/Batch-Instance-Normalization\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fdiscriminative features for classi\ufb01cation. Note that IN dilutes the information carried by the global\nstatistics of feature responses while leaving their spatial con\ufb01guration only, which can be undesirable\ndepending on the task at hand and the information encoded by a feature map. For example, even\nthough the global brightness of an image is commonly irrelevant to object classi\ufb01cation, it becomes\none of the key features in weather or time prediction. Similarly, the texture of a cloth may confuse\nclassifying clothing categories (shirts vs. skirts), but it is crucial for classifying fashion attributes\n(spotted vs. striped). In short, normalizing styles in a neural network needs to be investigated with a\ncareful consideration.\nIn this paper we propose Batch-Instance Normalization (BIN) to normalize the styles adaptively to the\ntask and selectively to individual feature maps. It learns to control how much of the style information\nis propagated through each channel of features leveraging a learnable gate parameter. If the style\nassociated with a feature map is irrelevant to or disturbs the task, BIN closes the gate to suppress the\nstyle using IN. If the style carries important information to the task, on the other hand, BIN opens\nthe gate to preserve the style though BN. This simple collaboration of two normalization methods\ncan be directly incorporated into existing BN- or IN-based models, requiring only a few additional\nparameters. Our extensive experiments show that the proposed approach surprisingly outperforms\nBN in general object classi\ufb01cation and multi-domain problems, and it also successfully substitutes\nIN in style transfer networks.\n\n2 Related Work\n\nStyle manipulation. Manipulating image styles or textures has been recently studied along with\nthe \ufb01nding that the feature statistics of a convolutional neural network effectively capture the style of\nan image. Cimpoi et al. [2] exploited the Fisher-Vector representation built on convolutional features,\nwhich encodes their high-order statistics, to perform texture recognition and segmentation. Gatys et\nal. [5] addressed texture synthesis by matching the second-order statistics between the source and\ngenerated images based on the Gram matrices of feature maps, which has been extended to neural\nstyle transfer algorithms [6, 14, 3]. Huang et al. [10] further showed that aligning only the \ufb01rst-order\nstatistics, i.e., the mean and variance, can also ef\ufb01ciently transfer image styles. Note that all the\nabove approaches focused on the tasks directly related to image styles\u2014texture recognition, texture\nsynthesis, style transfer, etc. We expand the idea of manipulating style information to general image\nrecognition problems to resolve impeditive style variations and facilitate training.\n\nNormalization. Batch normalization [12] (BN) has become a standard ingredient in constructing\na deep neural network, which normalizes internal activations using the statistics computed over\nthe examples in a minibatch. Several variants of BN such as batch renormalization [11], weight\nnormalization [19], layer normalization [1], and group normalization [24] have been developed mainly\nto reduce the minibatch dependencies inherent in BN. Instance normalization (IN) [22] exhibits\nanother property of normalizing image styles by adjusting per-instance feature statistics, which is\nfurther investigated by conditional instance normalization [3] and adaptive instance normalization [10].\nThe effectiveness of IN, however, has been restricted to image generation tasks such as style transfer\nand image-to-image translation, because IN may dilute discriminative information that is essential\nfor general visual recognition. To overcome this, our proposed normalization technique combines the\nadvantages of BN and IN by selectively maintaining or normalizing style information, which bene\ufb01ts\na wide range of applications.\n\n3 Batch-Instance Normalization\n\nThe style of an image is commonly interpreted as the information irrelevant to spatial con\ufb01guration,\nwhich is known to be captured by spatial summary statistics of feature responses in a deep convolu-\ntional network [5, 6, 14, 10]. Inspired by the work of Huang et al. [10], we assume the \ufb01rst-order\nstatistics\u2014the mean and variance\u2014of a convolutional feature map encodes a certain style attribute.\nIn other words, the information carried by each feature map can be divided into two components: a\nstyle (the mean and variance of activations) and shape (the spatial con\ufb01guration of activations).\nFrom this point of view, instance normalization (IN) can be considered as normalizing the style of\neach feature map while maintaining the shape only. Although it may help to reduce undesirable style\nvariation, it may also cause severe loss of information if a style itself carries an essential feature for\n\n2\n\n\fthe task (e.g., relevant to class labels). Batch normalization (BN), on the other hand, normalizes\nfeature responses in a minibatch level, which preserves the instance-level style variation unless the\nbatch size is too small. However, it lacks the ability to address the style inconsistency that complicates\nthe problem. Thus, we aim to allow BN and IN to complement each other in order to selectively\npreserve or normalize the style encoded by each feature map.\nLet x \u2208 RN\u00d7C\u00d7H\u00d7W be an input minibatch to a certain layer and xnchw denotes its nchw-th\nelement, where h and w indicate the spatial location, c is the channel index, and n is the index of the\nexample in the minibatch. BN normalizes each channel of features using the mean and variance\ncomputed over the minibatch:\n\n\u02c6x(B)\nnchw =\n\n\u00b5(B)\n\nc =\n\n\u03c32(B)\nc\n\n=\n\nxnchw \u2212 \u00b5(B)\n\nc\n\n(cid:113)\n\n\u03c32(B)\nc\n1\n\n+ \u0001\n\n(cid:88)\n(cid:88)\n\nN\n\n,\n\n(cid:88)\n(cid:88)\n\nH\n\n(cid:88)\n(cid:88)\n\nW\n\n(cid:16)\n\nN\n\nH\n\nW\n\nN HW\n\n1\n\nN HW\n\nxnchw,\n\nxnchw \u2212 \u00b5(B)\n\nc\n\n(cid:17)2\n\n,\n\n(1)\n\nwhere \u02c6x(B) = {\u02c6x(B)\nexample in the minibatch independently using per-instance feature statistics:\n\nnchw} is the batch-normalized response. On the other hand, IN normalizes each\n\n\u02c6x(I)\nnchw =\n\nxnchw \u2212 \u00b5(I)\n\nnc\n\n(cid:113)\n\n\u03c32(I)\nnc + \u0001\n\n(cid:88)\n(cid:88)\n\nH\n\n(cid:88)\n(cid:88)\n\nW\n\n,\n\n(cid:16)\n\nH\n\nW\n\n\u00b5(I)\n\nnc =\n\n\u03c32(I)\nnc =\n\n1\n\nHW\n\n1\n\nHW\n\nxnchw,\n\nxnchw \u2212 \u00b5(I)\n\nnc\n\n(cid:17)2\n\n,\n\n(2)\n\nwhere \u02c6x(I) = {\u02c6x(I)\nnchw} is the instance-normalized response. Note that the style difference between\nexamples is retained in \u02c6x(B) but removed in \u02c6x(I). Our goal is to adaptively balance \u02c6x(B) and \u02c6x(I) for\neach channel so that an important style attribute is preserved while a disturbing one is normalized.\nBatch-Instance Normalization (BIN) achieves this by introducing additional learnable parameters\n\u03c1 \u2208 [0, 1]C:\n\n(cid:16)\n\n\u03c1 \u00b7 \u02c6x(B) + (1 \u2212 \u03c1) \u00b7 \u02c6x(I)(cid:17) \u00b7 \u03b3 + \u03b2,\n\ny =\n\n(3)\n\nwhere \u03b3, \u03b2 \u2208 RC are the af\ufb01ne transformation parameters and y \u2208 RN\u00d7C\u00d7H\u00d7W is the output of\nBIN. We constrain the elements in \u03c1 to be in the range [0, 1] simply by imposing bounds at the\nparameter update step:\n\n\u03c1 \u2190 clip[0,1] (\u03c1 \u2212 \u03b7\u2206\u03c1) ,\n\n(4)\n\nwhere \u03b7 is the learning rate and \u2206\u03c1 indicates the parameter update vector (e.g., the gradient)\ndetermined by the optimizer. Intuitively, \u03c1 can be interpreted as a gate vector which decides whether\nto maintain or discard the style variation for each channel. If the style encoded by a channel is\nimportant to the task, the gate value increases toward 1 and the style is preserved by BN. If a style is\nunnecessary or disturbs the task, on the other hand, the corresponding gate approaches 0 to normalize\nthe style through IN.\nIn practice, we found that it is bene\ufb01cial to increase the learning rate for updating \u03c1. It fairly matches\nthe theoretical aspect that the gradient of loss (cid:96) with respect to \u03c1 tends to be small because it is\n\n3\n\n\fTable 1: Top-1 accuracy (%) on CIFAR-10/100 (test set) and ImageNet (validation set) evaluated with\nResNet-110 and ResNet-18, respectively. The results on CIFAR-10/100 are reported as the average\nand the 95% con\ufb01dence interval over 10 repetitions.\n\nCIFAR-10\nBN 93.72 \u00b1 0.18\nBIN 94.29 \u00b1 0.09\n\nCIFAR-100\n74.26 \u00b1 0.33\n75.88 \u00b1 0.30\n\nImageNet\n\n69.89\n70.68\n\nFigure 1: Training curves on CIFAR-100. We compare top-1 training accuracy (left) and testing\naccuracy (right) of ResNet-110 with different normalization methods. The solid lines and shaded\nareas represent the average and the 95% con\ufb01dence interval over 10 repetitions, respectively.\n\ncalculated by multiplying the difference between \u02c6x(B) and \u02c6x(I):\n\n(cid:88)\n\n(cid:16)\n\n(cid:88)\n\n(cid:88)\n\nN\n\nH\n\nW\n\n\u2202(cid:96)\n\u2202\u03c1c\n\n= \u03b3c\n\nnchw \u2212 \u02c6x(I)\n\u02c6x(B)\n\nnchw\n\n(cid:17)\n\n\u2202(cid:96)\n\n\u2202ynchw\n\n,\n\n(5)\n\nwhere ynchw is the nchw-th element of the output y. Since the difference \u02c6x(B) \u2212 \u02c6x(I) has small\nvalues especially when the style variation in the minibatch is marginal, training \u03c1 is facilitated by\namplifying the difference using a larger learning rate.\n\n4 Experiments\n\nWe demonstrate the effectiveness of BIN across a wide variety of scenarios in three different applica-\ntions: object classi\ufb01cation, multi-domain learning, and image style transfer. Ancillary experimental\nresults on character recognition are also provided in the supplementary material. Throughout all\nexperiments, the style gates (\u03c1) are initialized to 1 and trained with a learning rate multiplied by 10.\nWeight decay is not applied to the gate parameters.\n\n4.1 Object Classi\ufb01cation\n\nWe \ufb01rst evaluate BIN for general object classi\ufb01cation using CIFAR-10/100 [15] and ImageNet [18]\ndatasets. Deep Residual Networks [7]\u2014ResNet-110 (32\u00d732 input) on CIFAR-10/100 and ResNet-18\n(224\u00d7224 input) on ImageNet\u2014are employed as base architectures, which involve a BN layer after\nevery convolutional layer. To evaluate BIN, we simply switch the BN to BIN while keeping all other\nparts unchanged. The networks are trained by SGD with a batch size of 128, an initial learning rate\nof 0.1, a momentum of 0.9, and a weight decay of 0.0001. On CIFAR datasets, we train the networks\nfor 64K iterations, where the learning rate is divided by 10 at 32K and 48K iterations; on ImageNet,\ntraining runs for 90 epochs, where the learning rate is divided by 10 at 30 and 60 epochs.\n\nComparison of normalization methods. Table 1 compares the classi\ufb01cation results, where BIN\noutperforms BN by considerable margins in all three datasets. The networks with BN and BIN use\n\n4\n\n020406080100120140160epochs5060708090100accuracy (%)BINBNBN+ININ020406080100120140160epochs304050607080accuracy (%)BINBNBN+ININ\fFigure 2: Distributions of the style gate parameters (\u03c1) of BIN in object classi\ufb01cation. The lower,\nmiddle, and upper layers correspond to the three super-layers (i.e., layer1, layer2, layer3) of\nResNet-110 [7] con\ufb01gured as conv1-bn1-relu-layer1-layer2-layer3-avgpool-fc.\n\nTable 2: Top-1 accuracy (%) on CIFAR-100 with various network architectures. We follow each\nauthor\u2019s notation for specifying the network con\ufb01guration (e.g., WRN-[depth]-[widening factor]).\n\nAlexNet\n50.62\n51.00\n\nVGG-19\n\n72.29\n72.50\n\nResNet-56\n\n72.92\n75.05\n\nResNet-110\n\n74.26\n75.88\n\nPreResNet-110 WRN-28-10 ResNeXt-29, 8\u00d764d DenseNet-BC (L100, k12)\n\n76.49\n77.84\n\n80.92\n81.48\n\n80.50\n81.57\n\n76.93\n77.80\n\nBN\nBIN\n\nBN\nBIN\n\nalmost the same number of parameters (e.g., 1.73M and 1.74M for ResNet-110), which implies that\nthe performance gain is not merely a consequence of increasing model capacity. The training curves\non CIFAR-100 with varying normalization methods are also illustrated in Figure 1. BIN is compared\nwith not only BN, but also IN and BN+IN\u2014a na\u00efve ensemble of BN and IN (i.e., the \u03c1 in BIN is\n\ufb01xed to 0.5). We observe that replacing BN with IN deteriorates the performance signi\ufb01cantly as\nreported in the literature. BN+IN somewhat reduces the gap between BN and IN, but is still far\nbehind using BN only. On the other hand, BIN achieves higher training and testing accuracy than\nBN, as well as allows the network to train faster. The comparison between BIN and BN+IN veri\ufb01es\nthat the bene\ufb01t of BIN does not just come from an ensemble effect of BN and IN. Instead, it comes\nfrom BIN\u2019s ability to learn to balance BN and IN so as to selectively normalize the styles irrelevant\nto class labels.\n\nAnalysis on gate parameters. To further understand the behavior of BIN, we investigate the\nlearned values of the style gate parameters in BIN. Figure 2 shows the histogram of \u03c1 in the ResNet-\n110 trained on CIFAR-100. It forms a bimodal distribution where most of the parameters are biased\ntoward 0 or 1, which represents that BIN tends to select either IN or BN rather than somewhere in\nbetween. In addition, the gate values close to 1 (using BN) occupy a much greater fraction than the\nvalues close to 0 (using IN). It conforms to the intuition that BN plays a major role in classi\ufb01cation\nwhile IN plays a subsidiary role of reducing unnecessary style variations. Figure 2 also compares the\ngate distributions at different layers of the network. Lower layers spend relatively more parameters to\nIN (close to 0) than upper layers, because the styles\u2014i.e., global summary\u2014of low-level features\n(e.g., brightness, contrast, simple textures) are often irrelevant to the object classes. On the other\nhand, upper layers substantially rely on BN because they encode highly abstracted features that are\nclosely related to object classes, of which the styles are as important as the shapes.\n\n5\n\n0.00.10.20.30.40.50.60.70.80.91.0\u03c1 values0.00.20.40.60.81.0#params (normalized)totallower layersmiddle layersupper layers\fTable 3: Mixed-domain classi\ufb01cation accuracy (%) of ResNet-18 on the Of\ufb01ce-Home dataset averaged\nover 5-fold cross validation. The network is trained on the entire dataset and tested on each domain\nseparately.\n\nArt\nBN 70.04\nBIN 72.23\n\nClipart\n76.93\n77.27\n\nProduct Real-World Avg.\n78.95\n88.47\n89.12\n80.08\n\n80.38\n81.68\n\nTable 4: Domain adaptation accuracy (%) of DANN [4] (based on ResNet-18) on the Of\ufb01ce-Home\ndataset. X\u2192Y indicates X is the source domain and Y is the target domain.\nCl\u2192Pr\n50.90\n52.03\n\nBN\nBIN\n\nAr\u2192Cl Ar\u2192Pr Ar\u2192Rw Cl\u2192Ar\n49.38\n37.23\n37.46\n50.00\nPr\u2192Ar\n41.98\n43.00\n\nCl\u2192Rw\n63.30\n60.09\n60.55\n63.19\nPr\u2192Rw Rw\u2192Ar Rw\u2192Cl Rw\u2192Pr\n65.71\n71.17\n71.85\n65.94\n\n46.17\n46.73\nPr\u2192Cl\n37.34\n37.69\n\nBN\nBIN\n\n58.64\n59.67\n\n44.56\n45.02\n\nScalability to different architectures. We also evaluate the scalability of BIN by applying it to\na wide range of CNN architectures. We borrow a publicly available implementation2 of classi\ufb01ca-\ntion networks and experiment with every model in the repository\u2014AlexNet [16], VGGNet [20]3,\nResNet [7], PreResNet [8], WRN [26], ResNeXt [25], and DenseNet [9]. We construct two variants of\neach architecture using BN or BIN, train them on CIFAR-100 exactly following the hyper-parameters\nprovided in the repository, and compare their testing accuracies. As shown in Table 2, BIN improves\nthe performance across all tested architectures, which veri\ufb01es the scalability of BIN with respect to\nmodel variety. It also suggests that BIN brings a distinct bene\ufb01t that is not simply substituted by\nincreased model capacity or elaborated network con\ufb01guration.\n\n4.2 Multi-Domain Learning\n\nNext we apply BIN to multi-domain tasks to validate the advantage of BIN in handling the apparent\nstyle discrepancy between different domains. We employ the Of\ufb01ce-Home dataset [23] which is\nrecently introduced for evaluating domain adaptation with deep learning. It consists of 4 different\ndomains\u2014Art (Ar), Clipart (Cl), Product (Pr), and Real-World (Rw)\u2014exhibiting substantial\nstyle variations. Each domain contains images from 65 categories shared across domains and the\nentire dataset comprises 15,588 images.\n\nMixed-domain classi\ufb01cation. We \ufb01rst investigate mixed-domain classi\ufb01cation where images from\nheterogeneous domains are mixed in the training data. We construct a training set by combining\nthe 4 domains to train a domain-agnostic classi\ufb01cation network, and evaluate the model on each\ndomain separately. As done in the ImageNet classi\ufb01cation in Section 4.1, we adopt ResNet-18 as\na base architecture, and follow the same training policy except that the the network is pretrained\non ImageNet and the initial learning rate is set to 0.01. We perform 5-fold cross validation and\nreport the average accuracy in Table 3. BIN noticeably improves the performance on all of the 4\ndomains, which demonstrates that BIN effectively alleviates the style disparity between domains. It\nis especially effective in the Art domain which contains the fewest number of images (2,427 images),\nwhere exploiting other domains to learn style-invariant features is more valuable.\n\nDomain adaptation. We additionally explore the domain adaptation problem involving a shift\nbetween training (source) and test (target) domains. We randomly split each domain into training and\n\n2https://github.com/bearpaw/pytorch-classi\ufb01cation\n3Since AlexNet and VGGNet do not contain BN layers, we modify the networks by adding a BN (or BIN)\n\nlayer after every convolutional layer.\n\n6\n\n\fRain Princess\n\nCandy\n\nUdnie\n\nFigure 3: Training curves of style transfer networks with different styles and normalization methods.\nWe compare the content loss (top row) and style loss (bottom row) averaged over 100 iterations..\n\ntest sets containing 80% and 20% of the images, respectively. Then we follow the \"fully-transductive\"\nprotocol, where the training data from the source and the target domains are incorporated with and\nwithout class labels, respectively, which is also called as unsupervised domain adaptation. We employ\nDomain Adversarial Neural Network (DANN) [4] as a base algorithm, which is a recently proposed\ndomain adaptation framework based on deep adversarial learning. It consists of three components\ntrained end-to-end: a feature extractor to learn features shared across domains, a label predictor\nto classify object labels, and a domain classi\ufb01er to make the features invariant to the domain shift.\nWe reimplement the algorithm by constructing the feature extractor based on ResNet-184 where\na 256-dimensional bottleneck is attached after the \ufb01nal average pooling layer. Each of the label\npredictor and domain classi\ufb01er consists of two fully connected layers with a 1024-dimensional hidden\nlayer (256 \u2192 1024 \u2192 65 and 256 \u2192 1024 \u2192 2). We follow the training hyper-parameters provided\nby the author.\nThe test accuracies on the 12 transfer tasks in the Of\ufb01ce-Home dataset are presented in Table 4.\nAlthough not by signi\ufb01cant margins, BIN consistently surpasses BN on 11 out of the 12 tasks. In\nthis scenario BIN identi\ufb01es and normalizes the styles associated with domain labels as well as not\nassociated with class labels, allowing the features to be more invariant to domain changes. In addition,\nnoticing that the target domain is trained without class labels, BIN potentially bene\ufb01ts addressing\nproblems lacking labeled data by learning more generalizable features from another domain.\n\n4.3\n\nImage Style Transfer\n\nFinally we conduct experiments on style transfer where IN is more appropriate than BN, in order to\ncon\ufb01rm that BIN also retains the bene\ufb01t of IN. We adopt a feed forward style transfer algorithm [14]\nwhich consists of an image transformation network and a loss network for calculating the content\nloss and style loss. We train the image transformation network by switching normalization layers,\nusing content images from the MS-COCO dataset [17] following the same training policy as in [14].\n\nComparison of normalization methods. Figure 3 compares the training curves with different\nnormalization methods. As reported in [21, 22, 10], IN brings signi\ufb01cant improvement over BN by\ndiscarding style information from content images. A na\u00efve ensemble of BN and IN (BN+IN) produces\na comparable content loss with IN but degenerates the style loss with a considerable margin because\ndisturbing styles of content images are still remained. In comparison, BIN achieves slightly lower\ncontent loss than IN by preserving important styles\u2014that might be relevant to the contents\u2014from\n\n4DANN originally uses AlexNet, but we replace it by ResNet-18 for natural incorporation of normalization\n\nlayers and comparability with above experiments.\n\n7\n\n050000100000150000200000iterations3.64.04.44.85.2content lossBINBNBN+ININ050000100000150000200000iterations3.003.253.503.754.00content lossBINBNBN+ININ050000100000150000200000iterations1.601.751.902.052.20content lossBINBNBN+ININ050000100000150000200000iterations1.401.651.902.152.40style loss (\u00d710\u22125)BINBNBN+ININ050000100000150000200000iterations1.251.401.551.701.85style loss (\u00d710\u22125)BINBNBN+ININ050000100000150000200000iterations1.001.151.301.451.60style loss (\u00d710\u22125)BINBNBN+ININ\fFigure 4: Distributions of the style gate parameters (\u03c1) of BIN in image style transfer (Rain\nPrincess). The lower, middle, and upper layers correspond to the \ufb01rst, third, and last residual\nblocks, respectively, of the image transformation network [14] containing \ufb01ve residual blocks.\n\nStyle\n\nContent\n\nBN\n\nIN\n\nBN+IN\n\nBIN\n\nFigure 5: Example results of style transfer with different normalization methods. BIN mostly\nproduces similar results to IN or BN+IN (top), but sometimes generates improved results than the\nothers by selectively preserving the style of the content image (bottom).\n\ncontent images. Although it produces marginally higher style loss than IN due to the remaining styles\nof content images, the gap is clearly mitigated compared with BN+IN because it selectively discards\nimpeditive styles only.\n\nAnalysis on gate parameters. As done in Section 4.1, we perform analysis on the gate parameters\nof BIN. Figure 4 represents the histogram of \u03c1 in the trained image transformation network, which\nforms a bimodal distribution as in Figure 2. However, the fraction of gates close to 0 is much higher\n(even though they are initialized to 1) compared to Figure 2. This not only ensures that IN is more\nsuited to a style transfer task than BN, but also demonstrates that BIN exhibits completely different\nbehavior depending on the task. Furthermore, the distributions at different layers show almost no\ndifference unlike Figure 2, because the depth of the transformation network (image\u2192image) is less\nrelated with the level of abstraction than the depth of the classi\ufb01cation network (image\u2192class label).\n\nQualitative comparison. Some qualitative examples of style transfer are illustrated in Figure 5.\nWhile BN leads to inferior stylization quality as reported in [21, 22], the other options (IN, BN+IN,\nand BIN) produce better results with similar quality in most cases (\ufb01rst row). The distinct behavior\nof BIN is observed when the content image contains a meaningful style (second row). BIN tends to\npreserve the style of a content image to some degree as well as successfully transfers the target style,\nwhile IN and BN+IN lose the original style almost completely. More results are also provided in the\nsupplementary material.\n\n8\n\n0.00.10.20.30.40.50.60.70.80.91.0\u03c1 values0.00.20.40.60.81.0#params (normalized)totallower layersmiddle layersupper layers\f5 Conclusion\n\nIn this paper we propose Batch-Instance Normalization (BIN) to adaptively learn to normalize style\ninformation from images. It binds BN and IN leveraging the gate parameters to determine the\nproper normalization method based on the importance of style features. We demonstrate that simply\nreplacing BN with BIN considerably improves the performance of BN-based models in various\nimage recognition tasks. BIN is also successfully applied to image stylization as an alternative to IN,\nsuggesting the \ufb02exibility of BIN for different purposes.\nEver since the breakthrough achieved by batch normalization, a number of studies have suggested\nimproved normalization techniques (batch renormalization [12], weight normalization [19], layer\nnormalization [1], group normalization [24], etc.). They mostly focus on addressing the minibatch\ndependencies of BN in constrained situations, but still none of them is as generalizable as BN to a wide\nrange of applications. We hope our work sheds light on another aspect of improving normalization,\nidentifying and removing unnecessary variations from data, which might be the fundamental role of\nnormalization.\n\nReferences\n[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint\n\narXiv:1607.06450, 2016.\n\n[2] Mircea Cimpoi, Subhransu Maji, and Andrea Vedaldi. Deep \ufb01lter banks for texture recognition and\n\nsegmentation. In CVPR, 2015.\n\n[3] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In\n\nICLR, 2017.\n\n[4] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Fran\u00e7ois Laviolette,\nMario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of\nMachine Learning Research, 17(1):2096\u20132030, 2016.\n\n[5] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Texture synthesis using convolutional neural\n\nnetworks. In NIPS, 2015.\n\n[6] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural\n\nnetworks. In CVPR, 2016.\n\n[7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.\n\nIn CVPR, 2016.\n\n[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks.\n\nIn ECCV, 2016.\n\n[9] Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolu-\n\ntional networks. In CVPR, 2017.\n\n[10] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization.\n\nIn ICCV, 2017.\n\n[11] Sergey Ioffe. Batch renormalization: Towards reducing minibatch dependence in batch-normalized models.\n\nIn NIPS, 2017.\n\n[12] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing\n\ninternal covariate shift. In ICML, 2015.\n\n[13] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional\n\nadversarial networks. In CVPR, 2017.\n\n[14] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and\n\nsuper-resolution. In ECCV, 2016.\n\n[15] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical\n\nreport, 2009.\n\n[16] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classi\ufb01cation with deep convolutional\n\nneural networks. In NIPS, 2012.\n\n9\n\n\f[17] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll\u00e1r,\n\nand C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.\n\n[18] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang,\nAndrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large\nScale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211\u2013252, 2015.\n\n[19] Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate\n\ntraining of deep neural networks. In NIPS, 2016.\n\n[20] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni-\n\ntion. In ICLR, 2015.\n\n[21] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Instance normalization: The missing ingredient\n\nfor fast stylization. arXiv preprint arXiv:1607.08022, 2016.\n\n[22] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Improved texture networks: Maximizing quality\n\nand diversity in feed-forward stylization and texture synthesis. In CVPR, 2017.\n\n[23] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing\n\nnetwork for unsupervised domain adaptation. In CVPR, 2017.\n\n[24] Yuxin Wu and Kaiming He. Group normalization. arXiv preprint arXiv:1803.08494, 2018.\n\n[25] Saining Xie, Ross Girshick, Piotr Doll\u00e1r, Zhuowen Tu, and Kaiming He. Aggregated residual transforma-\n\ntions for deep neural networks. In CVPR, 2017.\n\n[26] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.\n\n[27] J. Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent\n\nadversarial networks. In ICCV, 2017.\n\n10\n\n\f", "award": [], "sourceid": 1283, "authors": [{"given_name": "Hyeonseob", "family_name": "Nam", "institution": "Lunit Inc."}, {"given_name": "Hyo-Eun", "family_name": "Kim", "institution": "Lunit Inc"}]}