{"title": "Image-to-image translation for cross-domain disentanglement", "book": "Advances in Neural Information Processing Systems", "page_first": 1287, "page_last": 1298, "abstract": "Deep image translation methods have recently shown excellent results, outputting high-quality images covering multiple modes of the data distribution. There has also been increased interest in disentangling the internal representations learned by deep methods to further improve their performance and achieve a finer control. In this paper, we bridge these two objectives and introduce the concept of cross-domain disentanglement. We aim to separate the internal representation into three parts. The shared part contains information for both domains. The exclusive parts, on the other hand, contain only factors of variation that are particular to each domain. We achieve this through bidirectional image translation based on Generative Adversarial Networks and cross-domain autoencoders, a novel network component. Our model offers multiple advantages. We can output diverse samples covering multiple modes of the distributions of both domains, perform domain- specific image transfer and interpolation, and cross-domain retrieval without the need of labeled data, only paired images. We compare our model to the state-of-the-art in multi-modal image translation and achieve better results for translation on challenging datasets as well as for cross-domain retrieval on realistic datasets.", "full_text": "Image-to-image translation for cross-domain\n\ndisentanglement\n\nAbel Gonzalez-Garcia\nComputer Vision Center\nagonzalez@cvc.uab.es\n\nJoost van de Weijer\n\nComputer Vision Center\n\nUniversitat Aut\u00f2noma de Barcelona\n\nYoshua Bengio\n\nMILA\n\nUniversit\u00e9 de Montr\u00e9al\n\nAbstract\n\nDeep image translation methods have recently shown excellent results, outputting\nhigh-quality images covering multiple modes of the data distribution. There has\nalso been increased interest in disentangling the internal representations learned\nby deep methods to further improve their performance and achieve a \ufb01ner control.\nIn this paper, we bridge these two objectives and introduce the concept of cross-\ndomain disentanglement. We aim to separate the internal representation into three\nparts. The shared part contains information for both domains. The exclusive\nparts, on the other hand, contain only factors of variation that are particular to\neach domain. We achieve this through bidirectional image translation based on\nGenerative Adversarial Networks and cross-domain autoencoders, a novel network\ncomponent. Our model offers multiple advantages. We can output diverse samples\ncovering multiple modes of the distributions of both domains, perform domain-\nspeci\ufb01c image transfer and interpolation, and cross-domain retrieval without the\nneed of labeled data, only paired images. We compare our model to the state-of-\nthe-art in multi-modal image translation and achieve better results for translation\non challenging datasets as well as for cross-domain retrieval on realistic datasets.\n\nIntroduction\n\n1\nDeep learning has greatly improved the quality of image-to-image translation methods. These\nmethods aim to learn a mapping that transforms images from one domain to another. Examples\ninclude colorization, where the aim is to map a grayscale image to a plausible colored image of\nthe same scene [21, 47], and semantic segmentation, where an RGB image is translated to a map\nindicating the semantic class of each pixel in the RGB image [13, 31]. Isola et al. [23] proposed\na general purpose image-to-image translation method. Their method is successfully applied to a\nwide range of problems when paired data is available. The theory is further extended to unpaired\ndata by introducing a cycle consistency loss [49]. The U-Net [40] architecture is commonly used for\nimage-to-image translation. This network can be interpreted as an encoder-decoder network. The\nencoder extracts the relevant information from the input domain and passes it on to the decoder,\nwhich then transforms this information to the output domain. In spite of the current popularity of\nthese models, the learned representation (output of the encoder) has only been studied for some\nrelated tasks [32, 44]. Here we investigate and impose structure to the speci\ufb01c representation learned\nin image-to-image translation models.\nDisentangling the accidental scene events, such as illumination, shadows, viewpoint and object\norientation from the intrinsic scene properties has been a long desired goal of computer vision [6, 42].\nWhen applied to deep learning, this allows deep models to be aware of isolated factors of variation\naffecting the represented entities [7, 34]. Therefore, models can marginalize information along a\nparticular factor of variation, should it be not relevant for the task at hand. Such a process can be\nespecially bene\ufb01cial for tasks that are hindered by the presence of particular factors, for example,\nvarying illumination conditions in object recognition. Moreover, disentangled representations grant a\nmore precise control for those tasks that perform actions based on the representation.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fFigure 1: (Left) Example of a pair of domains containing images with colored digits on black background\nor white digits on colored background. (Right) Disentangled representation, separated into shared part across\ndomains (digit) and domain-exclusive parts (color in the background or on the digit).\n\nIn this paper, we combine the disentanglement objective with image-to-image translation, and\nintroduce the concept of cross-domain disentanglement. The aim is to disentangle the domain speci\ufb01c\nfactors from the factors that are shared across the domains. To do so, we partition the representation\ninto three parts; the shared part containing information that is common to both domains, and two\nexclusive parts, which only represent those factors of variation that are particular to each domain\n(see example in \ufb01gure 1). Cross-domain disentanglement for image-to-image translation has several\nadvantages that allow for applications that would otherwise not be feasible: (i) Sample diversity: we\ncan generate a distribution of images conditioned on the input image, whereas most image-to-image\narchitectures can only generate deterministic results [23, 49]. Our approach is similar to the recent\nwork of Zhu et al. [50], although we explicitly model variations in both domains, whereas they only\nconsider variations in the output domain; (ii) Cross domain retrieval: we can retrieve similar images\nin both domains based on the part of the representation that is shared between the domains, and,\nunlike [4], we do not require labeled data to learn the shared representation; (iii) Domain-speci\ufb01c\nimage transfer: domain-speci\ufb01c features can be transferred between images; and (iv) Domain-speci\ufb01c\ninterpolation: we can interpolate between two images with respect to domain-speci\ufb01c features.\nOur model is based on bidirectional image translation across domains, using a pair of Generative\nAdversarial Networks (GANs) [17]. We enforce a disentangled structure in the learned representation\nthrough an adequate combination of multiple losses and a new network component called cross-\ndomain autoencoder. We demonstrate the disentanglement properties of our method on variations on\nthe MNIST dataset [26], and apply it to bidirectional multi-modal image translation in more complex\ndatasets [3, 38], achieving better results than state-of-the-art methods [23, 50] due to the \ufb01ner control\nand generality granted by our disentangled representation. Additionally, we outperform [50] in\ncross-domain retrieval on realistic datasets [23, 45]. Our code and models are publicly available at\nhttps://github.com/agonzgarc/cross-domain-disen.\n\n2 Cross-domain disentanglement networks\nThe goal of our method is to learn deep structured representations that are clearly separated in three\nparts. Let X ,Y be two image domains (e.g. \ufb01g. 1) and let R be an image representation in either\ndomain. We split R into sub-representations depending on whether the information contained in that\npart belongs exclusively to domain X (EX ), domain Y (EY), or it is shared between both domains\n(SX /SY). Figure 1 depicts an example of this representation for images of digits with colors in\ndifferent areas (digit or background). In this case, the shared part of the representation is the actual\ndigit without color information, i.e. \u201cthe image contains a 5\u201d. The exclusive parts are the color\ninformation in the different parts of the image, e.g. \u201cthe digit is yellow\u201d or \u201cthe background is purple\u201d.\nFigure 2 presents an overview of our model, which can be separated into image translation modules\n(left) and cross-domain autoencoders (right). The translation modules G and F translate images from\ndomain X to domain Y, and from Y to X , respectively. They follow an encoder-decoder architecture.\nEncoders Ge and Fe process the input image through a series of convolutional layers and output a\nlatent representation R. Traditionally in these architectures (e.g. [23, 49, 50]), the decoder takes the\nfull representation R and generates an image in the corresponding output domain. In our model,\nhowever, the latent representation is split into shared and exclusive parts, i.e. R = (S, E), and only\nthe shared part of the representation is used for translation. Decoders Gd and Fd combine S with\nrandom noise z that accounts for the missing exclusive part, which is unknown for the other domain\nat test time. This enables the generation of multiple plausible translations given an input image. The\nother component of the model, the cross-domain autoencoders, is a new type of module that helps\naligning the latent distributions and enforce representation disentanglement. The following sections\ndescribe all the components of the model and detail how we achieve the necessary constraints on the\nlearned representation. For simplicity, we focus on input domain X , the model for Y is analogous.\n\n2\n\n\fFigure 2: Overview of our model. (Left) Image translation blocks, G and F , based on an encoder-decoder\narchitecture. We enforce representation disentanglement through the combination of several losses and a GRL.\n(Right) Cross-domain autoencoders help aligning the latent space and impose further disentanglement constraints.\n\nImage translation modules\n\n2.1\nGenerative Adversarial Networks (GAN) are a popular framework [17] consisting of two networks that\ncompete against each other. The generator tries to synthesize realistic images to fool a discriminator,\nwhose task is to detect whether images come from the generator or from the real data distribution.\nWhen the generated images are conditioned using an input image, the task becomes image translation.\nOur image translation modules are inspired by the successful architecture used in pix2pix [23], based\non convolutional GANs [36]. The generator encoder consists of several convolutional layers of stride\n2, followed by batch normalization [22] and leaky ReLU activations. The decoder uses fractionally\nstrided convolutions to upsample the internal representation back to the image resolution. We adapt\nthis architecture for the disentanglement problem with the following modi\ufb01cations.\nExclusive representation. The exclusive representation EX of an image x \u2208 X must not contain\ninformation about domain Y. Therefore, it should not be possible to use only EX to generate an\nimage in Y. To enforce this desirable behavior, we try to generate Y images from EX but also actively\nguide the feature learning to prevent this from happening. For this, we propose a novel application of\nthe Gradient Reversal Layer (GRL), originally introduced in [16] to learn domain-agnostic features.\nDuring the forward pass of the network, this layer acts as the identity function. On the backward\npass, however, the GRL reverses the gradients \ufb02owing back from the corresponding branch. Inspired\nby this idea, our model includes a small decoder in each image translation module, GX\nd in the G\ncase, that tries to generate images in Y with EX as input. We add a GRL at the beginning of GX\nd ,\nimmediately after EX (orange dashed line in \ufb01g. 2). The GRL inverts the sign of the gradient that\nis backpropagated to the encoder Ge, affecting only those units involved in the generation of the\nexclusive features EX . We train GX\nd with an adversarial loss on the generated images. In theory, this\napproach will force EX not to contain information that might generate images in the Y domain.\nShared representation. The shared parts of the representations SX and SY of a pair of correspond-\ning images (x, y) \u2208 (X ,Y) should contain similar information and be invariant to the domain. Some\ndomain adaptation approaches [16, 8, 9] have successfully used a GRL to create domain-invariant\nfeatures. However, we have found here that this approach can quickly become unstable when the\nloss starts diverging. A possible solution for this consists in bounding the loss by the performance\nof random chance [14]. In our case, and due to the fact that our images are paired, we can attain\nthe desired invariance simply by adding an L1 loss on these features, which forces them to be\nindistinguishable for both domains:\n\nLS = Ex\u223cX ,y\u223cY(cid:2)(cid:13)(cid:13)SX \u2212 SY(cid:13)(cid:13)(cid:3).\n\nAdding noise in the representation. A drawback of loss (1) on the shared representation is that it\nthe similarity between SX and SY. We have found out that adding small noise (N (0, 0.1)) to the\noutput of the encoder as in [46] prevents this from happening and leads to better results1.\n\nencourages the model to use a small signal(cid:13)(cid:13)SX(cid:13)(cid:13) \u2192 0, which reduces the loss but does not increase\n1We also investigated constraining(cid:13)(cid:13)SX(cid:13)(cid:13) =(cid:13)(cid:13)SY(cid:13)(cid:13) = 1 but found this to be less stable.\n\n(1)\n\n3\n\n\fArchitectural bottleneck. Most image translation approaches [5, 23, 33, 49, 50] are devised for\npairs of domains that retain the spatial structure (e.g. grayscale to color images), and thus a great\namount of information is shared between input and output. In the usual encoder-decoder architecture,\nall this information passes through a bottleneck, here called the latent representation, that connects\nthe two components. To prevent the loss of details at higher resolutions, it is common to use skip\nconnections (e.g. U-Net [5, 23, 33, 40, 49, 50]). When disentangling the latent representation,\nhowever, skip connections pose a problem. The higher resolution features operated by the encoder\ncontain both shared and exclusive information, but the decoder must receive only the shared part of\nthe representation from the encoder. Therefore, instead of using skip connections, we reduce the\narchitectural bottleneck by increasing the size of the latent representation. In fact, we only increase\nthe spatial dimensions of the shared part of the representation, from 1 \u00d7 1 \u00d7 512 to 8 \u00d7 8 \u00d7 512. We\nfound out that in the considered domains, the exclusive part can be successfully modeled by a 1\u00d71\u00d78\nvector, which is later tiled and concatenated with the shared part before decoding. We implement the\ndifferent size of the latent representation by parallel last layers in the encoder, convolutional for the\nshared part and fully connected for the exclusive part.\nReconstructing the latent space. The input of the translation decoders is the shared representation\nS and random input noise that takes the role of the exclusive part of the representation. Concretely,\nwe use an 8-dimensional noise vector z sampled from N (0, I). The exclusive representation must be\napproximately distributed like the input noise, as both take the same place in the input of the decoder\n(see sec. 2.2). To achieve this, we add a discriminator Dz that tries to distinguish between the output\nexclusive representation EX and input noise z, and train it with the original GAN loss [17]. This\npushes the distribution of EX towards N (0, I) and makes the input of the decoder consistent.\nCommonly, adversarial image translation approaches [23, 49, 50] attempt to achieve some stochastic-\nity by adding random noise to the input or the internal features. However, in many cases this noise is\nignored and the generated outputs are uni-modal [23, 49]. We follow an idea explored in [11, 12, 50]\nto avoid this and reconstruct the latent representation from the generated image by feeding it to\nthe encoder. The reconstructed representation should match the decoder input, so we add an L1\nloss between the original and reconstructed SX , as well as the input noise z and the reconstructed\nexclusive part\n\nrecon = Ex\u223cX(cid:2)||Ge(Gd(SX , z)) \u2212 (SX , z)||(cid:3).\n\nLX\n\n(2)\n\nWGAN-GP loss. Since the original formulation [17], more advanced GAN losses have appeared.\nFor example, the use of the Wasserstein-1 distance in WGAN [2] has been shown to provide desirable\nconvergence properties and to correlate well with perceptual quality of the generated images. We\nadopt the more stable Gradient Penalty variant [18] (WGAN-GP) for our model. Let D be a\nconvolutional discriminator with a single scalar as output. Following [23], we condition D on the\ncorresponding paired image in the input domain, which is concatenated to the real or generated image\n(omitted from following notation). Our discriminator and generator losses are then de\ufb01ned as\n\n\u02c6x\u223c(cid:98)X [(||\u2207\u02c6xD(\u02c6x)||2 \u2212 1)2],\n\n(3)\n\nLX\nDisc = E\n\n\u02dcx\u223c(cid:101)X [D(\u02dcx)] \u2212 Ex\u223cX [D(x)] + \u03bb \u00b7 E\n\u02dcx\u223c(cid:101)X [D(\u02dcx)],\n\nLX\nGen = \u2212E\n\n(4)\n\nwhere (cid:98)X is the distribution obtained by randomly interpolating between real images x and generated\n\nimages \u02dcx [18]. Contrarily to [23], we do not include a reconstruction term ||\u02dcx \u2212 x|| in the generator\nloss. Our outputs should cover multiple modes of the output distribution and thus they do not\nnecessarily match the paired image in the other domain. We combine both GAN losses in LX\n2.2 Cross-domain autoencoders\nThe image translation modules impose three main constraints: (1) the shared part of the representation\nmust be identical for both domains, (2) the exclusive part only has information about its own domain,\nand (3) the generated output must belong to the other domain. However, there is no force that\naligns the generated output with the corresponding input image to show the same concept (e.g. same\nnumber) but in different domains. In fact, the generated images need not correspond to the input if the\nencoders learn to map different concepts to the same shared latent representation. In order to achieve\nconsistency across domains, we introduce the idea of cross-domain autoencoders (\ufb01g. 2, right).\nA classic autoencoder would take the full representation encoded for input image x, Ge(x) =\n(SX , EX ) and input it in the decoder of module F , which outputs images in X , with the goal of\nreconstructing x. Since the shared representations in our model must be indistinguishable, we could\n\nGAN.\n\n4\n\n\fuse the shared representation SY from the other domain instead of SX . This provides an extra\nincentive for the encoder to place useful information about domain X in EX , as SY does not contain\nany domain-exclusive information. Our cross-domain autoencoders use this combination to generate\nthe reconstructed input x(cid:48) = Fd(SY , EX ). We train them with the standard L1 reconstruction loss\n(5)\n\nLX\n2.3 Bi-directional image translation\nGiven the multi-modal nature of our system in both domains, our architecture is uni\ufb01ed to perform\nimage translation in the two directions simultaneously. This is paramount to learn how to disentangle\nwhat part of the representation can be shared across domains and what parts are exclusive to each.\nWe train our model jointly in an end-to-end manner, minimizing the following total loss\n\nauto = Ex\u223cX(cid:2)||x(cid:48) \u2212 x||(cid:3).\n\nd\n\n(6)\n\nL =wGAN(LX\n\nGAN + LY\n+ wL1(LS + LX\n\nGAN) + wEx(LGX\nauto + LY\nauto + LX\n\nGAN + LF Y\nGAN)\nrecon + LY\nrecon).\n\nd\n\n3 Related work\nDisentangling deep representations. A desirable property of learned representations is the ability\nto disentangle the factors of variation [7]. For this reason, there has been a substantial interest on\nlearning disentangled representations [19, 43], including some work based on generative models [25,\n34, 37]. One of the earliest architectures for learning disentangled representations using deep learning\nwas applied to the task of emotion recognition [39]. The work of [34] combines a Variational\nAutoencoder (VAE) with a GAN to disentangle representations depending on what is speci\ufb01ed (i.e.\nlabeled in the dataset) and the remaining factors of variation, which are unspeci\ufb01ed. In a similar\nintra-domain spirit, InfoGAN [10] optimizes a lower bound on the mutual information between the\nrepresentation and the images, successfully controlling some factors of variation in the considered\nimages. Reed et al. [37] propose learning each factor of variation of the image manifold as its\nown sub-manifold using a higher-order Boltzmann machine. Alternatively, the analogy-making\napproach of [38] attempts to disentangle the factors of variation by using representation arithmetic.\nFinally, some domain adaptation approaches [8, 9, 16, 30] aim at obtaining invariant features for\nthe classi\ufb01cation task, granting some level of disentanglement but depending on class labels. Even\nthough representation disentangling has been widely studied, we are unaware of any work studying\ntrue cross-domain representation disentangling, which is the focus of this paper.\nImage translation. Lately, image generation using adversarial training methods has attracted a\ngreat amount of attention [2, 17]. We consider the image translation task, in which the generative\nprocess is conditioned on an input image [23, 47, 50]. While some approaches had previously applied\nadversarial losses for speci\ufb01c image translation tasks such as style transfer [28] or colorization [47],\nthe approach of Isola et al. [23], called pix2pix, was the \ufb01rst GAN-based image translation approach\nthat was not tailored to a speci\ufb01c application. Despite the excellent results of these models, they are\nlimited by the lack of variation of their generated outputs, which are virtually deterministic, as the\ninput noise is mostly ignored. As a consequence, they can only provide a one-to-one mapping across\nimage domains, a phenomenon named mode collapse [41].\nIn order to reduce this limitation, Zhu et al. [50] extended the pix2pix framework. They minimize the\nreconstruction error of the latent code by a reverse decoder with the generated output as input, forcing\nthe generator to take the input noise into account. Furthermore, they combine a conditional GAN\nwith a conditional VAE, whose goal is to provide a plausible latent vector given a target image. The\nresulting model, called BicyleGAN, effectively achieves one-to-many image translations. However,\nthere are several differences with our method. Theirs is restricted in only one direction, whereas our\nmethod operates in a many-to-many setting. Moreover, our representation grants a \ufb01ner control on\nthe stochastic factors of the generated images as we also model variations on the inputs, allowing\nus to keep selected properties \ufb01xed. Finally, the obtained disentangled image features are useful for\nadditional tasks beyond image translation, such as cross-domain retrieval or visual analogies.\nConcurrently to our work, several approaches [1, 20, 27, 29] have attempted to improve on image-to-\nimage translation by disentangling the internal representation into content and style, which enable the\neffective generation of multi-modal outputs. In a similar spirit, Ma et al. [32] disentangle the features\nlearned for person image generation into foreground, background, and pose. In our case, however,\nimage translation is not the only \ufb01nal task: we demonstrate the generality of our distentangled features\nby applying them to other tasks such as cross-domain retrieval or domain-speci\ufb01c image transfer.\n\n5\n\n\fFigure 3: (a) Samples generated by our model using random noise as exclusive representation, where X =\nMNIST-CD and Y = MNIST-CB. (b) Visual analogies, combining shared and exclusive parts of different samples.\n4 Experiments\n4.1 Representation disentangling on MNIST variations\nWe evaluate the properties of our representation by following the protocol introduced by [34]\nto measure the disentanglement of a representation. However, [34] operates within one domain\nonly, whereas we learn cross-domain representations. For this reason, we extend MNIST [26], the\nhandwritten-digit dataset used in [34], with variations that correspond to two different domains. In our\nvariations, we either colorize the digit (MNIST-CD) or the background (MNIST-CB) with a randomly\nchosen color (\ufb01g.1). We use the standard splits for train (50K images) and test (10K images). We\ndetail the architectures and hyperparameters used for all experiments in the supplementary material.\nSample diversity. Figure 3a shows samples generated by our model in both domains using random\ninput noise. We can observe how our model successfully generates diverse samples for different noise\nvalues, varying the color where appropriate but maintaining the digit information. Note, however, that\nthe model has no knowledge of the digit in the image as labels are not provided, it effectively learns\nwhat information is shared across both domains. This demonstrates that we achieve many-to-many\nimage translation through proper manipulation of the disentangled latent representation.\nDomain-speci\ufb01c image transfer. We evaluate our domain-speci\ufb01c transferring capabilities using\nvisual analogy generation, which consists in applying a particular property of a given reference\nimage to a query image [38] (e.g. changing the digit color in one MNIST-CD image to the another\nimage\u2019s color, as in \ufb01g. 3b). Our disentangled representation grants us a precise control over the\nimage generation process, facilitating the visual analogy task as it can be seen as applying domain-\nspeci\ufb01c properties (encoded in the exclusive part of the representation) from one image to another.\nOur model can generate visual analogies as follows. Let us consider two input images from one\ndomain x1, x2 \u2208 X and their disentangled representations R1 = (SX\n2 , EX\n2 ),\nrespectively. We use x1 as input query and x2 as reference. We can generate the desired visual\nanalogy by simply combining the shared part of the query with the exclusive part of the reference\nand running it through the decoder, i.e. Fd(SX\n2 ). Fig. 3b illustrates this process and shows\nqualitative results. The query images acquire the corresponding properties of the reference images,\nas the output images have the correct digit and color. Note how our model has not been explicitly\ntrained to achieve this behavior, it is a natural consequence of a correctly disentangled representation.\nDomain-speci\ufb01c image interpolation. Beyond transferring domain-speci\ufb01c properties between\nimages, our representation allows us to interpolate between two images along domain-speci\ufb01c\nproperties. To do this, we simply keep one part of the representation \ufb01xed while we interpolate\nbetween two samples in the other part. Finally, we combine each interpolated value with the \ufb01xed\npart and run it through the decoder to generate the image. Figure 4a shows results for interpolations\non the exclusive and shared parts for various random samples of both domains. When interpolating\non the exclusive part, we generate samples along domain-speci\ufb01c factors of variation, i.e. color, and\nmaintain the digit, which is the shared information. Analogously, when we interpolate on the shared\npart, the domain-speci\ufb01c properties stay stable while one digit smoothly transforms into the other.\nCross-domain retrieval. Given an image query and an image database, the goal of retrieval is\nselecting those images that are similar to the query, either semantically or visually. In cross-domain\nretrieval [4, 24, 35], query and database are from different domains. Generally, the shared information\nbetween domains is semantic whereas the exclusive is stylistic, and so our disentangled representation\nenables both semantic and visual retrieval using either of its parts. We perform cross-domain retrieval\nusing Euclidean distance between shared features, and compare it with a simple baseline using\ndistances on image pixels. Table 1 (left) presents the results in terms of the common retrieval\nmetric of Recall@1. The high values obtained by shared features show how using our representation\nprovides an effective approach for cross-domain retrieval, clearly superior to directly using image\npixels. Moreover, we do not need image labels, as opposed to specialized approaches such as [4].\n\n1 ), and R2 = (SX\n\n1 , EX\n\n1 , EX\n\n6\n\n\fTable 1: Cross-domain retrieval (Recall@1) on MNIST-CD/CB and Facades [45]/Maps [23].\nMaps\n\nMNIST-CD \u2192 CB MNIST-CB \u2192 CD\n\nFacades\n\nMethod\nPixels\nShared\nExclusive\n\n30.45\n99.99\n9.93\n\n40.02\n99.95\n9.80\n\nMethod\n\nF \u2192 L\n\nBicycleGAN\nOurs\n\n-\n95\n\nL \u2192 F\n45\n97\n\nS \u2192 M M \u2192 S\n68.0\n96.9\n\n91.4\n\n-\n\nFigure 4: (a) Interpolation between samples on either part of the representation. (b) Random image queries and\ntheir 10 nearest-neighbors in the set union of both domains, using distance on pixels or parts of our representation.\n\nTo further demonstrate representation disentanglement we perform retrieval from one domain to a\ndatabase of images from both domains. Figure 4b displays examples of random queries in this setting,\nshowing the 10 nearest-neighbors using distances on image pixels, shared, and exclusive features.\nShared features retrieve images of the same digit from both domains (46%), and thus contain scarce\ndomain-speci\ufb01c information, whereas using pixels clearly prioritizes images from the query domain\n(almost 100%). Moreover, exclusive features retrieve images that are visually similar regardless of\nthe digit, indicating that our exclusive features may be used for purely visual retrieval.\n\n4.2 Many-to-many image translation\nIn this section, we demonstrate the performance of our method for the task of bi-directional multi-\nmodal image translation. We use pix2pix [23] and the state-of-the-art multi-modal approach of\nBicycleGAN [50] as baselines, combining two independently trained models in either direction.\nDespite the truly remarkable results of these approaches, they are limited by the underlying assumption\nof spatial correspondence between images across domains, and thus cannot be applied when domains\nundergo signi\ufb01cant structural changes such as viewpoint, as con\ufb01rmed experimentally. Our method,\non the other hand, removes this assumption as it does not rely on additional side information to\ngenerate its samples, only on the learned latent representation. To provide a fair comparison, we\nremove the skip connections in [23, 50] and increment the latent space to 8 \u00d7 8, as in our architecture.\nThis architectural adjustment increases the model\u2019s ability to translate images with signi\ufb01cant\nstructural changes, such as those used in this section. For completeness, we also provide quantitative\nresults for the original BicycleGAN in Table 2. We measure the performance quantitatively with\nthe Learned Perceptual Image Patch Similarly (LPIPS) metric of [48], which is based on differences\nbetween network features and correlates very well with human judgments. We use the of\ufb01cial\nimplementation and default settings by the authors [48].\n3D car models. We use the 3D car images [38] of the 199 CAD models in [15], rendered from 24\nequally spaced viewpoints. Let X be the frontal/rear car views, and Y the pro\ufb01le views. We set 5\nrandom cars for test and train with the remaining 796 images. Fig. 5 shows generated samples by our\nmethod and the baselines. The deterministic nature of pix2pix con\ufb02ates both views into one, making\nit unable to output realistic cars with a speci\ufb01c viewpoint. BicycleGAN generates better samples, but\nthe quality is still rather poor. Our method generates good quality samples and covers multiple modes\nof the output distribution. Moreover, it maintains shared information across domains (e.g. car color),\nwhereas BicycleGAN\u2019s samples might not correspond to the input image. We attribute this to our\n\ufb01ner control over the latent representation, as we model image variations also in the input domain.\nTable 2 presents quantitative results. For each image in the test set (two views per car model/domain)\nwe generate three samples and compute the LPIPS [48] metric between them and both possible\nground-truths as domains are bi-modal, e.g. left and right pro\ufb01les. Then, we select the minimum\ndistance to either ground-truth and average over samples. In both directions, our model outperforms\nthe two baselines, generating samples that are perceptually more similar to the actual examples.\nFig. 5b shows visual analogies for cars (created as in \ufb01g. 3b). With our exclusive representation, we\ncan apply the orientation of one car to another while maintaining other properties such as style or\ncolor. Finally, we show in \ufb01g. 5c samples of our model when domain X is the frontal view and Y\ncontains all other views. Even for this more challenging case, we manage to output samples covering\n\n7\n\n\fTable 2: LPIPS metric on samples generated for cars [38] and chairs [3].\n\nMethod\n\npix2pix\nBicycleGAN - Skip\nBicycleGAN - No Skip\nOurs\n\nCars\n\nX (cid:1) Y\n\n9.39\u00b12.3\n17.30\u00b112.7\n10.61\u00b12.2\n8.30\u00b12.5\n\nY(cid:1) X\n\n7.27\u00b11.5\n5.69\u00b10.9\n6.63\u00b11.0\n4.66 \u00b1 1.1\n\nChairs\n\nX (cid:1) Y\n\n9.60\u00b13.0\n11.01\u00b14.5\n14.19\u00b1 3.7\n8.28\u00b12.9\n\nY(cid:1) X\n\n8.50 \u00b1 2.4\n11.20\u00b15.1\n14.51\u00b14.0\n9.45\u00b14.5\n\nFigure 5: (a) Generated samples for the 3D car dataset [38] by our method, pix2pix [23], and BicycleGAN [50].\n(b) Car analogies. (c) Samples when Y is all viewpoints except frontal. (d) Our generated 3D chairs [3].\n\nmany modes of the output distribution. Moreover, outputs are uni-modal when necessary (last row),\nas only domain-exclusive information is varied during generation and X only has one viewpoint.\n3D chair models. Similarly to the cars dataset, [3] offers rendered images of 1393 CAD chair\nmodels from different viewpoints. We arrange X and Y as before, and train the model with 5,372\nimages from 1,343 chairs, leaving 50 chairs for test. Figure 5d shows examples of the chairs generated\nby our model. In this case, the samples also effectively cover both modes of the target distribution.\nQuantitatively (table 2), we achieve the best results in one direction but worse than pix2pix for the\nother. This could be due to the fact that viewpoint changes in this dataset are less extreme than with\ncars, as the aspect-ratios of frontal and pro\ufb01le views are quite similar.\n4.3 Ablation study\nWe tried removing some network components and observed the effect on the model (table 3). We\nmeasure performance as the ability to create visual analogies, which guarantees a minimum of\ndisentanglement. We create or select the ground-truth target analogy (e.g. digit of the query with\nthe color of the reference) and compute the distance with the output of our model. We can see\nhow some components such as the cross-domain autoencoders are crucial for this task, since when\nremoved (No auto.) the performance decreases signi\ufb01cantly. The performance drop when using\nnormal autoencoders (Normal auto.) can be even greater. We attribute this to the information shortcut\nthey introduce, which allows the model to ignore the exclusive information and makes the output of\nthe visual analogy the reconstructed input. This effect is less noticeable for cars and chairs, as the\ndomain modes are perceptually closer (e.g. same aspect ratio).\nOur model manages to create visual analogies without some components, but it is negatively affected\nin other tasks (e.g. diverse sample generation) as well as training stability. For example, removing the\nGRL may increase the amount of shared information in the exclusive representation E. We measure\nthis by performing intra-domain retrieval on each MNIST domain using only E. The average increase\nin recall (i.e. more shared information) when removing the GRL is only 0.7% for the current setting\n(E=1x8), but it grows with E\u2019s size: 8% for 1x128 and 25% for 8x8x256. Therefore, the GRL is\nbene\ufb01cial for keeping shared information out of the exclusive representation for particular settings,\nbut its effect is more limited in the current con\ufb01guration and thus it could be safely removed.\n\n8\n\n\fTable 3: Ablation study based on performance of visual analogies, measured as the distance to the ground-truth.\nWe use Euclidean distance (\u00d710\u22122) for MNIST and LPIPS (\u00d7100) for Cars and Chairs.\n\nFull\n\nDataset\nMNIST-CB 13.0 \u00b1 3.2\nMNIST-CD 10.2 \u00b1 2.6\n8.9 \u00b1 2.8\nCars-P\n5.4 \u00b1 1.6\nCars-F/B\n11.4 \u00b1 2.4\nChairs-P\n12.1 \u00b1 2.8\nChairs-F/B\n\nNo auto.\n35.1\u00b113.2\n15.0 \u00b1 4.6\n18.3 \u00b1 2.4\n5.7 \u00b1 1.9\n14.3\u00b12.9\n10.6 \u00b1 2.8\n\nNormal auto.\n40.0 \u00b1 11.5\n16.8 \u00b1 5.04\n9.3 \u00b1 1.4\n5.6 \u00b11.5\n14.3 \u00b1 3.1\n9.9 \u00b1 3.2\n\nNo GRL\n11.4 \u00b1 3.2\n11.5 \u00b1 3.12\n8.7 \u00b1 1.9\n5.6 \u00b1 1.7\n11.3 \u00b1 2.5\n12.9 \u00b1 3.9\n\nNo noise\n13.0 \u00b1 3.7\n11.9 \u00b1 3.0\n8.7\u00b1 2.8\n5.6 \u00b1 1.6\n12.0 \u00b1 1.8\n10.1 \u00b1 3.2\n\nNo LS\n16.6 \u00b1 5.9\n12.4 \u00b1 3.7\n17.5 \u00b1 0.8\n18.0 \u00b1 2.5\n12.0 \u00b1 2.2\n13.9 \u00b1 3.7\n\nNo Lrecon\n18.9\u00b16.4\n12.9 \u00b1 3.1\n10.6 \u00b1 1.9\n6.0 \u00b1 1.9\n15.4 \u00b1 1.5\n10.2 \u00b1 3.4\n\nFigure 6: Retrieval results on Facades [45] and Maps [23] with our method and BicycleGAN [50]. We show the\ntop-3 neighbors using each approach and BicycleGAN\u2019s generated samples. Each last row presents failure cases.\n\n4.4 Cross-domain retrieval on realistic datasets\nWe have so far demonstrated representation disentanglement only on synthetic datasets. We explore\nhere the applicability of our method to other datasets by presenting cross-domain retrieval experiments\non two realistic datasets: maps \u2194 satellite images [23] and labels \u2194 facades [45], and comparing\nwith BicycleGAN [50]. Our method enables cross-domain retrieval through the shared features\n(sec. 4.1). On the other hand, BicycleGAN does not provide an obvious way to tackle the retrieval\ntask, as their internal features (both E\u2019s output and G\u2019s bottleneck [50]) are not disentangled and\nthus contain domain-speci\ufb01c information. One possible retrieval approach with BicycleGAN is:\n(i) translate the input image to the target domain (e.g. 10 random samples) and (ii) retrieve the\nmost similar images to any of the translated samples. Table 1 (right) presents results using the\nprovided train/test splits (100 test images for Facades and 1K for Maps) and the available pre-trained\nBicycleGAN models [50]. We measure performance as the percentage of cases for which the top\nretrieved image is the corresponding paired image. The excellent results of our model demonstrate\nrepresentation disentanglement also on realistic datasets, which is an advantage over BicycleGAN\ngiven its poor results. Figure 6 shows qualitative results. Our method is able to abstract the patterns\nexhibited for both domains (e.g. window size, street directions) in the shared representation, which\nresults in retrieved images that follow those patterns. Retrieval with BicycleGAN, however, generally\nfails at retrieving the corresponding image using the generated samples.\n\n5 Conclusions\nWe have presented the concept of cross-domain disentanglement and proposed a model to solve it.\nOur model effectively disentangles the representation into a part shared across domains and two parts\nexclusive to each domain. We applied this to multiple tasks such as diverse sample generation, cross-\ndomain retrieval, domain-speci\ufb01c image transfer and interpolation. We have tested on several datasets\nof different complexity, both synthetic and of real images. We also introduced the many-to-many\nimage translation setting and paved the way to overcome some limitations of current approaches\nthrough the use of a disentangled representation.\n\n9\n\n\fAcknowledgments We acknowledge the Spanish project TIN2016-79717-R and the CHISTERA\nproject M2CR (PCIN2015-251).\n\nReferences\n[1] A. Almahairi, S. Rajeswar, A. Sordoni, P. Bachman, and A. Courville. Augmented cyclegan:\n\nLearning many-to-many mappings from unpaired data. In ICML, 2018.\n\n[2] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In ICML,\n\n2017.\n\n[3] M. Aubry, D. Maturana, A. A. Efros, B. C. Russell, and J. Sivic. Seeing 3d chairs: exemplar\n\npart-based 2d-3d alignment using a large dataset of cad models. In CVPR, 2014.\n\n[4] Y. Aytar, L. Castrejon, C. Vondrick, H. Pirsiavash, and A. Torralba. Cross-modal scene networks.\n\nIEEE Trans. on PAMI, 2017.\n\n[5] V. Badrinarayanan, A. Kendall, and R. Cipolla. Segnet: A deep convolutional encoder-decoder\n\narchitecture for image segmentation. IEEE Trans. on PAMI, 2017.\n\n[6] H. Barrow and J. Tenenbaum. Recovering intrinsic scene characteristics. Comput. Vis. Syst,\n\n1978.\n\n[7] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives.\n\nIEEE Trans. on PAMI, 2013.\n\n[8] K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan, and D. Erhan. Domain separation\n\nnetworks. In NIPS, 2016.\n\n[9] K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan. Unsupervised pixel-level\n\ndomain adaptation with generative adversarial networks. In CVPR, 2017.\n\n[10] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable\nrepresentation learning by information maximizing generative adversarial nets. In NIPS, 2016.\n\n[11] J. Donahue, P. Kr\u00e4henb\u00fchl, and T. Darrell. Adversarial feature learning. In ICLR, 2017.\n\n[12] V. Dumoulin, I. Belghazi, B. Poole, O. Mastropietro, A. Lamb, M. Arjovsky, and A. Courville.\n\nAdversarially learned inference. In ICLR, 2017.\n\n[13] D. Eigen and R. Fergus. Predicting depth, surface normals and semantic labels with a common\n\nmulti-scale convolutional architecture. In ICCV, 2015.\n\n[14] C. Feutry, P. Piantanida, Y. Bengio, and P. Duhamel. Learning anonymized representations with\n\nadversarial neural networks. arXiv preprint arXiv:1802.09386, 2018.\n\n[15] S. Fidler, S. Dickinson, and R. Urtasun. 3d object detection and viewpoint estimation with a\n\ndeformable 3d cuboid model. In NIPS, 2012.\n\n[16] Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML,\n\n2015.\n\n[17] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and\n\nY. Bengio. Generative adversarial nets. In NIPS, 2014.\n\n[18] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of\n\nwasserstein gans. In NIPS, 2017.\n\n[19] G. E. Hinton, A. Krizhevsky, and S. D. Wang. Transforming auto-encoders. In ICANN, 2011.\n\n[20] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz. Multimodal unsupervised image-to-image\n\ntranslation. In ECCV, 2018.\n\n10\n\n\f[21] S. Iizuka, E. Simo-Serra, and H. Ishikawa. Let there be color!: joint end-to-end learning of\nglobal and local image priors for automatic image colorization with simultaneous classi\ufb01cation.\nACM TOG, 2016.\n\n[22] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing\n\ninternal covariate shift. In ICML, 2015.\n\n[23] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional\n\nadversarial networks. In CVPR, 2017.\n\n[24] X. Ji, W. Wang, M. Zhang, and Y. Yang. Cross-domain image retrieval with attention modeling.\n\nIn ACM Multimedia, 2017.\n\n[25] D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling. Semi-supervised learning with\n\ndeep generative models. In NIPS, 2014.\n\n[26] Y. LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998.\n\n[27] H.-Y. Lee, H.-Y. Tseng, J.-B. Huang, M. Singh, and M.-H. Yang. Diverse image-to-image\n\ntranslation via disentangled representations. In ECCV, 2018.\n\n[28] C. Li and M. Wand. Precomputed real-time texture synthesis with markovian generative\n\nadversarial networks. In ECCV, 2016.\n\n[29] J. Lin, Y. Xia, T. Qin, Z. Chen, and T.-Y. Liu. Conditional image-to-image translation. In CVPR,\n\n2018.\n\n[30] Y.-C. Liu, Y.-Y. Yeh, T.-C. Fu, S.-D. Wang, W.-C. Chiu, and Y.-C. F. Wang. Detach and adapt:\n\nLearning cross-domain disentangled deep representation. In CVPR, 2018.\n\n[31] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation.\n\nIn CVPR, 2015.\n\n[32] L. Ma, Q. Sun, S. Georgoulis, L. Van Gool, B. Schiele, and M. Fritz. Disentangled person\n\nimage generation. In CVPR, 2018.\n\n[33] M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square\n\nerror. In ICLR, 2016.\n\n[34] M. F. Mathieu, J. J. Zhao, J. Zhao, A. Ramesh, P. Sprechmann, and Y. LeCun. Disentangling\n\nfactors of variation in deep representation using adversarial training. In NIPS, 2016.\n\n[35] K. Pang, Y.-Z. Song, T. Xiang, and T. Hospedales. Cross-domain generative learning for\n\n\ufb01ne-grained sketch-based image retrieval. In BMVC, 2017.\n\n[36] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolu-\n\ntional generative adversarial networks. In ICLR, 2015.\n\n[37] S. Reed, K. Sohn, Y. Zhang, and H. Lee. Learning to disentangle factors of variation with\n\nmanifold interaction. In ICML, 2014.\n\n[38] S. E. Reed, Y. Zhang, Y. Zhang, and H. Lee. Deep visual analogy-making. In NIPS, 2015.\n\n[39] S. Rifai, Y. Bengio, A. Courville, P. Vincent, and M. Mirza. Disentangling factors of variation\n\nfor facial expression recognition. In ECCV, 2012.\n\n[40] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image\n\nsegmentation. In ICMICCAI, 2015.\n\n[41] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved\n\ntechniques for training gans. In NIPS, 2016.\n\n[42] M. F. Tappen, W. T. Freeman, and E. H. Adelson. Recovering intrinsic images from a single\n\nimage. In NIPS, 2003.\n\n[43] J. B. Tenenbaum and W. T. Freeman. Separating style and content. In NIPS, 1997.\n\n11\n\n\f[44] L. Tran, X. Yin, and X. Liu. Disentangled representation learning gan for pose-invariant face\n\nrecognition. In CVPR, 2017.\n\n[45] R. Tyle\u02c7cek and R. \u0160\u00e1ra. Spatial pattern templates for recognition of objects with regular structure.\n\nIn GCPR, 2013.\n\n[46] Y. Wang, J. van de Weijer, and L. Herranz. Mix and match networks: encoder-decoder alignment\n\nfor zero-pair image translation. In CVPR, 2018.\n\n[47] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In ECCV, 2016.\n\n[48] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of\n\ndeep networks as a perceptual metric. In CVPR, 2018.\n\n[49] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using\n\ncycle-consistent adversarial networks. In CVPR, 2017.\n\n[50] J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman. Toward\n\nmultimodal image-to-image translation. In NIPS, 2017.\n\n12\n\n\f", "award": [], "sourceid": 676, "authors": [{"given_name": "Abel", "family_name": "Gonzalez-Garcia", "institution": "Computer Vision Center"}, {"given_name": "Joost", "family_name": "van de Weijer", "institution": "Computer Vision Center Barcelona"}, {"given_name": "Yoshua", "family_name": "Bengio", "institution": "U. Montreal"}]}