{"title": "Sequential Neural Processes", "book": "Advances in Neural Information Processing Systems", "page_first": 10254, "page_last": 10264, "abstract": "Neural Processes combine the strengths of neural networks and Gaussian processes to achieve both flexible learning and fast prediction in stochastic processes. However, a large class of problems comprise underlying temporal dependency structures in a sequence of stochastic processes that Neural Processes (NP) do not explicitly consider. In this paper, we propose Sequential Neural Processes (SNP) which incorporates a temporal state-transition model of stochastic processes and thus extends its modeling capabilities to dynamic stochastic processes. In applying SNP to dynamic 3D scene modeling, we introduce the Temporal Generative Query Networks. To our knowledge, this is the first 4D model that can deal with the temporal dynamics of 3D scenes. In experiments, we evaluate the proposed methods in dynamic (non-stationary) regression and 4D scene inference and rendering.", "full_text": "Sequential Neural Processes\n\nGautam Singh\u2217\nRutgers University\n\nsingh.gautam@rutgers.edu\n\nJaesik Yoon\u2217\n\nSAP\n\njaesik.yoon01@sap.com\n\nysson@etri.re.kr\n\nYoungsung Son\n\nETRI\n\nSungjin Ahn\n\nRutgers University\n\nsungjin.ahn@rutgers.edu\n\nAbstract\n\nNeural Processes combine the strengths of neural networks and Gaussian pro-\ncesses to achieve both \ufb02exible learning and fast prediction in stochastic processes.\nHowever, a large class of problems comprises underlying temporal dependency\nstructures in a sequence of stochastic processes that Neural Processes (NP) do not\nexplicitly consider. In this paper, we propose Sequential Neural Processes (SNP)\nwhich incorporates a temporal state-transition model of stochastic processes and\nthus extends its modeling capabilities to dynamic stochastic processes. In applying\nSNP to dynamic 3D scene modeling, we introduce the Temporal Generative Query\nNetworks. To our knowledge, this is the \ufb01rst 4D model that can deal with the tem-\nporal dynamics of 3D scenes. In experiments, we evaluate the proposed methods\nin dynamic (non-stationary) regression and 4D scene inference and rendering.\n\n1\n\nIntroduction\n\nNeural networks consume all training data and computation through a costly training phase to engrave\na single function into its weights. While this makes us entertain fast prediction on the learned function,\nunder this rigid regime changing the target function means costly retraining of the network. This lack\nof \ufb02exibility thus plays as a major obstacle in tasks such as meta-learning and continual learning where\nthe function needs to be changed over time or on-demand. Gaussian processes (GP) do not suffer from\nthis problem. Conditioning on observations, it directly performs inference on the target stochastic\nprocess. Consequently, Gaussian processes show the opposite properties to neural networks: it is\n\ufb02exible in making predictions because of its non-parametric nature, but this \ufb02exibility comes at a\ncost of having slow prediction. GPs can also capture the uncertainty on the estimated function.\nNeural Processes (NP) (Garnelo et al., 2018b) are a new class of methods that combine the strengths\nof both worlds. By taking the meta-learning framework, Neural Processes learn to learn a stochastic\nprocess quickly from observations while experiencing multiple tasks of stochastic process modeling.\nThus, in Neural Processes, unlike typical neural networks, learning a function is fast and uncertainty-\naware while, unlike Gaussian processes, prediction at test time is still ef\ufb01cient.\nAn important aspect for which Neural Processes can be extended is that in many cases, certain\ntemporal dynamics underlies in a sequence of stochastic processes. This covers a broad range of\nproblems from learning RL agents being exposed to increasingly more challenging tasks to modeling\ndynamic 3D scenes. For instance, Eslami et al. (2018) proposed a variant of Neural Processes, called\nthe Generative Query Networks (GQN), to learn representation and rendering of 3D scenes. Although\nthis was successful in modeling static scenes like \ufb01xed objects in a room, we argue that to handle\n\n\u2217Equal contribution\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fmore general cases such as dynamic scenes where objects can move or interact over time, we need to\nexplicitly incorporate a temporal transition model into Neural Processes.\nIn this paper, we introduce Sequential Neural Processes (SNP) to incorporate the temporal state-\ntransition model into Neural Processes. The proposed model extends the potential of Neural Processes\nfrom modeling a stochastic process to modeling a dynamically changing sequence of stochastic\nprocesses. That is, SNP can model a (sequential) stochastic process of stochastic processes. We also\npropose to apply SNP for dynamic 3D scene modeling by developing the Temporal Generative Query\nNetworks (TGQN). In experiments, we show that TGQN outperforms GQN in terms of capturing\ntransition stochasticity, generation quality, generalization to time-horizons longer than those used\nduring training.\nOur main contributions are: We introduce Sequential Neural Processes (SNP), a meta-transfer\nlearning framework for a sequence of stochastic processes. We realize SNP for dynamic 3D scene\ninference by introducing Temporal Generative Query Networks (TGQN). To our knowledge, this is\nthe \ufb01rst 4D generative model that models dynamic 3D scenes. We describe the training challenge\nof transition-collapse unique to SNP modeling and resolve it by introducing the posterior-dropout\nELBO. We demonstrate the generalization capability of TGQN beyond the sequence lengths used\nduring training. We also demonstrate meta-transfer learning and improved generation quality in\ncontrast to Consistent Generative Query Networks (Kumar et al., 2018) gained from the decoupling\nof temporal dynamics from the scene representations.\n\n2 Background\n\n(cid:90)\n\nP (Y |X, C) =\n\nP (Y |X, z)P (z|C)dz\n\nIn this section, we introduce notations and foundational concepts that underlie the design of our\nproposed model as well as motivating applications.\nNeural Processes. Neural Processes (NP) model a stochastic process mapping an input x \u2208 Rdx to a\nrandom variable Y \u2208 Rdy. In particular, an NP is de\ufb01ned as a conditional latent variable model where\na set of context observations C = (XC, YC) = {(xi, yi)}i\u2208I(C) is given to model a conditional\nprior on the latent variable P (z|C), and the target observations D = (X, Y ) = {(xi, yi)}i\u2208I(D) are\nmodeled by the observation model p(yi|xi, z). Here, I(S) stands for the set of data-point indices in\na dataset S. This generative process can be written as follows:\n\nwhere P (Y |X, z) =(cid:81)\n\n(1)\ni\u2208I(D) P (yi|xi, z). The dataset {(Ci, Di)}i\u2208Idataset as a whole contains multi-\nple pairs of context and target sets. Each such pair (C, D) is associated with its own stochastic process\nfrom which its observations are drawn. Therefore NP \ufb02exibly models multiple tasks i.e. stochastic\nprocesses and this results in a meta-learning framework.\nIt is sometimes useful to condition the\nobservation model directly on the context C as well, i.e., p(yi|xi, sC, z) where sC = fs(C) with fs\na deterministic context encoder invariant to the ordering of the contexts. A similar encoder is also\nused for the conditional prior giving p(z|C) = p(z|rC) with fr(C). In this case, the observation\nmodel uses the context in two ways: a noisy latent path via z and a deterministic path via sC.\nThe design principle underlying this modeling is to infer the target stochastic process from contexts\nin such a way that sampling z from P (z|C) corresponds to a function which is a realization of a\nstochastic process. Because the true posterior is intractable, the model is trained via variational\napproximation which gives the following evidence lower bound (ELBO) objective:\n\nlog P\u03b8(Y |X, C) \u2265 EQ\u03c6(z|C,D) [log P\u03b8(Y |X, z)] \u2212 KL(Q\u03c6(z|C, D) (cid:107) P\u03b8(z|C)).\n\n(2)\n\nThe ELBO is optimized using the reparameterization trick (Kingma & Welling, 2013).\nGenerative Query Networks. The Generative Query Network (GQN) can be seen as an application\nof the Neural Processes speci\ufb01cally geared towards 3D scene inference and rendering. In GQN,\nquery x corresponds to a camera viewpoint in a 3D space, and output y is an image taken from the\ncamera viewpoint. Thus, the problem in GQN is cast as: given a context set of viewpoint-image\npairs, (i) to infer the representation of the full 3D space and then (ii) to generate an observation image\ncorresponding to a given query viewpoint.\nIn the original GQN, the prior is conditioned also on the query viewpoint in addition to the context,\ni.e., P (z|x, rC), and thus results in inconsistent samples across different viewpoints when modeling\n\n2\n\n\funcertainty in the scene. The Consistent GQN (Kumar et al., 2018) (CGQN) resolved this by\nremoving the dependency on the query viewpoint from the prior. This resulted in z to be a summary\nof a full 3D scene independent of the query viewpoint. Hence, it is consistent across viewpoints and\nmore similar to the original Neural Processes. For the remainder of the paper, we use the abbreviation\nGQN for CGQN unless stated otherwise.\nFor inferring representations of 3D scenes, a more complex modeling of latents is needed. For\nthis, GQN uses ConvDRAW (Gregor et al., 2016), an auto-regressive density estimator performing\nl=1 P (zl|z<l, rC) where L is the number of auto-regressive rollout steps and rC is a\n\nP (z|C) =(cid:81)L\npooled context representations(cid:80)\n\ni\u2208I(C) fr(xi, yi) with fr an encoding network for context.\n\nState-Space Models. State-space models (SSMs) have been one of the most popular models in\nmodeling sequences and dynamical systems. The model is speci\ufb01ed by a state transition model\nP (zt|zt\u22121) that is sometimes also conditioned on an action at\u22121, and an observation model P (yt|zt)\nthat speci\ufb01es the distribution of the (partial and noisy) observation from the latent state. Although\nSSMs have good properties like modularity and interpretability due to the Markovian assumption,\nthe closed-form solution is only available for simple cases like the linear Gaussian SSMs. Therefore,\nin many applications, SSMs show dif\ufb01culties in capturing nonlinear non-Markovian long-term\ndependencies (Auger-M\u00e9th\u00e9 et al., 2016). To resolve this problem, RNNs have been combined with\nSSMs (Zheng et al., 2017). In particular, the Recurrent State-Space Model (RSSM) (Hafner et al.,\n2018) maintains both a deterministic RNN state ht and a stochastic latent state zt that are updated as\nfollows:\n\nht = fRNN(ht\u22121, zt\u22121),\n\nzt \u223c p(zt|ht),\n\nyt \u223c p(yt|ht, zt).\n\n(3)\n\nThus, in RSSM, the state transition is dependent on all the past latents z<t and thus non-Markovian.\n\n3 Sequential Neural Processes\n\nIn this section, we describe the proposed Sequential Neural Processes which combines the merits of\nSSMs and Neural Processes and thus enabling it to model temporally-changing stochastic processes.\n\ni, yt\n\n3.1 Generative Process\nConsider a sequence of stochastic processes P1, . . . ,PT . At each\ntime-step t \u2208 [1, T ], for a true stochastic process Pt, consider drawing\na set of context observations Ct = {(xt\ni )}i\u2208I(Ct) where I(Ct) are\nthe indices of the observations. Size of this context set may differ over\ntime or it may even be empty. The Ct are provided to the model at their\nrespective time-steps and we want SNP to model Pt as a distribution\nover a latent variable zt, as modeled in NP.\nWhile NP models zt only using Ct i.e., P (zt|Ct), in SNP we want to\nutilize the underlying temporal structure which governs the temporal\nchange in the true stochastic processes Pt\u22121 \u2192 Pt. We achieve this\nby providing the latents of the past stochastic processes z<t to the\ndistribution of the current zt resulting in P (zt|z<t, Ct). Here z<t\nmay be represented as an RNN encoding. The sampled latent zt is\nthen used to model the target observation set Dt = (Xt, Yt) through\nP (Yt|Xt, zt). Like Ct, we assume that Dt is also drawn from the true\nprocess Pt. With an abuse of notation, we use C, D, X, and Y to\nbundle together the Ct, Dt, Xt, and Yt for all time-steps t \u2208 [1, T ],\ne.g., C = (C1, . . . , CT ). With these notations, the generative process\nof SNP is as follows:\n\nT(cid:89)\n\nt=1\n\n3\n\nFigure 1: Generative and infer-\nence (shown as dashed edges)\nmodels in TGQN.\n\nP (Y, Z|X, C) =\n\nP (Yt|Xt, zt)P (zt|z<t, Ct)\n\n(4)\n\nwhere P (Yt|Xt, zt) =(cid:81)\n\non an action at\u22121, but we omit this throughout the paper for brevity.\n\ni\u2208I(Dt) P (yt\n\ni|xt\n\ni, zt) and z0 = null. The transition can also be conditioned\n\nytixtiztCththt+1at\u22121i\u2208I(Dt)\fAlthough we use the RSSM version of SNP in Eqn. (4) where the transition depends on all the past\nz<t, what we propose is a generic SNP class of models that is compatible with a wide range of\ntemporal transition models including the traditional state-space model (Krishnan et al., 2017) as long\nas the latents do not access the previous contexts C<t directly.\nSome of the properties of SNPs are as follows: (i) SNPs can be seen as a generalization of NPs in\ntwo ways. First, if T = 1, an SNP equals an NP. Second, if Dt is empty for all t < T and non-empty\nwhen t = T , SNP becomes an NP which uses the state transition as the (stochastic) context aggregator\ninstead of the standard sum encoding. It then becomes an order sensitive encoding that can in practice\nbe dealt with the order-shuf\ufb02ing on the contexts {Ct}. (ii) SNPs are a meta-transfer learning\nmethod. Consider, for example, a game-playing agent which, after clearing up the current stage,\nlevels up to the next stage where more and faster enemies are placed than the previous stage. With\nSNP, the agent can not only meta-update the policy with only a few observations Ct from the new\nstage, but it can also transfer the general trend from the past, namely, that there will be more and\nfaster enemies in the future stages. As such, we can consider SNP to be a model combining temporal\ntransfer-learning via zt and meta-learning via Ct.\n\n3.2 Learning and Inference\n\nBecause a closed-form solution for learning and inference is not available for general non-linear\ntransition and observation models, we train the model via variational approximation. For this, we\napproximate the true posterior with the following temporal auto-regressive factorization\n\nP (Z|C, D) \u2248 T(cid:89)\n\nt=1\n\nQ\u03c6(zt|z<t, C, D)\n\n(5)\n\nwith z0 = null. Chung et al. (2015); Fraccaro et al. (2016); Krishnan et al. (2017); Hafner et al.\n(2018) provide various implementation options for the above approximation based on RNNs (forward\nor bi-directional) and the reparameterization-trick used. In the next section, we introduce a particular\nimplementation of the above approximate posterior for an application to dynamic 3D-scene modeling.\nWith this approximate posterior, we train the model using the following evidence lower bound\n(ELBO): log P (Y |X, C) \u2265 LSNP(\u03b8, \u03c6) =\n\nEQ\u03c6(zt|V) [log P\u03b8(Yt|Xt, zt)] \u2212 EQ\u03c6(z<t|V) [KL(Q\u03c6(zt|z<t,V) (cid:107) P\u03b8(zt|z<t, Ct))]\n\n(6)\n\nwhere V = (C, D) and log P\u03b8(Yt|Xt, zt) =(cid:80)\n\nt=1\n\ni, zt). We use the reparameteriza-\ntion trick to compute the gradient of the objective. For the derivation of Eqn. (6), see Appendix B.1.\n\ni\u2208I(D) log P\u03b8(yt\n\ni|xt\n\nT(cid:88)\n\n3.3 Temporal Generative Query Networks\n\nConsider a room placed with an object. An agent can control the object by applying some actions\nsuch as translation or rotation. For such setups, whenever an action is applied, the scene changes and\nthus the viewpoint-to-image mapping of GQN learned in the past become stale because the same\nviewpoint now maps to a different image altogether. Although the new scene can be learned again\nfrom scratch using new context from the new scene, an ideal model would also be able to transfer\nthe past knowledge such as object colors as well as utilizing the action to update its belief about the\nnew scene. With a successful transfer, the model would adapt to the new scene with only small or no\ncontext from the new scene.\nTo develop this model, we propose applying SNP to extend GQN into Temporal GQN (TGQN) for\nmodeling complex dynamic 3D scenes. In this setting, at time t, Ct becomes the camera observations,\nat the action provided to the scene objects, zt a representation of the full 3D scene, Xt the camera\nviewpoints and Yt the images. TGQN draws upon the GQN implementation in multiple ways. We\nencode raw image observations and viewpoints into Ct using the same encoder network and use\na DRAW-like recurrent image renderer. Unlike GQN, to capture the transitions, we introduce the\nTemporal-ConvDRAW (T-ConvDRAW) where we condition zl\nt on the past z<t via a concatenation of\n, z<t, Ct). Taking an RSSM approach (Hafner\net al., 2018), ht is transitioned using a ConvLSTM (Xingjian et al., 2015). (See Fig. 1). In inference,\nto realize the distribution in Equation (5), Ct \u222a Dt is provided like in GQN (see Appendix C.2).\n\n(Ct, ht, at). That is, P (zt|z<t, Ct) =(cid:81)L\n\nl=1 P (zl\n\nt|z<l\n\nt\n\n4\n\n\f3.4 Posterior Dropout for Mitigating Transition Collapse\nA novel part of SNP model is the use of the state transition P (zt|z<t, Ct) which is not only condi-\ntioned on the past latents z<t but also on the context Ct. While this makes our model perform the\nmeta-transfer learning, we found that it creates a tendency to ignore the context Ct in the transition\nmodel. It seems that the problem lies in the KL term in Eqn. (6) which drives the training of the\ntransition p\u03b8(zt|z<t, Ct). We note that the two distributions q\u03c6 and p\u03b8 are conditioned on the previous\nlatents z<t which are sampled by providing all the available information C and D. This produces a\nrich posterior with low uncertainty that makes good reconstructions via the decoder. While this is\ndesirable modeling in general, we found that in practice it can make the KL collapse as the transition\nrelies more on z<t while ignoring Ct.\nThis is a similar but not the same problem as the posterior collapsing (Bowman et al., 2015) because in\nour case the cause of the collapse is not an expressive decoder (e.g., auto-regressive), but a conditional\nprior which is already provided rich information about the sequence of tasks from one path via z<t\nand thus open a possibility to ignore the other path Ct. We call this the transition collapse problem.\nTo resolve this, we need a way to (i) limit the information available in z<t to incentivize the use of\nCt information when available while (ii) maintaining the high quality of the reconstructions. We\nintroduce the posterior-dropout ELBO where we randomly choose a subset of time-steps T \u2286 [1, T ].\nFor these time-steps, the zt are sampled using the prior transition p\u03b8. For the remaining time-steps\nin \u00afT \u2261 [1, T ] \\ T , the zt are sampled using the posterior transition q\u03c6. This leads to the following\napproximate posterior:\n\n\u02dcQ(Z) =\n\nP\u03b8(zt|z<t, Ct)\n\nQ\u03c6(zt|z<t, C, D)\n\n(7)\n\n(cid:89)\n\nt\u2208T\n\n(cid:89)\n\nt\u2208 \u00afT\n\nSuch a posterior limits the information contained in the past latents z<t and encourages p\u03b8 to use\nthe context Ct for reducing the KL term. Furthermore, we reconstruct images only for time-steps\nt \u2208 \u00afT using latents sampled from q\u03c6. This is because reconstructing the observations at those\ntime-steps that use prior transitions does not satisfy the principle of auto-encoding, i.e., it then tries to\nreconstruct an observation that is not provided to the encoder and, not surprisingly, would result in\nblurry reconstructions and poorly disentangled latent space. Therefore, the posterior-dropout ELBO\nbecomes: E \u02dcT log P (Y \u02dcT |X, C) \u2265 LPD(\u03b8, \u03c6) =\n\n[log P\u03b8(Yt|Xt, zt) \u2212 KL (Q\u03c6(zt|z<t, C, D) (cid:107) P\u03b8(zt|z<t, Ct))]\n\n(8)\n\n\uf8ee\uf8f0E\n\nE \u02dcT\n\nZ\u223c \u02dcQ\n\n\uf8ee\uf8f0(cid:88)\n\nt\u2208 \u02dcT\n\n\uf8f9\uf8fb\uf8f9\uf8fb\n\nCombining (6) and (8), we take the complete maximization objective as LSNP + \u03b1LPD with \u03b1 an\noptional hyper-parameter. In experiments, we simply set \u03b1 = 0 at the start of the training and set\n\u03b1 = 1 when the reconstruction loss had saturated (see Appendix C.2.5). For derivation of Eqn. (8),\nsee Appendix B.2.\n\n4 Related Works\n\nModeling \ufb02exible stochastic processes with neural networks has seen signi\ufb01cant interest in recent\ntimes catalyzed by its close connection to meta-learning. Conditional Neural Processes (CNP) (Gar-\nnelo et al., 2018a) is a precursor to Neural Processes (Garnelo et al., 2018b) which models the\nstochastic process without an explicit global latent. Without it, the sampled outputs at different query\ninputs are uncorrelated given the context. This is addressed by NP by introducing an explicit latent\npath. A discussion on NP, GQN (Eslami et al., 2018) and CGQN (Kumar et al., 2018) has been\npresented in Sec. 2. To improve the NP modeling further, one line of work pursues the problem of\nunder-\ufb01tting of the meta-learned function on the context. To resolve this, attention on the relevant\ncontext points at query time is shown to be bene\ufb01cial in ANP (Kim et al., 2019). Rosenbaum et al.\n(2018) apply GQN to more complex 3D maps (such as in Minecraft) by performing patch-wise\nattention on the context images.\nIn the domain of SSMs, Deep Kalman Filters (Krishnan et al., 2017) and DVBF (Karl et al., 2016)\nconsist of Markovian state transition models for the hidden latents and an emission model for the\nobservations. But instead of a Markovian latent structure, VRNN (Chung et al., 2015) and SRNN\n\n5\n\n\f(Fraccaro et al., 2016) introduce skip-connections to the past latents making roll-out auto-regressive.\nZheng et al. (2017) and Hafner et al. (2018) propose Recurrent State-Space Models which also takes\nadvantage of the RNNs to model long-term non-linear dependencies. Other variants and inference\napproximations have been explored by Buesing et al. (2018), Fraccaro et al. (2017), Eleftheriadis\net al. (2017), Goyal et al. (2017) and Krishnan et al. (2017). To further model the long-term nonlinear\ndependencies, Gemici et al. (2017) and Fraccaro et al. (2018) attach a memory to the transition\nmodels. Mitigating transition-collapse through posterior-dropout broadly tries to bridge the gap\nbetween what the transition model sees during training and the test time. This intuition is related to\nscheduled sampling introduced by Bengio et al. (2015) which mitigates the teacher-forcing problem.\n\n5 Experiments\n\nWe evaluate SNP on a toy regression task, and 2D and 3D scene modeling tasks. We use NP and\nCGQN as the baselines. We note that these baselines, unlike our model, directly access all the context\ndata points observed in the past at every time-step of an episode and thus result in a strong baseline.\n\n5.1 Regression\n\nWe generate a dataset consisting of sequences of functions. Each function is drawn from a Gaussian\nprocess with squared-exponential kernels. For temporal dynamics between consecutive functions in\nthe sequence, we gradually change the kernel hyper-parameters with an update function and add a\nsmall Gaussian noise for stochasticity. For more details on the data generation, see Appendix D.1.\nWe explore three sub-tasks with different context regimes. In task (a), we are interested in how the\ntransition model generalizes over the time steps. Therefore, we provide context points only in the\n\ufb01rst 10 time-steps out of 20. In task (b), we provide the context intermittently on randomly chosen\n10 time steps out of 20. Our goal is to see how the model incorporates the new context information\nand updates its belief about the time-evolving function. In (a) and (b), the number of revealed points\nare randomly picked between 5 and 50 for each time-step chosen for showing the context. On the\ncontrary, in task (c), we shrink this context size to 1 and provide it in 45 randomly chosen time-steps\nout of 50. Our goal is to test how such highly partial observations can be accumulated and retained\nover the long-term. The models were trained in these settings before performing validation. In\nAppendix C.1, we describe the architectures of SNP and the baseline NP for the 1D regression setting.\nWe present our quantitative results in Fig. 4. We report the target NLL on a held-out set of 1600\nepisodes computed by sampling the latents conditioned on the context as in Kim et al. (2019). In task\n(a), in the absence of context for t \u2208 [11, 20] we expect the transition noise to accumulate for any\nmodel since the underlying true dynamics are also noisy. We note that in contrast to NP, SNP shows\nless degradation in prediction accuracy. In task (b) and (c) as well, the proposed SNP outperforms\nthe NP baseline. In fact, SNP\u2019s accuracy improves with accumulating context while NP\u2019s accuracy\ndeteriorates with time. This is particularly interesting because NP is allowed to access the past\ncontext directly whereas SNP is not. This demonstrates a more effective transfer of past knowledge\nin contrast to the baseline. More qualitative results are provided in Appendix A.1 (Fig. 9). PD was\nnot particularly crucial for training success on the 1D regression tasks (see Fig. 4). Fig. 2 compares\nthe sampled functions.\n\n5.2\n\n2D and 3D Dynamic Scene Inference\n\nWe subject our model to the following 2D and 3D visual scene environments. The 2D environments\nconsist of a white canvas having two moving objects. Objects are picked with a random shape and\ncolor which, to test stochastic transition, may randomly be changed once in any episode with a \ufb01xed\nrule e.g., red \u2194 magenta or blue \u2194 cyan. When two objects overlap, one covers the other based\non a \ufb01xed rule (See Appendix D.2). Given a 2D viewpoint, the agent can observe a 64 \u00d7 64-sized\ncropped portion of the canvas around it. The 3D environments consist of movable object(s) inside a\nwalled-enclosure. The camera is always placed on a circle facing the center of the arena. Based on the\ncamera\u2019s angular position u, the query viewpoint is a vector (cos u, sin u, u). We test the following\ntwo 3D environments: a) Color Cube Environment contains a cube with different colors on each\nface. The cube moves or rotates at each time-step based on the translation actions (Left, Right, Up,\nDown) and the rotation actions (Anti-clockwise, Clockwise) b) Multi-Object Environment: The arena\n\n6\n\n\fDataset\n\nRegime T GQN TGQN\n\nno PD PD\n489 564\nPredict 20 5348\nColor Shapes\nColor Cube (Det.)\n221 226\n380\nPredict 10\nMulti-Object (Det.) Predict 10\n346 357\n844\n482 513\n20 5285\nTrack\nColor Shapes\nColor Cube (Jit.)\n153 156\n20\nTrack\n783\nMulti-Object (Jit.) Track\n450 475\n20 1777\nTable 1: Negative log p(Y |X, C) estimated using\nimportance-sampling from posterior with K = 40.\n\nFigure 2: Sample prediction in 1D regression task (c)\nat t = 33. Blue dots: Past context. Black dots: Current\ncontext. Black dotted line: True function. Blue line:\nPrediction. Blue shaded region: Prediction uncertainty.\n\n(a) Context Set\n\n(b) Generation Roll-Out\n\nFigure 3: TGQN demonstration in Color-Cube Environment. Left: The contexts and actions provided in t < 5.\nTop Right: Scene maps showing the queried camera locations and the true cube and the wall colors. Bottom\nRight: TGQN predictions in 5 \u2264 t \u2264 10.\n\ncontains a randomly colored sphere, a cylinder and a cube with translation actions given to them (see\nAppendix D.3). The action at each time-step is chosen uniformly. The 3D datasets have two versions:\ndeterministic and jittery. In the former, each action has a deterministic effect on the objects. In the\njittery version, a small Gaussian jitter is added to the object motion after the action is executed. The\npurpose of these two versions is described next.\nContext Regimes. We explore two kinds of context regimes: prediction and tracking.\nIn the\nprediction regime, we evaluate the model\u2019s ability to predict future time-steps without any assistance\nfrom the context. So we provide up to 4 observations in each of the \ufb01rst 5 time-steps and let the model\npredict the remaining time-steps (guided only by the actions in the 3D tasks). We also predict beyond\nthe training sequence length (T = 10) to test the generalization capability. This regime is used with\nthe 2D and the deterministic 3D datasets. In the tracking regime, we seek to demonstrate how the\nmodel can transfer past knowledge while also meta-learning the process from the partial observations\nof the current time-step. We, therefore, provide only up to 2 observations at every time-step of the\nroll-out of length T = 20. We test this regime with the 2D and the jittery 3D datasets since, in these\nsettings, the model would keep \ufb01nding new knowledge in every observation.\nBaseline and Performance Metrics. We compare TGQN to GQN as baseline. Since GQN\u2019s original\ndesign does not consume actions, we concatenate the camera viewpoint and the RNN encoding of the\naction sequence up to that time-step to form the GQN query. In the action-less environments, the\nquery is the camera viewpoint concatenated with the normalized t (see Appendix C.3). We report the\nNLL of the entire roll out \u2212 log P (Y |X, C) estimated using 40 samples of Z from Q(Z|C, D). To\nreport the time-step wise generation quality, we compute the pixel MSE per target image averaged\nover 40 generated samples using the prior P (Z|C).\nQuantitative Analysis. In Table 1 and Fig. 4, we compare TGQN trained with posterior dropout (PD)\nversus GQN and versus TGQN trained without PD. TGQN outperforms GQN in all environments\nin both NLL and pixel MSE. In terms of image generation quality in the prediction regime, the\n\n7\n\nRotateAnti-ClockwiseMove\tDownMove\tRightRotateAnti-Clockwisett=5t=6t=7t=8t=9t=10C1C2C3C4Move\tLeftRotate\tClockwiseMove\tUpRotate\tClockwiseMove\tDownC1C2C3C4C1C2C3C4C1C2C3C4C1C2C3C4C1C2C3C4C1C2C3C4\fFigure 4: Comparison of generations of SNP with NP or GQN and comparison between SNP with and without\nposterior-dropout (PD). The latents are rolled-out from the prior conditioned on the context. For 1D regression,\nwe report target NLL. For 2D and 3D settings, we report pixel MSE per image generated at each time-step.\n\npixel MSE gap is sustained even beyond the training horizon. In tracking regime, TGQN with PD\nconverges in the fewest time-steps of observing the contexts. While TGQN continually improves by\nobserving contexts over time, GQN\u2019s performance starts to deteriorate after a certain point. This is\ninteresting since GQN can directly access all the past observations. This demonstrates TGQN\u2019s better\ntemporal modeling and transfer of past knowledge. In general, the use of PD improves generation\nquality in all the explored cases. However, we note that the NLL of TGQN with PD is slightly higher\nthan TGQN without PD. This is reasonable because TGQN with PD does not ignore Ct when the\npast scene modeling in z<t is incorrect. This means that the model must carry extra modeling power\nto temporarily model the incorrect scene until more observations are available and then remodel the\ncorrect scene latent. This explains the tendency towards a slightly higher NLL.\nQualitative Analysis. In Fig. 3, we show a demonstration of TGQN\u2019s predictions for the Color\nCube task. In Fig. 5, we qualitatively show the TGQN generations compared against the true images\nand the GQN generations. We infer the following from the \ufb01gure. a) The dynamics modeled using\np\u03b8(zt|z<t, Ct), can be used to sample long possible futures. This differentiates our modeling from\nthe baselines where a single latent z must compress all the inde\ufb01nite future possibilities. In the 2D\n\n8\n\n14710131619\u22120.6\u22120.4\u22120.200.20.40.60.81TargetNLL1DGPRegressionTasks(a)and(b)NP(a)SNP(a)SNP-PD(a)NP(b)SNP(b)SNP-PD(b)49141950100150200ContextHorizonTPixelMSE3DColor-Cube(Prediction)TGQNGQN49141950100150200ContextHorizonT3DMulti-Object(Prediction)TGQNGQN16111621263136414600.511.52TargetNLL1DGPRegressionTask(c)NP(c)SNPwithPD(c)49141950100150200ContextHorizonTPixelMSE3DColor-Cube(Prediction)TGQNwithPDTGQNwithoutPD49141950100150200ContextHorizonT3DMulti-Object(Prediction)TGQNwithPDTGQNwithoutPD491419200400600800PixelMSE2DColor-Shapes(Tracking)TGQNGQN491419501001502003DJitteryColor-Cube(Tracking)TGQNGQN491419501001502003DJitteryMulti-Object(Tracking)TGQNGQN491419200400600800GenerationTime-SteptPixelMSE2DColor-Shapes(Tracking)TGQNwithPDTGQNwithoutPD49141950100150200GenerationTime-Stept3DJitteryColor-Cube(Tracking)TGQNwithPDTGQNwithoutPD49141950100150200GenerationTime-Stept3DJitteryMulti-Object(Tracking)TGQNwithPDTGQNwithoutPD\fFigure 5: Qualitative comparison of TGQN with GQN (more in Appendix A.1 and A.2). Top: Prediction and\ngeneralization in 2D and deterministic 3D tasks. Bottom: Uncertainty modeling and meta-transfer learning in\n3D jittery color-cube data set. The cube map shows the true face colors and the time-step at which it is revealed.\n\ntask, TGQN keeps generating plausible shape, motion and color changes. GQN fails here because\nthe sampled z does not contain information beyond t = 20, its training sequence-length. b) In the\nColor Cube and the Multi-Object tasks, we observe that TGQN keeps executing the correct object\ntransitions. In contrast, GQN is susceptible to forgetting the face colors in longer-term generations.\nAlthough GQN can generate object positions correctly, this can be credited to the RNN that encodes\nthe action sequence into the query. (Note that this RNN action-encoding is what we additionally\nendow to the vanilla GQN to make a strong baseline.) However, since this RNN is deterministic, this\nmodeling would fail to capture stochasticity in the transitions. c) GQN models the whole roll-out in\na single latent. It is therefore limited in its capacity in modeling \ufb01ner details of the image. We see\nthis through the poorer reconstruction and generation quality in the 3D tasks. d) TGQN can model\nuncertainty and perform meta-transfer learning. We test this in the jittery color-cube task by avoiding\nrevealing the yellow face in the early context and then revealing it at a later time-step. When the\nyellow face is unseen, TGQN samples a face color from the true distribution. Upon seeing the face, it\nupdates its belief and makes the correct color while still remembering the face colors seen earlier.\n\n6 Conclusion\n\nWe introduced SNP, a generic modeling framework for meta-learning temporally-evolving stochastic\nprocesses. We showed that this allows for richer scene representations evidenced by the improved\ngeneration quality that can generalize to longer time-horizons in contrast to NP and GQN while also\nperforming meta-transfer learning. We resolved the problem of transition collapse in training SNP\nusing posterior dropout. This work leaves multiple avenues for improvement. NPs are susceptible to\nunder-\ufb01tting (Kim et al., 2019) and it may also be the case with SNP. It would be interesting to see\nhow the ef\ufb01ciency on the number of observations needed to meta-learn new information could be\nimproved. It would also be interesting to see if an SNP-augmented RL agent can perform better in\nmeta-RL settings than the one without.\n\n9\n\nOne\tface\thidden\tphaseFace\trevelation\tphaset=0t=1910Cube\tMap001t=10t=12t=4ContextsActionsGroundTruthTGQNSample\t1TGQNSample\t2Training\tTime-HorizonGeneralizationt=5t=9t=10t=29Training\tTime-HorizonGeneralizationt=0t=19t=20t=29Context\tHorizont=4t=5ActionsGroundTruthTGQNGQNActionsGroundTruthTGQNGQNGroundTruthTGQNGQN\fAcknowledgments\n\nThis work was supported by Electronics and Telecommunications Research Institute (ETRI) grant\nfunded by the Korean government. [19ZH1100, Distributed Intelligence Core Technology of Hyper-\nConnected Space]. SA thanks to Kakao Brain, Center for Super Intelligence (CSI), and Element AI\nfor their support. JY thanks to Kakao Brain and SAP for their support.\n\nReferences\n\nAbadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving,\nG., Isard, M., et al. Tensor\ufb02ow: A system for large-scale machine learning. In 12th {USENIX}\nSymposium on Operating Systems Design and Implementation ({OSDI} 16), pp. 265\u2013283, 2016.\nAuger-M\u00e9th\u00e9, M., Field, C., Albertsen, C. M., Derocher, A. E., Lewis, M. A., Jonsen, I. D., and\nFlemming, J. M. State-space models\u2019 dirty little secrets: even simple linear gaussian models can\nhave estimation problems. Scienti\ufb01c reports, 6:26677, 2016.\n\nBengio, S., Vinyals, O., Jaitly, N., and Shazeer, N. Scheduled sampling for sequence prediction with\nrecurrent neural networks. In Advances in Neural Information Processing Systems, pp. 1171\u20131179,\n2015.\n\nBowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., and Bengio, S. Generating\n\nsentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015.\n\nBrockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W.\n\nOpenai gym. arXiv preprint arXiv:1606.01540, 2016.\n\nBuesing, L., Weber, T., Racaniere, S., Eslami, S., Rezende, D., Reichert, D. P., Viola, F., Besse, F.,\nGregor, K., Hassabis, D., et al. Learning and querying fast generative models for reinforcement\nlearning. arXiv preprint arXiv:1802.03006, 2018.\n\nChung, J., Kastner, K., Dinh, L., Goel, K., Courville, A. C., and Bengio, Y. A recurrent latent variable\nmodel for sequential data. In Advances in neural information processing systems, pp. 2980\u20132988,\n2015.\n\nEleftheriadis, S., Nicholson, T., Deisenroth, M., and Hensman, J. Identi\ufb01cation of gaussian process\nstate space models. In Advances in neural information processing systems, pp. 5309\u20135319, 2017.\n\nEslami, S. A., Rezende, D. J., Besse, F., Viola, F., Morcos, A. S., Garnelo, M., Ruderman, A., Rusu,\nA. A., Danihelka, I., Gregor, K., et al. Neural scene representation and rendering. Science, 360\n(6394):1204\u20131210, 2018.\n\nFraccaro, M., S\u00f8nderby, S. K., Paquet, U., and Winther, O. Sequential neural models with stochastic\n\nlayers. In Advances in neural information processing systems, pp. 2199\u20132207, 2016.\n\nFraccaro, M., Kamronn, S., Paquet, U., and Winther, O. A disentangled recognition and nonlinear\ndynamics model for unsupervised learning. In Advances in Neural Information Processing Systems,\npp. 3601\u20133610, 2017.\n\nFraccaro, M., Rezende, D., Zwols, Y., Pritzel, A., Eslami, S. A., and Viola, F. Generative temporal\nmodels with spatial memory for partially observed environments. In International Conference on\nMachine Learning, pp. 1544\u20131553, 2018.\n\nGarnelo, M., Rosenbaum, D., Maddison, C. J., Ramalho, T., Saxton, D., Shanahan, M., Teh, Y. W.,\nRezende, D. J., and Eslami, S. Conditional neural processes. arXiv preprint arXiv:1807.01613,\n2018a.\n\nGarnelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D. J., Eslami, S., and Teh, Y. W. Neural\n\nprocesses. arXiv preprint arXiv:1807.01622, 2018b.\n\nGemici, M., Hung, C.-C., Santoro, A., Wayne, G., Mohamed, S., Rezende, D. J., Amos, D., and\nLillicrap, T. Generative temporal models with memory. arXiv preprint arXiv:1702.04649, 2017.\n\n10\n\n\fGoyal, A. G. A. P., Sordoni, A., C\u00f4t\u00e9, M.-A., Ke, N. R., and Bengio, Y. Z-forcing: Training stochastic\nrecurrent networks. In Advances in neural information processing systems, pp. 6713\u20136723, 2017.\n\nGregor, K., Besse, F., Rezende, D. J., Danihelka, I., and Wierstra, D. Towards conceptual compression.\n\nIn Advances In Neural Information Processing Systems, pp. 3549\u20133557, 2016.\n\nHafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent\n\ndynamics for planning from pixels. arXiv preprint arXiv:1811.04551, 2018.\n\nKarl, M., Soelch, M., Bayer, J., and van der Smagt, P. Deep variational bayes \ufb01lters: Unsupervised\n\nlearning of state space models from raw data. arXiv preprint arXiv:1605.06432, 2016.\n\nKim, H., Mnih, A., Schwarz, J., Garnelo, M., Eslami, A., Rosenbaum, D., Vinyals, O., and Teh, Y. W.\n\nAttentive neural processes. arXiv preprint arXiv:1901.05761, 2019.\n\nKingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114,\n\n2013.\n\nKrishnan, R. G., Shalit, U., and Sontag, D. Structured inference networks for nonlinear state space\n\nmodels. In Thirty-First AAAI Conference on Arti\ufb01cial Intelligence, 2017.\n\nKumar, A., Eslami, S., Rezende, D. J., Garnelo, M., Viola, F., Lockhart, E., and Shanahan, M.\n\nConsistent generative query networks. arXiv preprint arXiv:1807.02033, 2018.\n\nMordatch, I., Lowrey, K., and Todorov, E. Ensemble-cio: Full-body dynamic motion planning that\ntransfers to physical humanoids. In 2015 IEEE/RSJ International Conference on Intelligent Robots\nand Systems (IROS), pp. 5307\u20135314. IEEE.\n\nNair, V. and Hinton, G. E. Recti\ufb01ed linear units improve restricted boltzmann machines.\n\nIn\nProceedings of the 27th international conference on machine learning (ICML-10), pp. 807\u2013814,\n2010.\n\nRosenbaum, D., Besse, F., Viola, F., Rezende, D. J., and Eslami, S. Learning models for visual 3d\n\nlocalization with implicit mapping. arXiv preprint arXiv:1807.03149, 2018.\n\nSrivastava, N., Mansimov, E., and Salakhudinov, R. Unsupervised learning of video representations\n\nusing lstms. In International conference on machine learning, pp. 843\u2013852, 2015.\n\nXingjian, S., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., and Woo, W.-c. Convolutional\nlstm network: A machine learning approach for precipitation nowcasting. In Advances in neural\ninformation processing systems, pp. 802\u2013810, 2015.\n\nZheng, X., Zaheer, M., Ahmed, A., Wang, Y., Xing, E. P., and Smola, A. J. State space lstm models\n\nwith particle mcmc inference. arXiv preprint arXiv:1711.11179, 2017.\n\n11\n\n\f", "award": [], "sourceid": 5407, "authors": [{"given_name": "Gautam", "family_name": "Singh", "institution": "Rutgers University"}, {"given_name": "Jaesik", "family_name": "Yoon", "institution": "SAP"}, {"given_name": "Youngsung", "family_name": "Son", "institution": "Electronics and Telecommunications Research Institute"}, {"given_name": "Sungjin", "family_name": "Ahn", "institution": "Rutgers University"}]}