{"title": "Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 137, "page_last": 146, "abstract": "Predicting the future trajectories of multiple interacting pedestrians in a scene has become an increasingly important problem for many different applications ranging from control of autonomous vehicles and social robots to security and surveillance. This problem is compounded by the presence of social interactions between humans and their physical interactions with the scene. While the existing literature has explored some of these cues, they mainly ignored the multimodal nature of each human's future trajectory which is noticeably influenced by the intricate social interactions. In this paper, we present Social-BiGAT, a graph-based generative adversarial network that generates realistic, multimodal trajectory predictions for multiple pedestrians in a scene. Our method is based on a graph attention network (GAT) that learns feature representations that encode the social interactions between humans in the scene, and a recurrent encoder-decoder architecture that is trained adversarially to predict, based on the features, the humans' paths. We explicitly account for the multimodal nature of the prediction problem by forming a reversible transformation between each scene and its latent noise vector, as in Bicycle-GAN. We show that our framework achieves state-of-the-art performance comparing it to several baselines on existing trajectory forecasting benchmarks.", "full_text": "Social-BiGAT: Multimodal Trajectory Forecasting\nusing Bicycle-GAN and Graph Attention Networks\n\nVineet Kosaraju1\u2217 Amir Sadeghian1,2\u2217 Roberto Mart\u00edn-Mart\u00edn1\n\nS. Hamid Rezato\ufb01ghi1,3 Silvio Savarese1\n\nIan Reid3\n\n1Stanford University\n\n2 Aibee Inc\n\n3 University of Adelaide\n\nvineetk@stanford.edu\n\nAbstract\n\nPredicting the future trajectories of multiple interacting agents in a scene has be-\ncome an increasingly important problem for many different applications ranging\nfrom control of autonomous vehicles and social robots to security and surveillance.\nThis problem is compounded by the presence of social interactions between humans\nand their physical interactions with the scene. While the existing literature has\nexplored some of these cues, they mainly ignored the multimodal nature of each\nhuman\u2019s future trajectory. In this paper, we present Social-BiGAT, a graph-based\ngenerative adversarial network that generates realistic, multimodal trajectory pre-\ndictions by better modelling the social interactions of pedestrians in a scene. Our\nmethod is based on a graph attention network (GAT) that learns reliable feature\nrepresentations that encode the social interactions between humans in the scene,\nand a recurrent encoder-decoder architecture that is trained adversarially to predict,\nbased on the features, the humans\u2019 paths. We explicitly account for the multimodal\nnature of the prediction problem by forming a reversible transformation between\neach scene and its latent noise vector, as in Bicycle-GAN. We show that our frame-\nwork achieves state-of-the-art performance comparing it to several baselines on\nexisting trajectory forecasting benchmarks.\n\n1\n\nIntroduction\n\nFor a variety of applications, accurate pedestrian trajectory forecasting is becoming a crucial compo-\nnent. Autonomous vehicles such as self-driving cars, and social robotics such as delivery vehicles\nmust be able to understand human movement to avoid collisions [1\u20134]. Intelligent tracking and\nsurveillance systems used for city planning must be able to understand how crowds will interact to bet-\nter manage infrastructure [5\u20138]. Trajectory prediction is also becoming crucial enabling downstream\ntasks, such as tracking and re-identi\ufb01cation [9]. However, trajectory prediction is still a challenging\ntask because of several properties inherent to human behavior:\n\u2022 Social Interactions When humans move in public, they often interact socially with other pedes-\ntrians [10]. From taking actions to avoid collisions, to walking in groups, there are several ways\nhumans interact while moving that require prediction methods to model social behavior [11, 12].\nThese social interactions may not be necessarily in\ufb02uenced by people\u2019s spatial proximity.\n\u2022 Scene Context Pedestrian behavior is not only dependent on the people around them, but is also\nhighly dependent on the physical scene around them [12\u201316]. This includes not just stationary\nobstacles that cannot be avoided, such as buildings, but also different physical cues present visually,\nsuch as sidewalks or grass which may enable or restrict human movement.\n\u2217indicates equal contribution\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fFigure 1: We show multimodal behavior for the blue pedestrian, who must make a decision about which\ndirection they will take to avoid the red-green pedestrian group.\n\u2022 Multimodal Behavior Pedestrians may follow several plausible trajectories, as there is a rich\ndistribution of potential human behavior [10, 11, 17, 18]. For example, when two pedestrians\nare walking towards each other, several modes of behavior develop, such as moving to the left or\nmoving to the right. Within each mode, there is also a large variance, allowing pedestrians to vary\nfeatures like their speed.\n\nPrior work in trajectory forecasting have tackled several of the previously listed challenges and\nhave informed our architectural design. Helbing et al. [19] and Pellegriniet al. [20] successfully\ndemonstrated the bene\ufb01t of modeling social interactions but require handcrafted rules that are less\nable to generalize to new scenes. Alahi et al. [10] utilized recurrent architectures to consider multiple\ntimesteps of pedestrian behavior, but do not consider the physical cues of the scene. Other prior\nresearch has also focused on understanding the physical scene. Lee et al. [15] and Sadeghian et\nal. [16] use raw scene images and soft attention on the scene to highlight important cues. Their work\nis limited by not considering social cues jointly with the scene.\nBy contrast, Gupta et al. [11] and Sadeghian et al. [12] utilize GANs with social mechanisms that\ndo take into account all people in the scene. However both models fall short of learning the truly\nmultimodal distribution of human behavior, and instead learn a single mode of behavior with high\nvariance. Further, both models are limited by how they learn social behavior: while the former\nloses information by using the same social vector for all pedestrians in a scene, the latter requires a\nhand-de\ufb01ned sorting operation that may not perform optimally in all cases.\nTo address the limitations of these works, we propose Social-BiGAT, a GAN [21] based approach to\nconstruct a generative model that can learn these essential multimodal trajectory distributions. The\nmain contributions of this work are as follows. First, we improve the modeling of social interactions\nbetween pedestrians in a scene by introducing a \ufb02exible graph attention network [22], where all\npedestrians in a scene are allowed to interact. This improves over prior works where either interactions\nwere limited locally, or interactions were modelled using hand-de\ufb01ned rules. Next, we encourage\ngeneralization towards a multimodal distribution by constructing a reversible mapping between\noutputted trajectories and latents that represent the pedestrian behavior in a scene, as previously\nperformed for images by Zhu et al. [23]. This allows us to generate trajectories that are socially and\nphysically acceptable, while also learning a larger multimodal trajectory distribution, despite only\nhaving access to single samples from single modes of behavior across scenes. Finally, we incorporate\nphysical scene cues using soft attention as in [12, 16] to make our model more generalizable.\n\n2 Related Work\n\nIn recent years due to the rise of popularity in development of autonomous driving systems and\nsocial robots, the problem of trajectory forecasting has received signi\ufb01cant attention from many\nresearchers in the community. The majority of existing works have been focused on the effects of\nincorporating physical features of the scene into human-space models [15, 16], as well as learning\nhow to model social behavior between pedestrians in human-human models [10, 24]. Other works\nhave approached the problem from a generative setting [11] and have jointly modeled these features\nin one framework [12]. While these works have greatly advanced the \ufb01eld, they have drawbacks that\nwe address by incorporating graph attention networks [22] and image translation networks [23].\n\n2\n\n\fTrajectory Forecasting Traditionally, pedestrian trajectory prediction has been tackled by de\ufb01ning\nhandcrafted rules and energy parameters that capture human motion but fail to generalize properly [19,\n20, 24\u201326]. Instead of handcrafting these features, modern approaches rely on recurrent neural\nnetworks that learn these parameters directly from the data [10, 16], while incorporating some means\nof capturing human interaction features [15, 27, 28]. Several of these prior methods have been limited\nin scope, as they often limit interactions to nearby pedestrian neighbors [10, 29, 30] and do not\nmodel global interactions or cannot generalize to a variable number of humans. Other approaches\nhave explored trajectory prediction from a generative standpoint, including Lee et al. [15], Gupta et\nal. [11], and Sadeghian et al. [12], with their own limitations. The former only considers interactions\nwithin a limited local scope, and the latter two result in models with high variances. Speci\ufb01cally,\nalthough human motion is inherently multimodal, these methods are not able to expressively learn this\nmultimodal behavior and instead learn one mode with a high variance. In our work we incorporate\nideas from image to image translation to generate multimodal pedestrian trajectories. Furthermore,\nour model uses graph attention networks [22] to more ef\ufb01ciently and robustly model the interactions\nbetween the agents in the scene, whereas prior research [12, 31] depend on hand-de\ufb01ned rules.\nGraph Attention Networks Proposed by Velickovi et al. [22], graph attention networks (GAT) allow\nfor the application of a self-attention based architecture over any type of structured data that can\nbe represented as a graph. These networks build upon the prior advances of graph convolutional\nnetworks (GCN) [32] by also allowing for the model to implicitly assign different importances to\nnodes in the graph. In our case, we can formulate pedestrian interactions as a graph, where nodes\nrefer to human humans, and edges are these interactions; higher edge weights correspond to more\nimportant interactions. By leaving the graph fully connected, we can model interactions between\nhumans without using pooling [11] or sorting [12] that may lose important features.\nImage Translation The \ufb01eld of image domain translation has gone through several seminal advance-\nments in the past couple years. The \ufb01rst advancement was made with the pix2pix framework [33],\nwhich enabled translation but was limited by requiring paired training examples. Zhu et al. improved\nthis model with CycleGAN [34], which was able to learn these domain mappings with unpaired\nexamples from each domain through a cycle consistency loss. Newer research has focused on learning\nmultimodality of the output: InfoGAN [35] focuses on maximizing variational mutual information,\nwhile BicycleGAN [23] introduces a latent noise encoder and learns a bijection between noise and\noutput. In our model we draw upon the advancements suggested by BicycleGAN to propose a latent\nspace encoder that allows for multimodal pedestrian trajectory generation.\n\n3 Social-BiGAT\n\n3.1 Problem De\ufb01nition\n\nFormally de\ufb01ned, human trajectory prediction is the problem of predicting the future navigation\nmovements of pedestrians (namely their x and y coordinates on a 2D map representation), given their\nprior movements and additional contextual information about the scene. We assume the route taken by\neach pedestrian is in\ufb02uenced by the location of other humans and the physical constraints on its path,\nas well as its own goal, which is to some extent encoded in its past course of movements. For any\nparticular scene, the inputs to our model are twofold: 1) scene information, in the form of a top-down\nor side-view image of the scene, I t, and 2) the previously observed trajectory within the scene of each\nof the N currently visible pedestrians, Xi = {(xt\ni ) \u2208 R2|t = 1, . . . , tobs} for \u2200i \u2208 {1, . . . , N}.\nGiven all above inputs and the ground truth future trajectory for each pedestrian between tpred and\ntobs timesteps, i.e. Yi = {(xt\ni ) \u2208 R2|t = tobs + 1, . . . , tpred} for \u2200i \u2208 {1, . . . , N}, our goal is to\nlearn the underlying (and potentially, multimodal) distribution which can generate feasible samples\nfor their future trajectories, i.e. \u02c6Yi for \u2200i \u2208 {1, . . . , N}.\n\ni, yt\n\ni, yt\n\n3.2 Overall Model\n\nOur overall model consists of four main networks, each of which is made up of three key modules\n(Figure 2). Speci\ufb01cally, we construct a generator, two forms of discriminators (one that operates at\nlocal pedestrian scale, and one that operates at a global scene-level scale), and a latent space encoder.\nOur generator is composed of a feature encoder module (Section 3.3), an attention network module\n(Section 3.4), and a decoder module (Section 3.5). The feature encoder module extracts encodings\nfrom raw features for use in the attention network, which in turn learns which features are most\n\n3\n\n\fFigure 2: Architecture for the proposed Social-BiGAT model. The model consists of a single generator, two\ndiscriminators (one at local pedestrian scale, and one at global scene scale), and a latent encoder that learns\nnoise from scenes. The model makes use of a graph attention network (GAT) and self-attention on an image to\nconsider the social and physical features of a scene.\n\nimportant in generation. These weighted features are then passed into the decoder module, which\nuses LSTMs to generate multiple timesteps of trajectories. The architecture is trained adversarially\nwith both discriminators, as motivated by Isola et al. [33] and to encourage realistic local and global\ntrajectories, and we also train a latent scene encoder that learns to generate a mean and variance for\nthe noise that best represents a scene jointly, as in Zhu et al. [23] to encourage multimodality.\n\n3.3 Feature Encoder\n\nThe feature encoder has two main components: a social pedestrian encoder, in order to learn\nrepresentations of observed pedestrian trajectories, and a physical scene encoder, in order to learn the\nrepresentation of the scene features. For the social encoder, for each pedestrian we \ufb01rst embed the\npedestrian\u2019s relative displacements into a higher dimension using a multilayer perceptron (MLP), and\nthen encode these pedestrian movements across timesteps into a single embedding using a LSTM,\nresulting in encoding Vs(i) for pedestrian i. For the physical feature encoder, we simply pass the\ntop-down image view of the scene through a convolutional neural network (CNN), resulting in Vp:\n(1)\n(2)\n\nVs(i) = LST Men(M LPemb(Xi, Wemb), hen(i); Wen)\nVp = CN N (I; Wcnn)\n\n3.4 Attention Network\n\nMuch like how humans intuitively know which other pedestrians to focus on to avoid collisions, we\nwant our model to better understand the relative weight that interactions have: we accomplish this\ngoal by applying attention over our extracted features.\nPhysical Attention To apply attention over our physical features relative to a speci\ufb01c pedestrian,\nwe take in Vs(i), and apply soft attention, where the network is parameterized by Wp and outputs\ncontext vector Cp\n\nt(i):\n\nCp(i) = AT Tp(Vp, Vs(i); Wp)\n\n(3)\n\n4\n\n\fSocial Attention Similar to physical attention, we use as input to our social attention model the\nembeddings of pedestrians, Vs(i). The social attention model encodes pedestrians as weighted (at-\ntended) sum of the neighbor pedestrians they interact with. Prior research has used either permutation\ninvariant symmetric functions, such as max or average [11], or ordering functions such as sorting\nbased on euclidean distance [12]. In the former, the downside is that each pedestrian receives an\nidentical joint feature representation that discards some uniqueness. While the latter technique does\nnot suffer from this drawback, it does require setting a maximum number of pedestrians and does\nimpose a human bias on the model that is not necessarily always true. Namely, it assumes that\neuclidean distance ordering is a key component of understanding social interactions.\nTo avoid these \ufb02aws, we utilize graph attention networks [22, 36]. Given pedestrian i\u2019s embedding,\nVs(i), for all pedestrians in the scene, we apply several stacked graph attention layers. Each layer, (cid:96),\nis applied as follows, where Wgat parameterizes a shared linear transformation and a is the shared\nattentional mechanism:\n\neij = a(WgatVs(i), WgatVs(j))\n\u03b1ij = softmaxj(eij)\n\n(cid:88)\n\nj\u2208N\n\nCs\n\n(cid:96)(i) =\n\n\u03b1ijWgatVs(j)\n\n(4)\n(5)\n(6)\n\nWe use the features C L\ns from the last GAT layer where (cid:96) = L as the \ufb01nal social features. We allow\nthe graph of pedestrians to remain fully connected and do not apply any mask. This allows each\npedestrian to interact with each other and does not impose any restriction on pedestrian orders.\n\n3.5 GAN Network\n\nIn this section we present how our feature encoder and attention network serve as core building blocks\nin developing the LSTM based Generative Adversarial Network (GAN). GANs typically consist of\ntwo networks that compete with each other: a generator, and a discriminator. While the generator\nlearns to generate realistic samples from input data, the discriminator learns to discern which samples\nare real, and which are generated, thereby engaging in a two-player min-max game.\nGenerator The generator is built using a decoder LSTM. Similar to conditional GANs [37], our\ngenerator takes as input a noise vector z sampled from a multivariate normal distribution, and\nL(i), and\nis conditioned on the physical scene context, Cp(i), the pedestrian scene context, Cs\nthe previous pedestrian encoding, Vs(i). These are all concatenated together such that Cg(i) =\nL(i), Cp(i), z]. Generation of trajectories across multiple timesteps is then performed\n[Vs(i), Cs\nthrough a decoder LSTM, such that:\n\n\u02c6Yi = M LPd(LST Mdec(Cg(i), hdec(i); Wdec); Wd)\n\n(7)\n\nDiscriminator The discriminator architecture mirrors that of the generator, with encoder LSTMs\nused to represent pedestrians, and a CNN used to represent scene features. We propose two versions\nof this core discriminator architecture: one at local scale, operating on pedestrians, and one at global\nscale, operating on an entire scene. The former performs classi\ufb01cation directly on encodings of\nconcatenated past and future trajectories, such that:\n\n\u02c6L(i) = M LPclf (LST Men(M LPemb([Xi, \u02dcYi], Wemb), hen(i); Wen); Wclf ),\n\n(8)\nwhere \u02dcYi \u223c p(Yi, \u02c6Yi) is a randomly chosen future trajectory sample from the either ground truth or\npredicted path. \u02c6L(\u00b7) is classi\ufb01cation score representing the sample is a ground truth (real) or predicted\n(fake) with the truth label L(i) = 1 and L(i) = 0, respectively.\nThe global discriminator performs the same classi\ufb01cation operation, but on the global context vector\nfor the pedestrian trajectory; namely, the concatenation of the physical scene context, Cp(i), the\npedestrian scene context, Cs\n\nL(i), and the pedestrian encoding, Vs(i).\n\n3.6 Latent Encoder\n\nIn order to generate trajectories that are truly multimodal, we encourage our model to develop a\nbijection between the outputted trajectories and the latent space inputted to the generator. Speci\ufb01cally,\n\n5\n\n\fFigure 3: Training process for the Social-BiGAT model. We teach the generator and discriminators using\ntraditional adversarial learning techniques, with an additional L2 loss on generated samples to encourage\nconsistency. We further train the latent encoder by ensuring it can recreate noise passed into the generator, and\nby making sure it mirrors a normal distribution.\n\nwe want to map both the latent noise to an output trajectory, as well as map that trajectory back to the\noriginal latent. While the former task is accomplished by a generator, we perform the latter using a\nlatent scene encoder, as previously performed in Zhu et al. [23].\nThe architecture for the latent scene encoder is relatively similar to the local discriminator. First,\npedestrians are encoded in the scene using a LSTM encoder. Embeddings from this LSTM are passed\nin two parallel MLPs that are trained to output a mean \u00b5i and log variance \u03c32\ni for each pedestrian:\n(9)\n(10)\nMeans and log variances across pedestrians are max pooled together to generate a single mean and\nlog variance representation of the latent for a given scene.\n\n\u00b5i = M LP\u00b5(M LPL(Vs(i), WL), W\u00b5)\ni = M LP\u03c3(M LPL(Vs(i), WL), W\u03c3)\n\nlog \u03c32\n\n3.7 Losses\n\nAs illustrated in Figure 3, to train these four models we have a multistep training process, where\nwe not only perform a transformation starting from the noise, z \u2192 \u02c6Yi \u2192 \u02c6z, but also perform a\ntransformation starting from the trajectories, Yi \u2192 z \u2192 \u02c6Yi. In the former we have two main loss\nterms to consider: the GAN loss (Lgan1) from the generator fooling the discriminator, and the\ndiscriminator correctly classifying the generator, as well as a loss term on reconstructing the noise\n(Lz). We calculate these as follows, where G refers to the generator, D to the discriminator and E to\nthe latent encoder:\n\nLgan1 = E log D(Xi, Yi) + E log(1 \u2212 D(Xi, \u02c6Yi))\nLz = ||E( \u02c6Yi) \u2212 z||1\n\n(11)\n(12)\nIn the latter, we have three additional loss terms: the GAN loss (Lgan2 ), a L2 loss on trajectories\n(Ltraj), enforcing the generation of real samples, and a KL loss on the generated noise (Lkl) such\nthat it resembles noise drawn from a random Gaussian:\n\nLgan2 = E log D(Xi, Yi) + E log(1 \u2212 D(Xi, G(Xi, E(Yi))))\nLtraj = ||Yi \u2212 G(Xi, E(Yi))||2\nLkl = E[Dkl(E(Yi)||N (0, I))]\n\n(13)\n(14)\n(15)\n\n(16)\n\nWe ultimately combine all these loss terms using \u03bb weights that are chosen as hyperparameters:\n\nargmax\n\n[Lgan1 + \u03bbzLz + Lgan2 + \u03bbtrajLtraj + \u03bbklLkl]\n\nG\u2217, D\u2217, E\u2217 = argmin\n\nG,E\n\nD\n\n4 Experiments\n\nWe perform experiments on two relevant datasets: ETH [20] and UCY [38]. Both contain annotated\ntrajectories of socially interacting pedestrians in real world scenes. The datasets include different types\n\n6\n\n\fDiscriminative\n\nGenerative\n\nLin\nDataset\nETH\n1.33 / 2.94\nHOTEL 0.39 / 0.72\nUNIV\n0.82 / 1.59\nZARA1\n0.62 / 1.21\nZARA2\n0.77 / 1.48\nAVG\n0.79 / 1.59\n\nS-LSTM S-GAN-P\n0.87 / 1.62\n1.09 / 2.35\n0.67 / 1.37\n0.79 / 1.76\n0.76 / 1.52\n0.67 / 1.40\n0.47 / 1.00\n0.35 / 0.68\n0.42 / 0.84\n0.56 / 1.17\n0.72 / 1.54\n0.61 / 1.21\n\nSophie\n0.70 / 1.43\n0.76 / 1.67\n0.54 / 1.24\n0.30 / 0.63\n0.38 / 0.78\n0.54 / 1.15\n\nGAT\n0.68 / 1.29\n0.68 / 1.40\n0.57 / 1.29\n0.29 / 0.60\n0.37 / 0.75\n0.52 / 1.07\n\nOurs\n\nBiGAN\n0.72 / 1.47\n0.54 / 1.12\n0.55 / 1.34\n0.32 / 0.65\n0.49 / 0.88\n0.52 / 1.09\n\nSocial-BiGAT\n0.69 / 1.29\n0.49 / 1.01\n0.55 / 1.32\n0.30 / 0.62\n0.36 / 0.75\n0.48 / 1.00\n\nTable 1: Baseline models compared to our architectures when predicting 12 future timesteps, given the previous\n8. Errors reported are ADE / FDE in meters, with generative models being evaluated using K = 20 samples.\n\nModel\nS-GAN-P\nSophie\nGAT\nBiGAN\nSocial-BiGAT\n\nK = 20\n0.558 / 1.118\n0.526 / 1.030\n0.518 / 1.064\n0.523 / 1.091\n0.476 / 0.998\n\nK = 10\n0.594 / 1.214\n0.566 / 1.122\n0.529 / 1.127\n0.531 / 1.144\n0.488 / 1.096\n\nK = 5\n0.650 / 1.316\n0.604 / 1.266\n0.584 / 1.241\n0.579 / 1.298\n0.527 / 1.260\n\nK = 1\n0.846 / 1.758\n0.712 / 1.456\n0.682 / 1.494\n0.662 / 1.439\n0.606 / 1.328\n\n% Increase\n51.6% / 57.2%\n35.3% / 41.4%\n31.6% / 40.4%\n26.6% / 31.9%\n27.3% / 33.1%\n\nTable 2: Effect of varying K in evaluation results for generative models. We see that reducing K results in a\nhigher average ADE/FDE across the \ufb01ve scenes for S-GAN-P and Sophie, due to higher distribution variances.\n\nof social interactions, ranging from group formation to collision avoidance, the type of interaction\nwe aim to encode with our Social-BiGAT model. The datasets contain \ufb01ve unique scenes: Zara1,\nZara2, Univ, Eth, and Hotel. We evaluate Social-BiGAT on these datasets and compare to several\ndeterministic baselines, including a linear regressor that minimizes least square error, Linear, and\na predictive model using LSTMs and social pooling, S-LSTM [10], as well as two main generative\nmodels: S-GAN-P, which applies generative modeling to social LSTMs [11], and Sophie, which\napplies attention networks to social GANs [12]. We present evaluation results of three versions\nof our model: one trained without the latent scene encoder but with the graph attention network,\nGAT, one trained without the graph attention network but with the latent scene encoder, BiGAN, and\nour \ufb01nal model with all components included, Social-BiGAT. Models are evaluated using two main\nmetrics: average displacement error (ADE), and \ufb01nal displacement error (FDE). Both are de\ufb01ned as\nthe average L2 distance between the ground truth and predicted trajectories. Evaluation occurs over a\ntimescale of 8 seconds, where the \ufb01rst 3.2 seconds (8 timesteps) correspond to observed data, and the\nlast 4.2 seconds (12 timesteps) correspond to predicted future data. We evaluate using a hold-one-out\ncross evaluation strategy in meter space, with N-K variety loss, as previously performed [11, 12].\n\n4.1 Quantitative Results\n\nWe compare our model to various baselines in Table 1, reporting the average displacement error\n(ADE) and \ufb01nal displacement error (FDE) for 12 timesteps of pedestrian movement. As expected\nwe see that both the discriminative Social LSTM baseline outperforms the simple linear model,\nand that the generative baselines, which are evaluated from K = 20 samples, improve upon the\ndiscriminative ones by generating a full distribution of possible human trajectories. In terms of our\nproposed architectures, we see that incorporating the GAT alone does indeed improve performance,\nas the network is able to more \ufb02exibly account for pedestrian interactions. Alternatively the BiGAN\nalone does not help performance. Our combined GAT and BiGAN architecture, Social-BiGAT, does\nhowever achieve the best performance of all our models, resulting in a 0.15 meter decrease in average\nFDE from the previous state-of-the-art model. This is due to the reduced errors for the Hotel scene\ncompoared to other generative architectures.\nWhile the BiGAN architecture does not help performance much when K = 20, we show in Table 2\nthat it does help improve generalization at lower settings of K. Speci\ufb01cally, while S-GAN-P and\nSophie suffer from higher variances, causing their ADE and FDE to increase dramatically when\nK is lowered, Social-BiGAT\u2019s ADE and FDE increase more slowly. Further, we see that the GAT\narchitecture initially performs better than the BiGAN, but with fewer samples the GAT error increases\nfaster. This aligns with our intuition that the inclusion of the BiGAN in our architecture enables\nfor better capturing of a multimodal distribution, instead of generating samples from a unimodal\ndistribution. This suggests that the inclusion of the latent scene encoder in BiGAN and Social-BiGAT\nallow for the architecture to reduce the variance of the outputted trajectory distributions while also\nallowing for better generalization.\n\n7\n\n\fFigure 4: Generated trajectories visualized for the S-GAN-P, Sophie, and Social-BiGAT models across four\nmain scenes. Observed trajectories are shown as solid lines, ground truth future movements are shown as dashed\nlines, and generated samples are shown as contour maps. Different colors correspond to different pedestrians.\n\nFigure 5: Visualizations of Social-BiGAT behavior, with attention weights shown with respect to the red agent\nin (a, b, c, d) and latent space exploration in scene (e). The attention weights derive social features beyond\ndistance, such as collision avoidance, while adjusting the latents modi\ufb01es the aggressiveness and speed of agents.\n\n4.2 Qualitative Results\n\nIn order to better understand the contribution of the graph attention network and bicycle training\nstructure in improving understanding of social behavior, we visualize the generated trajectories\nfor four scenes, comparing our proposed Social-BiGAT model to S-GAN-P and Sophie (Figure\n4). We draw three main conclusions from these visualizations. First, as shown in scenes 1 and 2,\nSocial-BiGAT often has a lower variance than S-GAN-P and Sophie, suggesting that it can generate\nmore ef\ufb01ciently. Second, as shown in scenes 2 and 3, the model is better able to model the interactions\nof people travelling in crowds or groups. Finally, as scene 4 demonstrates, the model can generate\nrealistic trajectories for pedestrians that are attempting to avoid collisions. Each of these \ufb01ndings are\ncrucial in ensuring that the model performs optimally across a wide range of social behavior.\nIn addition to visualizing our model\u2019s trajectories in comprison to prior generative baselines, we also\ndepict the attention weights of our model and the impact of modifying z while keeping the scene \ufb01xed\nin Figure 5. In scenes (a, b), the attention weight roughly lines up with Euclidean distance. Scenes\n(c, d) show that the attention further generalizes in learning which agents are important socially: in\nScene (c) it pays large attention to the blue agent it may collide with in the future, even though that\nagent is farther away from it than the green one, and in Scene (d) it ignores the blue agent for the\nfarther green agent with whom it might collide with. Finally in (e), we adjust the latent z resulting in\nbehavior between the blue and green agents that ranges from cautious (top) to aggressive (bottom).\n\n5 Conclusion\n\nWe presented Social-BiGAT, a novel architecture for forecasting pedestrian movements that outper-\nforms prior state-of-the-art methods across several widely used trajectory benchmarks. Unlike prior\nresearch, our model is able to generate multiple trajectories for multiple humans in a multimodal\nfashion. Through our evaluations and visualizations we demonstrated that Social-BiGAT is able\nto capture the intricate social nature of pedestrian movements and that we are able to control the\npredictions by adjusting the latents at test time. We further introduced several important architectural\nimprovements to the generation process: 1) we utilize a social attention graph network (GAT) to better\nlearn pedestrian interactions through the data, and 2) we train using two discriminators that operate at\nlocal and global scale. As shown experimentally, with these design patterns our Social-BiGAT model\nis able to generate pedestrian trajectories that more realistically predict human motion.\n\n6 Acknowledgement\n\nThe research reported in this publication was supported by funding from the TRI gift, ONR (1165419-\n10-TDAUZ), Nvidia, and Samsung.\n\n8\n\na.b.c.d.+10.47%-10.47%+2.08%+1.91%-3.99%+41.27%-41.27%+10.01%-10.01%AgressivenessLinearitySpeedAgressivenessLinearitySpeede.\fReferences\n[1] Timur M. Bagautdinov, Alexandre Alahi, Fran\u00e7ois Fleuret, Pascal Fua, and Silvio Savarese. Social scene\nunderstanding: End-to-end multi-person action localization and collective activity recognition. 2017 IEEE\nConference on Computer Vision and Pattern Recognition (CVPR), pages 3425\u20133434, 2017.\n\n[2] Wei-Chiu Ma, De-An Huang, Namhoon Lee, and Kris M. Kitani. Forecasting interactive dynamics of\npedestrians with \ufb01ctitious play. 2017 IEEE Conference on Computer Vision and Pattern Recognition\n(CVPR), pages 4636\u20134644, 2017.\n\n[3] S\u00e9bastien Forestier, Yoan Mollard, Damien Caselli, and Pierre-Yves Oudeyer. Autonomous exploration,\nactive learning and human guidance with open-source poppy humanoid robot platform and explauto library.\nIn The Thirtieth Annual Conference on Neural Information Processing Systems (NIPS 2016), 2016.\n\n[4] Julia Kantorovitch, Janne V\u00e4re, Vesa Pehkonen, Arto Laikari, and Heikki Sepp\u00e4l\u00e4. An assistive household\n\nrobot\u2013doing more than just cleaning. Journal of Assistive Technologies, 8(2):64\u201376, 2014.\n\n[5] Brendan Tran Morris and Mohan Manubhai Trivedi. A survey of vision-based trajectory learning and\nanalysis for surveillance. IEEE transactions on circuits and systems for video technology, 18(8):1114\u20131127,\n2008.\n\n[6] Sangmin Oh, Anthony Hoogs, Amitha Perera, Naresh Cuntoor, Chia-Chih Chen, Jong Taek Lee, Saurajit\nMukherjee, JK Aggarwal, Hyungtae Lee, Larry Davis, et al. A large-scale benchmark dataset for event\nIn Computer vision and pattern recognition (CVPR), 2011 IEEE\nrecognition in surveillance video.\nconference on, pages 3153\u20133160. IEEE, 2011.\n\n[7] Nick Mould, James L Regens, Carl J Jensen III, and David N Edger. Video surveillance and counterterror-\nism: the application of suspicious activity recognition in visual surveillance systems to counterterrorism.\nJournal of Policing, Intelligence and Counter Terrorism, 9(2):151\u2013175, 2014.\n\n[8] Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. 2018\n\nIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6479\u20136488, 2018.\n\n[9] Irtiza Hasan, Francesco Setti, Theodore Tsesmelis, Alessio Del Bue, Marco Cristani, and Fabio Galasso.\n\"seeing is believing\": Pedestrian trajectory forecasting using visual frustum of attention. 2018 IEEE Winter\nConference on Applications of Computer Vision (WACV), pages 1178\u20131185, 2018.\n\n[10] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio\nSavarese. Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE\nConference on Computer Vision and Pattern Recognition, pages 961\u2013971, 2016.\n\n[11] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially\n\nacceptable trajectories with generative adversarial networks. arXiv preprint arXiv:1803.10892, 2018.\n\n[12] Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezato\ufb01ghi, and Silvio Savarese.\nSoPhie: An attentive GAN for predicting paths compliant to social and physical constraints. In CVPR,\n2019.\n\n[13] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel,\nand Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In\nInternational Conference on Machine Learning, pages 2048\u20132057, 2015.\n\n[14] Lamberto Ballan, Francesco Castaldo, Alexandre Alahi, Francesco Palmieri, and Silvio Savarese. Knowl-\nedge transfer for scene-speci\ufb01c motion prediction. In European Conference on Computer Vision, pages\n697\u2013713. Springer, 2016.\n\n[15] Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chan-\n\ndraker. Desire: Distant future prediction in dynamic scenes with interacting agents. 2017.\n\n[16] Amir Sadeghian, Ferdinand Legros, Maxime Voisin, Ricky Vesel, Alexandre Alahi, and Silvio Savarese.\n\nCar-net: Clairvoyant attentive recurrent network. arXiv preprint arXiv:1711.10061, 2017.\n\n[17] Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social etiquette:\nHuman trajectory understanding in crowded scenes. In European conference on computer vision, pages\n549\u2013565. Springer, 2016.\n\n[18] Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert. Activity forecasting. In\n\nEuropean Conference on Computer Vision, pages 201\u2013214. Springer, 2012.\n\n[19] Dirk Helbing and Peter Molnar. Social force model for pedestrian dynamics. Physical review E, 51(5):\n\n4282, 1995.\n\n[20] Stefano Pellegrini, Andreas Ess, and Luc Van Gool.\n\nImproving data association by joint modeling\nof pedestrian trajectories and groupings. In European conference on computer vision, pages 452\u2013465.\nSpringer, 2010.\n\n[21] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron\nCourville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing\nsystems, pages 2672\u20132680, 2014.\n\n[22] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Alejandro Romero, Pietro Li\u00f3, and Yoshua Bengio.\n\nGraph attention networks. CoRR, abs/1710.10903, 2018.\n\n[23] Jun-Yan Zhu, Richard Y. Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, and Eli\n\nShechtman. Toward multimodal image-to-image translation. In NIPS, 2017.\n\n9\n\n\f[24] Alexandre Alahi, Vignesh Ramanathan, and Li Fei-Fei. Socially-aware large-scale crowd forecasting. In\n2014 IEEE Conference on Computer Vision and Pattern Recognition, number EPFL-CONF-230284, pages\n2211\u20132218. IEEE, 2014.\n\n[25] Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. You\u2019ll never walk alone: Modeling\nsocial behavior for multi-target tracking. In Computer Vision, 2009 IEEE 12th International Conference\non, pages 261\u2013268. IEEE, 2009.\n\n[26] Ramin Mehran, Alexis Oyama, and Mubarak Shah. Abnormal crowd behavior detection using social\nforce model. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages\n935\u2013942. IEEE, 2009.\n\n[27] Tharindu Fernando, Simon Denman, Sridha Sridharan, and Clinton Fookes. Soft+ hardwired atten-\ntion: An lstm framework for human trajectory prediction and abnormal event detection. arXiv preprint\narXiv:1702.05552, 2017.\n\n[28] Tharindu Fernando, Simon Denman, Aaron McFadyen, Sridha Sridharan, and Clinton Fookes. Tree\nmemory networks for modelling long-term temporal dependencies. arXiv preprint arXiv:1703.04706,\n2017.\n\n[29] Federico Bartoli, Giuseppe Lisanti, Lamberto Ballan, and Alberto Del Bimbo. Context-aware trajectory\n\nprediction. arXiv preprint arXiv:1705.02503, 2017.\n\n[30] Ronny Hug, Stefan Becker, Wolfgang H\u00fcbner, and Michael Arens. Particle-based pedestrian path prediction\n\nusing lstm-mdl models. arXiv preprint arXiv:1804.05546, 2018.\n\n[31] Javad Amirian, Jean-Bernard Hayet, and Julien Pettr\u00e9. Social ways: Learning multi-modal distributions of\n\npedestrian trajectories with gans. CoRR, abs/1904.09507, 2019.\n\n[32] Thomas N. Kipf and Max Welling. Semi-supervised classi\ufb01cation with graph convolutional networks.\n\nCoRR, abs/1609.02907, 2017.\n\n[33] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional\nadversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages\n5967\u20135976, 2017.\n\n[34] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using\ncycle-consistent adversarial networks. 2017 IEEE International Conference on Computer Vision (ICCV),\npages 2242\u20132251, 2017.\n\n[35] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel.\n\nInfogan:\nInterpretable representation learning by information maximizing generative adversarial nets. In NIPS,\n2016.\n\n[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz\n\nKaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.\n\n[37] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. CoRR, abs/1411.1784, 2014.\n[38] Alon Lerner, Yiorgos Chrysanthou, and Dani Lischinski. Crowds by example. In Computer Graphics\n\nForum, volume 26, pages 655\u2013664. Wiley Online Library, 2007.\n\n10\n\n\f", "award": [], "sourceid": 75, "authors": [{"given_name": "Vineet", "family_name": "Kosaraju", "institution": "Stanford University"}, {"given_name": "Amir", "family_name": "Sadeghian", "institution": "Stanford University"}, {"given_name": "Roberto", "family_name": "Mart\u00edn-Mart\u00edn", "institution": "Stanford University"}, {"given_name": "Ian", "family_name": "Reid", "institution": "University of Adelaide"}, {"given_name": "Hamid", "family_name": "Rezatofighi", "institution": "Stanford University // University of Adelaide"}, {"given_name": "Silvio", "family_name": "Savarese", "institution": "Stanford University"}]}