{"title": "Learning Temporal Point Processes via Reinforcement Learning", "book": "Advances in Neural Information Processing Systems", "page_first": 10781, "page_last": 10791, "abstract": "Social goods, such as healthcare, smart city, and information networks, often produce ordered event data in continuous time. The generative processes of these event data can be very complex, requiring flexible models to capture their dynamics. Temporal point processes offer an elegant framework for modeling event data without discretizing the time. However, the existing maximum-likelihood-estimation (MLE) learning paradigm requires hand-crafting the intensity function beforehand and cannot directly monitor the goodness-of-fit of the estimated model in the process of training. To alleviate the risk of model-misspecification in MLE, we propose to generate samples from the generative model and monitor the quality of the samples in the process of training until the samples and the real data are indistinguishable. We take inspiration from reinforcement learning (RL) and treat the generation of each event as the action taken by a stochastic policy. We parameterize the policy as a flexible recurrent neural network and gradually improve the policy to mimic the observed event distribution. Since the reward function is unknown in this setting, we uncover an analytic and nonparametric form of the reward function using an inverse reinforcement learning formulation. This new RL framework allows us to derive an efficient policy gradient algorithm for learning flexible point process models, and we show that it performs well in both synthetic and real data.", "full_text": "Learning Temporal Point Processes via Reinforcement Learning\n\nShuang Li\u22171, Shuai Xiao 2, Shixiang Zhu1, Nan Du3, Yao Xie1, and Le Song1,2\n\n1Georgia Institute of Technology\n\n2Ant Financial\n3Google Brain\n\nAbstract\n\nSocial goods, such as healthcare, smart city, and information networks, often pro-\nduce ordered event data in continuous time. The generative processes of these event\ndata can be very complex, requiring \ufb02exible models to capture their dynamics.\nTemporal point processes offer an elegant framework for modeling event data with-\nout discretizing the time. However, the existing maximum-likelihood-estimation\n(MLE) learning paradigm requires hand-crafting the intensity function beforehand\nand cannot directly monitor the goodness-of-\ufb01t of the estimated model in the\nprocess of training. To alleviate the risk of model-misspeci\ufb01cation in MLE, we\npropose to generate samples from the generative model and monitor the quality\nof the samples in the process of training until the samples and the real data are\nindistinguishable. We take inspiration from reinforcement learning (RL) and treat\nthe generation of each event as the action taken by a stochastic policy. We param-\neterize the policy as a \ufb02exible recurrent neural network and gradually improve\nthe policy to mimic the observed event distribution. Since the reward function is\nunknown in this setting, we uncover an analytic and nonparametric form of the\nreward function using an inverse reinforcement learning formulation. This new RL\nframework allows us to derive an ef\ufb01cient policy gradient algorithm for learning\n\ufb02exible point process models, and we show that it performs well in both synthetic\nand real data.\n\nIntroduction\n\n1\nMany natural and arti\ufb01cial systems produce a large volume of discrete events occurring in continuous\ntime, for example, the occurrence of crime events, earthquakes, patient visits to hospitals, \ufb01nancial\ntransactions, and user behavior in mobile applications [5]. It is essential to understand and model these\ncomplex and intricate event dynamics so that accurate prediction, recommendation or intervention\ncan be carried out subsequently depending on the context.\nTemporal point processes offer an elegant mathematical framework for modeling the generative\nprocesses of these event data. Typically, parametric (or semi-parametric) assumptions are made on the\nintensity function [11, 9] based on prior knowledge of the processes, and the maximum-likelihood-\nestimation (MLE) is used to \ufb01t the model parameters from data. These models often work well when\nthe parametric assumptions are correct. However, in many cases where the real event generative\nprocess is unknown, these parametric assumptions may be too restricted and do not re\ufb02ect the reality.\nThus there emerge some recent efforts in increasing the expressiveness of the intensity function using\nnonparametric forms [7] and recurrent neural networks [6, 19]. However, these more sophisticated\nmodels still rely on maximizing the likelihood which now involves intractable integrals and needs to\nbe approximated. Most recently, [27] proposed to bypass the problem of maximum likelihood by\nadopting a generative adversarial network (GAN) framework, where a recurrent neural network is\n\u2217Correspondence to: Shuang Li <sli370@gatech.edu>, Yao Xie <yao.xie@isye.gatech.edu>, and Le Song\n\n<lsong@cc.gatech.edu>\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\flearned to transform event sequence from a Poisson process to the target event sequence. However,\nthis approach is rather computationally intensive, since it requires \ufb01tting another recurrent neural\nnetwork as the discriminator, and it takes many iterations and careful tuning for both neural networks\nto reach equilibrium.\nIn this paper, we take a new perspective and establish an under-explored connection between temporal\npoint processes and reinforcement learning: the generation of each event can be treated as the action\ntaken by a stochastic policy, and the intensity function learning problem in temporal point processes\ncan be viewed as the policy learning problem in reinforcement learning.\nMore speci\ufb01cally, we parameterize a stochastic policy\n\u03c0 using a recurrent neural network over event history\nand learn the unknown reward function via inverse re-\ninforcement learning [1, 20, 28, 15]. Our algorithm for\npolicy optimization iterates between learning the reward\nfunction and the stochastic policy \u03c0. Inverse reinforce-\nment learning is known to be time-consuming, which\nrequires solving a reinforcement learning problem in\nevery inner-loop. To tackle this problem, we convert the\ninverse reinforcement learning step to a minimization\nproblem over the discrepancy between the expert point\nprocess and the learner point process. By choosing the\nfunction class of reward to be the unit ball in reproduc-\ning kernel Hilbert space (RKHS) [13, 16, 8], we can get\nan explicit nonparametric closed form for the optimal re-\nward function. Then the stochastic policy can be learned\nby a customized policy gradient with the optimal reward\nfunction having an analytical expression. An illustration\nof our modeling framework is shown in Figure 1.\nWe conducted experiments on various synthetic and real\nsequences of event data and showed that our approach\noutperforms the state-of-the-art regarding both data de-\nscription and computational ef\ufb01ciency.\n2 Preliminaries\nTemporal Point Processes. A temporal point process is a stochastic process whose realization is a\nsequence of discrete events {ti} with ti \u2208 R+ and i \u2208 Z+ abstracted as points on a timeline [5]. Let\nthe history st = {t1, t2, . . . , tn|tn < t} be the sequence of event times up to but not including time t.\nThe intensity function (rate function) \u03bb(t|st) conditioned on the history st uniquely characterizes the\ngenerative process of the events. Different functional forms of \u03bb(t|st)dt capture different generating\npatterns of events. For example, a plain homogeneous Poisson process has \u03bb(t|st) = \u03bb0 (cid:62) 0,\nimplying that each event occurs independently of each and uniformly on the timeline. A Hawkes\nexp(\u2212(t \u2212 ti)) where the occurrences of past events will\nboost future occurrences. Given the intensity function, the survival function de\ufb01ned as S(t|st) =\n\u03bb(\u03c4 )d\u03c4 ) is the conditional probability that no event occurs in the window [tn, t), and\nthe likelihood of observing event at time t is de\ufb01ned as f (t|st) = \u03bb(t|st)S(t|st). Then we can\nexpress the joint likelihood of observing a sequence of events sT = {t1, t2, . . . , tn|tn < T} up to an\nobservation window T as\n\nFigure 1: Illustration of our modeling frame-\nwork. The observed trajectories of events will\nbe viewed as the actions generated by an expert\npolicy \u03c0E. The goal is to learn a policy which\nwe call learner that mimics the distribution\nof the observed expert event sequences. The\nlearner policy \u03c0(a|st) provides the probabil-\nity of the next event occurring at a after t, and\nst := {ti}ti<t is the history of events before\nt. We parametrize \u03c0(a|st) by a recurrent neu-\nral network (RNN) with stochastic neurons [4],\nwhere the generated events are fed back to the\nRNN leading to a double stochastic point pro-\ncess [12]. Furthermore, each generated event\nti will be also associated with a reward r(ti),\nand the policy will be learned by maximizing\nthe expected cumulative rewards [26].\n\nprocess has \u03bb(t|st) = \u03bb0 +(cid:80)\nexp(\u2212(cid:82) t\n\nti\u2208st\n\ntn\n\np({t1, t2, . . . , tn|tn < T}) =\n\n\u03bb(ti|sti) \u00b7 exp\n\n\u2212\n\n\u03bb(\u03c4|s\u03c4 )d\u03c4\n\n(cid:89)\n\nti\u2208sT\n\n(cid:32)\n\n(cid:90) T\n\n0\n\n(cid:33)\n\n.\n\n(1)\n\nThe integral normalization in the likelihood function can be intensive to compute especially in cases\nwhere \u03bb(t|st) do not have a simple form. In this case, a numerical approximation is typically needed\nwhich may affect the accuracy of the \ufb01tting process.\nReproducing Kernel Hilbert Spaces. A reproducing kernel Hilbert space (RKHS) H on T\nwith a kernel k(t, t(cid:48)) is a Hilbert space of functions f (\u00b7) : T (cid:55)\u2192 R with inner product (cid:104)\u00b7,\u00b7(cid:105)H.\nIts element k(t,\u00b7) satis\ufb01es the reproducing property: (cid:104)f (\u00b7), k(t,\u00b7)(cid:105)H = f (t), and consequently,\n(cid:104)k(t,\u00b7), k(t(cid:48),\u00b7)(cid:105)H = k(t, t(cid:48)), meaning that we can view the evaluation of a function f at any point\n\n2\n\n0\tt1t2t3st\u03c0(a|st)0\tt1t2t3t4aPolicy\tGradient\t\u03c0Er(t1)r(t2)r(t3)r(t4)Updatereward\t\fT , we de\ufb01ne the mapping of P to RKHS, \u00b5P := EP[k(t,\u00b7)] = (cid:82)\n\nt \u2208 T as an inner product. Commonly used RKHS kernel function includes Gaussian radial basis\nfunction (RBF) kernel k(t, t(cid:48)) = exp(\u2212(cid:107)t \u2212 t(cid:48)(cid:107)2 /2\u03c32) where \u03c3 > 0 is the kernel bandwidth, and\npolynomial kernel k(t, t(cid:48)) = ((cid:104)t, t(cid:48)(cid:105) + a)d where a > 0 and d \u2208 N [23, 3, 13]. In this paper, if\nnot otherwise stated, we will assume that Gaussian RBF kernel is used. Let P be a measure on\nt\u2208T k(t,\u00b7) dP(t), as the Hilbert\nspace embedding of P [24]. Then for all f \u2208 H, EP[f (t)] = (cid:104)f, \u00b5P(cid:105)H by the reproducing property.\nSimilarly, one can also embed another measure Q on T into RKHS as \u00b5Q. Then a distance between\nmeasure P and Q can be de\ufb01ned as (cid:107)\u00b5P \u2212 \u00b5Q(cid:107)H := sup(cid:107)f(cid:107)H(cid:54)1 (cid:104)f, \u00b5P \u2212 \u00b5Q(cid:105)H. A characteristic\nRKHS is one for which the embedding is injective: that is, each measure has a unique embedding [25],\nand (cid:107)\u00b5P \u2212 \u00b5Q(cid:107)H = 0 if and only if P = Q. This property holds for many commonly used kernels.\nFor T = Rd, this includes the Gaussian kernels.\n3 A Reinforcement Learning Framework\nSuppose we are interested in modeling the daily crime patterns, or monthly occurrences of disease\nfor patients, then the data are collected as trajectories of events within a prede\ufb01ned time window T .\nWe regard the observed paths as actions taken by an expert (nature).\nLet \u03be = {\u03c41, \u03c42, . . . , \u03c4N \u03be\nT is the\ntotal number of events up to T , and it can be different for different sequences. Then, each trajectory\n\u03be \u223c \u03c0E can be seen as an expert demonstration sampled from the expert policy \u03c0E. Hence, on a high\nlevel, given a set of expert demonstrations D = {\u03be1, \u03be2, . . . , \u03bej, . . .|\u03bej \u223c \u03c0E}, we can treat \ufb01tting a\ntemporal point process to D as searching for a learner policy \u03c0\u03b8 which can generate another set of\nsequences \u02dcD = {\u03b71, \u03b72, . . . , \u03b7j, . . .|\u03b7j \u223c \u03c0\u03b8} with similar patterns as D. We will elaborate on this\nreinforcement learning framework below.\nReinforcement Learning Formulation (RL). Given a sequence of past events st = {ti}ti<t, the\nstochastic policy \u03c0\u03b8(a|st) samples an inter-event time a as its action to generate the next event\ntime as ti+1 = ti + a. Then, a reward r(ti+1) is provided and the state st will be updated to\nst = {t1, . . . , ti, ti+1}. Fundamentally, the policy \u03c0\u03b8(a|st) corresponds to the conditional probability\nof the next event time in temporal point process, which in turn uniquely determines the corresponding\n\u03c0\u03b8 (t\u2212ti|sti )\nintensity function as \u03bb\u03b8(t|sti ) =\n\u03c0\u03b8 (\u03c4\u2212ti|sti )d\u03c4 . This builds the connection between the intensity\nti\nfunction in temporal point processes and the stochastic policy in reinforcement learning. If reward\nfunction r(t) is given, the optimal policy \u03c0\u2217\n\n} represent a single trajectory of events from the expert where N \u03be\n\n\u03b8 can be directly computed via\n\n1\u2212(cid:82) t\n\nT\n\n(cid:20)(cid:88)N \u03b7\n\nT\n\ni=1\n\n(cid:21)\n\n,\n\n(2)\nr(ti)\n} is one sampled roll-out from\n\n\u03c0\u2217\n\u03b8 = arg max\n\n\u03c0\u03b8\u2208G J(\u03c0\u03b8) := E\u03b7\u223c\u03c0\u03b8\n\n(cid:18)\n\n(cid:20)(cid:88)N \u03be\n\n(cid:21)\n\n(cid:20)(cid:88)N \u03b7\n\nT\n\n(cid:21)(cid:19)\n\nT can be different for different roll-out samples.\n\nwhere G is the family of all candidate policies \u03c0\u03b8, \u03b7 = {t1, . . . , tN \u03b7\npolicy \u03c0\u03b8, and N \u03b7\nInverse Reinforcement Learning (IRL). Eq.(2) shows that when the reward function is given, the\noptimal policy can be determined by maximizing the expected cumulative reward. However, in our\ncase, only the expert\u2019s sequences of events can be observed, but the real reward function is unknown.\nGiven the expert policy \u03c0E, IRL can help to uncover the optimal reward function r\u2217(t) by\n\nT\n\nT\n\ni=1\n\nr(\u03c4i)\n\n\u2212 max\n\u03c0\u03b8\u2208G\n\nr\u2217 = max\nr\u2208F\n\nE\u03be\u223c\u03c0E\n\nE\u03b7\u223c\u03c0\u03b8\n(3)\n} is one event sequence generated\nwhere F is the family class for reward function, \u03be = {\u03c41, . . . , \u03c4N \u03be\nby the expert \u03c0E, and \u03b7 = {t1, . . . , tN \u03b7\n} is one roll-out sequence from the learner \u03c0\u03b8. The formula-\ntion means that a proper reward function should give the expert policy higher reward than any other\nlearner policy in G, and thus the learner can approach the expert performance by maximizing this\nreward. Denote the procedure (2) and (3) as RL(r) and IRL(\u03c0E), accordingly. The optimal policy\ncan be obtained by\n\nr(ti)\n\ni=1\n\n,\n\nT\n\nT\n\n\u03b8 = RL \u25e6 IRL(\u03c0E).\n\u03c0\u2217\n\n(4)\nOverview of the Proposed Learning Framework. Solving the optimization problem (3) is very\ntime-consuming in that it requires to solve the inner loop RL problem repeatedly. We relieve the\ncomputational challenge by choosing the space of functions F for r(t) to be the unit ball in RKHS\nH, which allows us to obtain an analytical expression for the updated reward function \u02c6r(t) given any\n\n3\n\n\fcurrent learner policy \u02c6\u03c0(\u03b8). This \u02c6r(t) is determined by \ufb01nite sample expert trajectories and \ufb01nite\nsample roll-outs from the current learner policy, and it directly quanti\ufb01es the discrepancy between\nthe expert\u2019s policy (or intensity function) and current learner policy (or intensity function). Then by\nsolving a simple RL problem as in (2), the learner policy can be improved to close its gap with the\nexpert policy using a simple policy gradient type of algorithm.\n\n4 Model\n\nIn this section, we present model parametrization and the analytical expression of optimal reward\nfunction.\nPolicy Network. The function class of the policy \u03c0\u03b8 \u2208 G should be \ufb02exible and expressive enough to\ncapture the potential complex point process patterns of the expert. We, therefore, adopt the recurrent\nneural network (RNN) with stochastic neurons [4] which is \ufb02exible to capture the nonlinear and\nlong-range sequential dependency structure. More speci\ufb01cally,\n\nai \u223c \u03c0(a| \u0398(hi\u22121)), hi = \u03c8(V ai + W hi\u22121), h0 = 0,\n\n(5)\nwhere the hidden state hi \u2208 Rd encodes the sequence of past events {t1, . . . , ti}, ai \u2208 R+, V \u2208 Rd,\nand W \u2208 Rd\u00d7d. Here \u03c8 is a nonlinear activation function applied element-wise, and \u0398 is a nonlinear\nmapping from Rd to the parameter space of the probability distribution \u03c0. For instance, one can\nchoose \u03c8(z) = ez\u2212e\u2212z\nez+e\u2212z to be the tanh function, and design the output layer of \u0398 such that \u0398(hi\u22121)\nis a valid parameter for a probability density function \u03c0. The output ai = ti \u2212 ti\u22121, serves as the\ni-th inter-event time (let t0 = 0), and ai > 0. The choice of model \u03c0 is quite \ufb02exible, only with\nthe constraint that the random variable should be positive since a is always positive. Common\ndistributions such as exponential and Rayleigh distributions would satisfy such constraint, leading\nto \u03c0(a|\u0398(hi\u22121)) = \u0398(h)e\u2212\u0398(h)a and \u03c0(a|\u0398(hi\u22121)) = \u0398(h)ae\u2212\u0398(h)a2/2 respectively. In this way,\nwe specify a nonlinear and \ufb02exible dependency over the history.\nThe architecture of our model in (5) is shown in Fig-\nure 2. Different from traditional RNN, the outputs\nai are sampled from \u03c0 rather than obtained by deter-\nministic transformations. This is what \u201cstochastic\"\npolicy means. Randomly sampling will allow the\npolicy to explore the temporary space. Furthermore,\nthe sampled time point will be fed back to the RNN.\nThe proposed model aims to capture that the state hi\nis attributed by two parts. One is the deterministic in\ufb02uence from the previous hidden state hi\u22121, and\nthe other is the stochastic in\ufb02uence from the latest sampled action ai. Action ai is sampled from the\nprevious distribution \u03c0(a|\u0398(hi\u22121)) with parameter \u0398(hi\u22121) and will be fed back to in\ufb02uence the\ncurrent hidden state hi.\nIn some sense, our RNN with stochastic neurons mimics the event generating mechanism of the\ndoubly stochastic point process, such as Hawkes process and self-correcting process. For these types\nof point processes, the intensity is stochastic, which depends on history, and the intensity function\nwill control the occurrence rate of the next event.\nReward Function Class. The reward function directly quanti\ufb01es the discrepancy between \u03c0E and\n\u03c0\u03b8, and it guides the learning of the optimal policy \u03c0\u2217\n\u03b8. On the one hand, we want its function class\nr \u2208 F to be suf\ufb01ciently \ufb02exible so that it can represent the reward function of various shapes. On\nthe other hand, it should be restrictive enough to be ef\ufb01ciently learned with \ufb01nite samples [3, 13].\nWith these competing considerations, we choose F to be the unit ball in RKHS H, (cid:107)r(cid:107)H (cid:54) 1. An\nimmediate bene\ufb01t of this function class is that we can show the optimal policy can be directly learned\nvia a minimization formulation given in Theorem 1 instead of the original minimax formulation (3).\nA sketch of proof is provided as follows. For short notation, we denote\n\nFigure 2: Illustration of generator \u03c0\u03b8.\n\nk(t,\u00b7)dN (\u03b7)\n\n,\n\nt\n\nand\n\n\u00b5\u03c0\u03b8 := E\u03b7\u223c\u03c0\u03b8 [\u03c6(\u03b7)]\n\n(cid:124)\n\n(cid:123)(cid:122)\n\n(cid:125)\n\nfeature mapping from data space to R\n\nmean embeddings of the intensity function in RKHS\n\n(cid:90)\n\n(cid:123)(cid:122)\n\n[0,T )\n\n\u03c6(\u03b7) :=\n\n(cid:124)\n\n(cid:125)\n\n4\n\nhi\u22121a1h0h1ai\u22121hiaihi+1ai+1\u2026\u2026anhnan+10T\u2026\u2026t1ti\u22121titi+1tn\f\uf8ee\uf8f0N (\u03b7)\nT(cid:88)\n\n\uf8f9\uf8fb = E\u03b7\u223c\u03c0\u03b8\n\n(cid:34)(cid:90)\n\n(cid:35)\n\nwhere dN (\u03b7)\nkernel. Then using the reproducing property,\n\nt\n\nis the counting process associated with sample path \u03b7, and k(t, t(cid:48)) is a universal RKHS\n\nJ(\u03c0\u03b8) := E\u03b7\u223c\u03c0\u03b8\n\n(cid:104)r, k (t,\u00b7)(cid:105)HdN (\u03b7)\nSimilarly, we can obtain J(\u03c0E) = (cid:104)r, \u00b5\u03c0E(cid:105)H. From (3), r\u2217 is obtained by\n\nr(ti)\n\n[0,T )\n\ni=1\n\nt\n\n= (cid:104)r, \u00b5\u03c0\u03b8(cid:105)H.\n\nmax\n(cid:107)r(cid:107)H\u22641\n\n\u03c0\u03b8\u2208G (cid:104)r, \u00b5\u03c0E \u2212 \u00b5\u03c0\u03b8(cid:105)H = min\n\nmin\n\n(cid:104)r, \u00b5\u03c0E \u2212 \u00b5\u03c0\u03b8(cid:105)H = min\n\n\u03c0\u03b8\u2208G (cid:107)\u00b5\u03c0E \u2212 \u00b5\u03c0\u03b8(cid:107)H,\n\nwhere the \ufb01rst equality is guaranteed by the minimax theorem, and\n\nr\u2217(\u00b7|\u03c0E, \u03c0\u03b8) =\n\n\u221d \u00b5\u03c0E \u2212 \u00b5\u03c0\u03b8\n\n(6)\n\n\u03c0\u03b8\u2208G max\n(cid:107)r(cid:107)H\u22641\n\u00b5\u03c0E \u2212 \u00b5\u03c0\u03b8\n(cid:107)\u00b5\u03c0E \u2212 \u00b5\u03c0\u03b8(cid:107)H\n\ncan be empirically evaluated by data. In this way, we change the original minimax formulation for\nsolving \u03c0\u2217\n\u03b8 to a simple minimization problem, which will be more ef\ufb01cient and stable to solve in\npractice. We summarize the formulation in Theorem 1.\nTheorem 1 Let the family of reward function be the unit ball in RKHS H, i.e., (cid:107)r(cid:107)H (cid:54) 1. Then the\noptimal policy obtained by (4) can also be obtained by solving\n\n(7)\nwhere D(\u03c0E, \u03c0\u03b8,H) is the maximum expected cumulative reward discrepancy between \u03c0E and \u03c0\u03b8,\n\n\u03c0\u2217\n\u03b8 = arg min\n\n\u03c0\u03b8\u2208G D(\u03c0E, \u03c0\u03b8,H)\n(cid:20)(cid:88)N (\u03be)\n(cid:21)\n\nT\n\nr(\u03c4i)\n\ni=1\n\n\u2212 E\u03b7\u223c\u03c0\u03b8\n\n(cid:20)(cid:88)N (\u03b7)\n\nT\n\ni=1\n\nD(\u03c0E, \u03c0\u03b8,H) := max\n(cid:107)r(cid:107)H(cid:54)1\n\nE\u03be\u223c\u03c0E\n\nr(ti)\n\n.\n\n(8)\n\n(cid:21)(cid:19)\n\n(cid:18)\n\nTheorem 1 implies that we can transform the inverse reinforcement learning procedure of (4) to a\nsimple minimization problem which minimizes the maximum expected cumulative reward discrep-\nancy between \u03c0E and \u03c0\u03b8. This enables us to sidestep the expensive computation of (4) caused by\nthe solving the inner RL problem repeatedly. What\u2019s more interesting, we can derive an analytical\nsolution to (8) given by (6).\nFinite Sample Estimation. Given L trajectories of expert point processes, and M trajectories\nof events generated by \u03c0\u03b8, mean embeddings \u00b5\u03c0E and \u00b5\u03c0\u03b8 can be estimated by their respective\n,\u00b7). Then for\nempirical mean: \u02c6\u00b5\u03c0E = 1\nany t \u2208 [0, T ), the estimated optimal reward is (without normalization) is\nL\n\n(cid:80)N (m)\n\ni=1 k(t(m)\n\ni=1 k(\u03c4 (l)\n\nl=1\n\nM\n\nT\n\nT\n\ni\n\ni\n\n\u02c6r\u2217(t) \u221d 1\nL\n\nk(\u03c4 (l)\n\ni\n\n, t) \u2212 1\nM\nand t(m)\n. Unbiased estimator can also be obtained and\n\nk(t(m)\n\n, t).\n\n(9)\n\nm=1\n\ni=1\n\ni\n\ni\n\nNote this empirical estimator is biased at \u03c4 (l)\nwill be provided in Algorithm RLPP discussed later for simplicity.\nKernel Choice. The unit ball in RKHS is dense and expressive. Fundamentally, our proposed\nframework and theoretical results are general and can be directly applied to other types of kernels.\nFor example, we can use the Mat\u00e9rn kernel, which generates spaces of differentiable functions known\nas the Sobolev spaces [10, 2]. In later experiments, we have used Gaussian kernel and obtained\npromising results.\n\ni\n\n(cid:80)L\n(cid:88)L\n\n(cid:80)N (l)\n(cid:88)N (l)\n\nT\n\nl=1\n\ni=1\n\n,\u00b7) and \u02c6\u00b5\u03c0\u03b8 = 1\n(cid:88)M\n\n(cid:80)M\n(cid:88)N (m)\n\nm=1\n\nT\n\n5 Learning Algorithm\nLearning via Policy Gradient. In practice, instead of minimizing D(\u03c0E, \u03c0\u03b8,H) as in (7), we can\nequivalently minimize D(\u03c0E, \u03c0\u03b8,H)2 since square is a monotonic transformation. Now, we can\nlearn \u03c0\u2217\n\u03b8 from the RL formulation (2) using policy gradient with variance reduction. First, with the\nlikelihood ratio trick, the gradient of \u2207\u03b8D(\u03c0E, \u03c0\u03b8,H)2 can be computed as\n\n(cid:18)(cid:88)N \u03b7\n\nT\n\n\u02c6r\u2217(ti)\n\ni=1\n\n(cid:19)\uf8f9\uf8fb ,\n\n(10)\n\n\u2207\u03b8D(\u03c0E, \u03c0\u03b8,H)2 = E\u03b7\u223c\u03c0\u03b8\n\n(\u2207\u03b8 log \u03c0\u03b8(ai|\u0398(hi\u22121))) \u00b7\n\nwhere (cid:80)N \u03b7\n\nT\n\ni=1 (\u2207\u03b8 log \u03c0\u03b8(ai|\u0398(hi\u22121))) is the gradient of the log-likelihood of a roll-out sample\n\n\u03b7 = {t1, . . . , tN \u03b7\n\n} using the learner policy \u03c0\u03b8.\n\nT\n\n\uf8ee\uf8f0 N \u03b7\nT(cid:88)\n\ni=1\n\n5\n\n\fAlgorithm RLPP: Mini-batch Reinforcement Learning\nfor Learning Point Processes\n1. Initialize model parameters \u03b8;\n2. For number of training iterations do\n\n1\n\n};\n\n};\n\nN\n\n(l)\nT\n\n, . . . , t(m)\nNT\n\n1 , . . . , \u03c4 (l)\n\n(cid:88)M\n\n(cid:18)(cid:88)N\n\n\u2022 Sample minibatch of L trajectories of events\n{\u03be(1), . . . , \u03be(L)} from expert policy \u03c0E, where\n\u03be(l) = {\u03c4 (l)\n\u2022 Sample minibatch of M trajectories of events\n{\u03b7(1), . . . , \u03b7(M )} from learner policy \u03c0\u03b8, where\n\u03b7(m) = {t(m)\n(cid:19)\n\u2022 Estimate policy gradient \u2207\u03b8D(\u03c0E, \u03c0\u03b8,H)2 as\nwhere log p\u03b8(\u03b7(m)) =(cid:80)N \u03b7\n\u2207\u03b8\n\u2217\n) log p\u03b8(\u03b7(m))\n\u02c6r\ni=1 (log \u03c0\u03b8(ai|\u0398(hi\u22121)))\nis the log-likelihood of the sample \u03b7(m), and\nr\u2217(t(m)\n) can be estimated by L expert trajectories\n(cid:88)N\nand (M \u2212 1) roll-out samples without \u03b7(m)\n\u2217\nM(cid:88)\n\u02c6r\n\n(cid:88)N\n\n(cid:88)L\n\nk(\u03c4 (l)\ni\n(m(cid:48) )\n\n(t(m)) =\n\n1\nM\n\n(t(m)\n\ni\n\n(m)\nT\n\nl=1\n\ni=1\n\nm=1\n\ni=1\n\n, t)\nk(t(m(cid:48))\n\nj\n\n, t);\n\n1\nL\n\u2212 1\n\nM \u2212 1\n\nT\n\n(l)\nT\n\ni\n\n(a)\n\n(b)\n\nFigure 3: The reward function \u02c6r\u2217(t) is estimated us-\ning 100 sampled sequences from \u03c0E and \u03c0\u03b8. In (a),\n\u02c6r\u2217(t) > 0 when the expert\u2019s intensity is above the\nlearner\u2019s intensity, and \u02c6r\u2217(t) < 0 when the expert\u2019s\nintensity is below the learner\u2019s intensity. In order to\nmaximize the cumulative reward given the current\nreward, the learner should generate more events in\nthe region when \u02c6r\u2217(t) > 0 and reduce the number\nof events when \u02c6r\u2217(t) < 0. Based on our formula-\ntion, the optimal reward function always quanti\ufb01es the\ndiscrepancy between the expert and current learner\nby considering the worst case. As a result, once the\nlearner is changed, the current optimal reward \u02c6r\u2217(t)\nis updated accordingly, and \u02c6r\u2217(t) guides the learner\nto update its policy towards mimicking the expert\u2019s\nbehavior until they exactly match each other in (b)\nwhere \u02c6r\u2217(t) becomes zero.\n\nT\n\nj=1\n\nm(cid:48)=1,m(cid:48)(cid:54)=m\n\u2022 Update policy parameters as\n\n\u03b8 \u2190 \u03b8 + \u03b1\u2207\u03b8D(\u03c0E, \u03c0\u03b8,H)2.\n\nT\n\nT\n\nwhere\n\n(cid:16)(cid:80)NT\n\n(cid:17)\nl=i \u02c6r\u2217(tl)\n\n(cid:17)(cid:105)\nl=i [\u02c6r\u2217(tl) \u2212 bl]\n\ni=1 (\u2207\u03b8 log \u03c0\u03b8(ai|\u0398(hi\u22121))) \u00b7(cid:16)(cid:80)N \u03b7\n(cid:104)(cid:80)N \u03b7\n\nTo reduce the variance of the gradient, we can exploit the observation that future actions do not\ndepend on past rewards. This leads to a variance reduced gradient estimate \u2207\u03b8D(\u03c0E, \u03c0\u03b8,H)2 =\nE\u03b7\u223c\u03c0\u03b8\nis referred to\nas the \u201creward to go\" and bl is the baseline to further reduce the variance. The overall procedure is\ngiven in Algorithm RLPP. In the algorithm, after we sample M trajectories from the current policy,\nwe use one trajectory \u03b7m for evaluation and the rest M \u2212 1 samples to estimate reward function. An\nexample reward function learned at a different stage of the algorithm is also illustrated in Figure 3.\nComparison with MLE. During training, our generative model directly compares the generated\ntemporal events with the observed events to iteratively correct the mistakes, which can effectively\navoid model misspeci\ufb01cation. Since the training only involves the policy gradient, it bypasses the\nintractability issue of the log-survival term in the likelihood (Eq. (1)). On the other hand, because the\nlearned policy is in fact the conditional density of a point process, our approach still resembles the\nform of MLE in the RL reformulation and can thus be interpreted in a statistically principled way.\nComparison with GAN and GAIL. By Theorem 1, our policy is learned directly by minimizing the\ndiscrepancy between \u03c0E and \u03c0\u03b8 which has a closed form expression. Thus, we convert the original\nIRL problem to a minimization problem with only one set of parameters with respect to the policy.\nIn each training iteration with the policy gradient, we have an unbiased estimator of the gradient,\nand the estimated reward function also depends on the current policy \u03c0\u03b8. In contrast, in GAN or\nGAIL formulation, they have two sets of parameters related to the generator and the discriminator.\nThe gradient estimator is biased because each min-/max-problem is in fact nonconvex and cannot be\nsolved in one-shot. Thus, our framework is more stable and ef\ufb01cient than the mini-max formulation\nfor learning point processes.\n6 Experiments\nWe evaluate our algorithm by comparing with state-of-the-arts on both synthetic and real datasets.\nSynthetic datasets. To show the robustness to model-misspeci\ufb01cations of our approach, we propose\nthe following four different point processes as the ground-truth: (I) Inhomogeneous Poisson (IP)\nwith \u03bb(t) = at + b where a = \u22120.2 and b = 3.5; Here we omit st since \u03bb(t) does not depend on\nti<t exp{\u2212(t \u2212 ti)} where \u00b5 = 2,\nand \u03b1 = 0.5. (III) Mixture of IP and HP version 1 (IP + HP1). For the IP component, its \u03bb(t)\n\nthe history. (II) Hawkes Process (HP) with \u03bb(t|st) = \u00b5 + \u03b1(cid:80)\n\n6\n\n05101520Time-15-10-5051015\u02c6r\u2217(t)\u03bb\u03c0E\u03bb\u03c0\u03b805101520Time-15-10-5051015\u02c6r\u2217(t)\u03bb\u03c0E\u03bb\u03c0\u03b8\f(a) IP\n\n(b) HP\n\n(c) IP + HP1\n\n(d) IP+HP2\n\nFigure 4: Comparison of empirical intensity functions on the synthetic data.\n\nWe select one beat zone data with call timestamps ranging from 7:00 AM to 1:00 PM.\n\nis piece-wise linear with monotonic increasing slopes of pieces from {0.2, 0.3, 0.4, 0.5}. The HP\ncomponent has the parameter \u00b5 = 1 and \u03b1 = 0.5; (IV) Mixture of IP and HP version 2 (IP + HP2)\nwhere the IP component also has piece-wise linear intensity but the slopes have the zig-zag pattern\nchosen from {1,\u22121, 2,\u22122}, and the HP component has the parameter \u00b5 = 1 and \u03b1 = 0.1.\nReal datasets. We evaluate our approach on four real datasets across a diverse range of domains:\n\u2022 911 call dataset contains 220,000 crime incident call records from 2011 to 2017 in Atlanta area.\n\u2022 Microsoft Academic Search (MAS) provides access to publication venues, time, citations, etc. We\n\u2022 Medical Information Mart for Intensive Care III (MIMIC-III) contains de-identi\ufb01ed clinical visit\nrecords from 2001 to 2012 for more than 40,000 patients. Our data contain 2,246 patients with at\nleast 3 visits. For a given patient, each clinical visit will be treated as an event.\n\u2022 NYSE contains 0.7 million high-frequency trading records from NYSE for a given stock within\none day. All transactions are evenly divided into 3,200 segments. All segments have the same\ntemporal duration. Each trading record is treated as a event.\n\ncollect citation records for 50,000 papers and treat each citation time as an event.\n\nBaselines. We compare our approach against two state-of-the-arts as well as conventional paramet-\nric baselines. The two state-of-the-art methods are WGANTPP [27] and RMTPP2 [6]. In addition,\nthree parametric methods based on maximum likelihood estimation are compared, including: (1)\nInhomogeneous Poisson process where the intensity function is modeled using a mixture of Gaussian\ncomponents, (2) Hawkes Process (or Self-Excitation process denoted as SE), and (3) Self-Correcting\n\nti<t \u03b1(cid:9). In contrast to Hawkes process, the self-correcting\n\nprocess (SC) with \u03bb(t|st) = exp(cid:8)\u00b5t \u2212(cid:80)\n\nprocess seeks to produce regular point patterns. The intuition is that while the intensity increases\nsteadily, every time when a new event appears, it is decreased by multiplying a constant e\u2212\u03b1 < 1, so\nthe chance of new points decreases after an event has occurred recently.\n\nExperimental Setup. The policy in our method RLPP is parameterized as LSTM with 64 hidden\nneurons, and \u03c0(a|\u0398(h)) is chosen to be exponential distribution. Batch size is 32 (the number of\nsampled sequences L and M are 32 in Algorithm 1, and learning rate is 1e-3. We use Gaussian\nkernel k(t, t(cid:48)) = exp(\u2212(cid:107)t \u2212 t(cid:48)(cid:107)2/\u03c32) for the reward function. The kernel bandwidth \u03c3 is estimated\nusing the \u201cmedian trick\u201d based on the observations [13]. For WGANTPP and RMTPP, we are using\nthe open source codes. For WGANTPP3, we have used the exact experimental setup as [27], which\nadopts Adam optimization method [17] with learning rate 1e-4, \u03b21 = 0.5, \u03b22 = 0.9, and the batch\nsize is 256. For RMTPP4, batch size is 256, state size is 64, and learning rate is 1e-4.\n\nComparison of Learned Empirical Intensity. We \ufb01rst compare the empirical intensity of the\nlearner point process to the expert point process. This is a straightforward comparison: one can\nvisually assess the performance and localize the discrepancy. Fig. 4 and Fig. 5 demonstrate the\nempirical intensity functions of generated sequences based on synthetic and real data. It clearly shows\nthat RLPP consistently outperforms RMTPP, and achieves comparable and sometimes even better\n\ufb01tting against WGANTPP. Furthermore, RLPP consistently outperforms the other three conventional\nparametric models when there exist model-misspeci\ufb01cations. Without any prior knowledge, RLPP\ncan capture the major trends in data and can accurately learn the nonlinear dependency structure\n\n2RMTPP has very similar performance with [19].\n3https://github.com/xiaoshuai09/Wasserstein-Learning-For-Point-Process\n4https://github.com/dunan/NeuralPointProcess\n\n7\n\n012344812Time IndexIntensityMethodsRealRLPPWGANIPSERMTPPSC02464812Time IndexIntensityMethodsRealRLPPWGANIPSERMTPPSC05104812Time IndexIntensityMethodsRealRLPPWGANIPSERMTPPSC0.02.55.07.54812Time IndexIntensityMethodsRealRLPPWGANIPSERMTPPSC\f(a) 911 Call\n\n(b) MAS\n\n(c) MIMIC III\n\n(d) NYSE\n\nFigure 5: Comparison of empirical intensity functions on the real datasets. For each dataset, we have\nused all learned models to generate new sequences. The comparisons are based on the empirical\nintensities estimated from the generated temporal events and those estimated from the observed\ntemporal events.\n\nhidden in data. In the Hawkes example, RLPP performs even as accurate as the ground-truth model.\nOn the real-world data, the underlying true model is unknown and the point process patterns are more\ncomplicated. RLPP still shows a decent performance in the real datasets.\n\nti\u22121\n\nintensity \u03bb(t), then the respective value achieved from the integral \u039b =(cid:82) ti\n\nComparison of Data Fitting. Quantile plot (QQ-plot) for residual analysis is a standard model\nchecking approach for general point processes. Given a set of real input samples t1, . . . , tn, by\nthe Time Changing Theorem [5], if such set of samples is one realization of a process with the\n\u03bb(t)dt should conform\nto the unit-rate exponential distribution [18]. For the synthetic experiments, since we know the\nexact ground-truth parametric form of \u03bb(t|st), we can perform this explicit transformation for a test.\nIdeally, the QQ-plot for the generated sequences should follow a 45-degree straight line. We use\nHawkes Process (HP) and Inhomogeneous Poisson Process + Hawkes Process (IP+HP1) dataset to\nproduce the QQ-plot and compare different methods in Fig. 6. In both cases, RLPP consistently\nstands out even without any prior knowledge about the parametric form of the true underlying\ngenerative point process and the \ufb01tting slope is very close to the diagonal line in both cases. More\nrigorously, we perform the KS test. Fig. 7 illustrates the cumulative distributions (CDF) of p-values.\nWe followed the experiment setup in [21]: we generated samples from each learned point process\nmodels, transformed the time interval, and applied the KS test to compare with unit rate exponential\ndistribution. Under this null hypothesis, the distribution of the p-values over tests should follow a\nuniform distribution, whose CDF should be a diagonal line. If the target distribution is the Hawkes\nprocess (Fig. 7), both the learned SE (Hawkes process) and the RLPP models are indistinguishable\nfrom that.\n\nFigure 6: QQ-plot for dataset HP (left) and HP+IP1 (right).\n\nFigure 7: KS test results: CDF of p-values.\nComparison of Runtime. The runtime for all methods averaged on all datasets is shown in Table\n1. We note that both RLPP and WGANTPP are written in Tensor\ufb02ow. However, WGANTPP adopts\nthe adversarial training framework based on Wasserstein divergence, where both the generator and\nthe discriminator are modeled as LSTMS. In contrast, RLPP only models the policy as a single\nLSTM with the reward function learned in an analytical form. As a consequence, RLPP requires less\nparameters and is more simpler to train while at the same time achieving comparable or even better\nperformance.\n\nTable 1: Comparison of runtime.\n\nMethod RLPP WGANTPP RMTPP\nTime\nRatio\n\n1560m\n780x\n\n60m\n30x\n\n80m\n40x\n\nSE\nSC IP\n2m 2m 2m\n1x\n1x\n\n1x\n\n8\n\n0.000.250.500.751.004812Time IndexIntensityMethodsRealRLPPWGANIPSERMTPPSC02464812Time IndexIntensityMethodsRealRLPPWGANIPSERMTPPSC012344812Time IndexIntensityMethodsRealRLPPWGANIPSERMTPPSC0.00.51.01.52.02.54812Time IndexIntensityMethodsRealRLPPWGANIPSERMTPPSC0246810Theoretical quantiles0246810Ordered ValuesProbability PlotRealIPSESCRMTPPWGANRLPP0246810Theoretical quantiles0246810Ordered ValuesProbability PlotRealIPSESCRMTPPWGANRLPP0.00.20.40.60.81.0p-value0.00.20.40.60.81.0cumulative distributionKS test for dataset HPRealRLRMTPPSESCIPWGAN\f(a): 911 dataset\n\nComparisons to LGCP and non-parametric\nHawkes. We also compared RLPP to log-\nGaussian Cox process (LGCP) model and non-\nparametric Hawkes with non-stationary back-\nground rate (Nonpar Hawkes) model regarding\nlearned empirical intensity function. Represen-\ntative comparison results are showed in Fig. 8.\nOur proposed method (RL) performs similarly\nto LGCP and outperforms Nonpar Hawkes on\nFigure 8: Comparison of empirical intensity functions.\nreal datasets. However, LGCP needs to dis-\ncretize time into windows and aggregate event into counts. This leads to some information loss and\nintroduces additional tuning parameters. Moreover, the standard LGCP is not scalable, typically\nrequiring O(n3) in computation and O(n2) in storage (n = sequence # \u00d7 window #). We used an\nimplementation in GPy package5, which requires 50% more time than our method (127 mins vs\n80 mins) in processing 5% of the dataset. The nonparametric Hawkes model is parametrized by\nweighted sum of basis functions, similar to that of the inhomogeneous Poisson process baseline, and\nit is dif\ufb01cult to generalize outside the observation window.\n7 Discussions\n1. RMTPP we compared in experiments is a state-of-the-art maximum-likelihood-based model,\nwhich uses a similar RNN outputting parametrization of exponential distributions but \ufb01ts the\nmodel parameters with maximum likelihood. Across our experiments over eight synthetic and\nreal-world datasets, our proposed method performs consistently better than the MLE.\n\n(b): MIMIC dataset\n\n2. In theory, although MLE has many attractive limiting properties, it has no optimum properties for\n\ufb01nite samples, in the sense that (when evaluated on \ufb01nite samples) other estimators may provide a\nbetter estimate for the true parameters, e.g. [22]. Likelihood is related to KL divergence. Since KL\ndivergence is asymmetric and has a number of drawbacks for \ufb01nite sample (such as high variance\nand mode dropping), many other divergences have been proposed and shown to perform better in\nthe \ufb01nite sample case, e.g. [14]. Our proposed discrepancy is inspired by a similar use of RKHS\ndiscrepancy in two sample tests in [14]. RKHS discrepancy has been shown to perform nicely on\n\ufb01nite sample and also preserve the asymptotic properties.\n\n3. Another potential bene\ufb01t of our proposed framework is that one may use the RNN to de\ufb01ne a\ntransformation for the temporal random variable instead of de\ufb01ning its output distribution. For\nexample, we can establish our policy as a transformation of a sample from a unit rate exponential\ndistribution. The same empirical objective in Eq. (8) will be used, but a different optimization\nalgorithm is needed. Since no explicit parameterization of the output distribution is needed, this\nmay lead to even more \ufb02exible models and this is left for future investigation.\n\n8 Conclusions\nThis paper proposes a reinforcement learning framework to learn point process models. We\nparametrized our policy as RNNs with stochastic neurons, which can sequentially sample dis-\ncrete events. The policy is updated by directly minimizing the discrepancy between the generated\nsequences with the observed sequences, which can avoid model misspeci\ufb01cation and the limitation\nof likelihood based approach. Furthermore, the discrepancy is explicitly evaluated in terms of the\nreward function in our setting. By choosing the function class of reward to be the unit ball in RKHS,\nwe successfully derived an analytical optimal reward which maximizes the discrepancy. The optimal\nreward will iteratively encourage the policy to sample events as close as the observation. We show\nthat our proposed approach performs well on both synthetic and real data.\nAcknowledgments\nThis project was supported in part by NSF grants CCF-1442635, CMMI-1538746, DMS-1830210, NSF\nCAREER Award CCF-1650913, Atlanta Police Foundation fund, and an S.F. Express fund awarded to Yao\nXie. This project was supported in part by NSF IIS-1218749, NIH BIGDATA 1R01GM108341, NSF CAREER\nIIS-1350983, NSF IIS-1639792 EAGER, NSF CNS-1704701, ONR N00014-15-1-2340, Intel ISTC, NVIDIA\nand Amazon AWS, NSF CCF-1836822, NSF IIS-1841351 EAGER, and Siemens awarded to Le Song.\n\n5https://github.com/SheffieldML/GPy\n\n9\n\n051015t00.51IntensityReal DataRLNonpar HawkesLGCP51015t02468IntensityReal DataRLNonpar HawkesLGCP\fReferences\n[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning.\nIn Proceedings of the twenty-\ufb01rst international conference on Machine learning, page 1. ACM,\n2004.\n\n[2] Robert A Adams and John JF Fournier. Sobolev spaces, volume 140. Academic press, 2003.\n[3] Alain Berlinet and Christine Thomas-Agnan. Reproducing kernel Hilbert spaces in probability\n\nand statistics. Springer Science & Business Media, 2011.\n\n[4] Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua\nBengio. A recurrent latent variable model for sequential data. In Advances in neural information\nprocessing systems, pages 2980\u20132988, 2015.\n\n[5] Daryl J Daley and David Vere-Jones. An introduction to the theory of point processes: general\n\ntheory and structure. Springer Science & Business Media, 2007.\n\n[6] Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and\nLe Song. Recurrent marked temporal point processes: Embedding event history to vector. In\nProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and\nData Mining, pages 1555\u20131564. ACM, 2016.\n\n[7] Nan Du, Le Song, Manuel Gomez Rodriguez, and Hongyuan Zha. Scalable in\ufb02uence estimation\nin continuous-time diffusion networks. In Advances in neural information processing systems,\npages 3147\u20133155, 2013.\n\n[8] Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural\nnetworks via maximum mean discrepancy optimization. arXiv preprint arXiv:1505.03906,\n2015.\n\n[9] Mehrdad Farajtabar, Yichen Wang, Manuel Gomez Rodriguez, Shuang Li, Hongyuan Zha,\nand Le Song. Coevolve: A joint point process model for information diffusion and network\nco-evolution. In Advances in Neural Information Processing Systems, pages 1954\u20131962, 2015.\n[10] Marc G Genton. Classes of kernels for machine learning: a statistics perspective. Journal of\n\nmachine learning research, 2(Dec):299\u2013312, 2001.\n\n[11] Manuel Gomez Rodriguez, Jure Leskovec, and Andreas Krause. Inferring networks of diffusion\nand in\ufb02uence. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge\ndiscovery and data mining, pages 1019\u20131028. ACM, 2010.\n\n[12] Jan Grandell. Doubly stochastic Poisson processes, volume 529. Springer, 2006.\n\n[13] Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Sch\u00f6lkopf, and Alex J Smola.\nA kernel method for the two-sample-problem. In Advances in neural information processing\nsystems, pages 513\u2013520, 2007.\n\n[14] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Sch\u00f6lkopf, and Alexander\nSmola. A kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723\u2013773,\n2012.\n\n[15] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in\n\nNeural Information Processing Systems, pages 4565\u20134573, 2016.\n\n[16] Beomjoon Kim and Joelle Pineau. Maximum mean discrepancy imitation learning. In Robotics:\n\nScience and systems, 2013.\n\n[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint\n\narXiv:1412.6980, 2014.\n\n[18] John Frank Charles Kingman. Poisson processes. Wiley Online Library, 1993.\n\n[19] Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self-modulating\nmultivariate point process. In Advances in Neural Information Processing Systems, pages\n6757\u20136767, 2017.\n\n10\n\n\f[20] Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml,\n\npages 663\u2013670, 2000.\n\n[21] Takahiro Omi, Yoshito Hirata, and Kazuyuki Aihara. Hawkes process model with a time-\ndependent background rate and its application to high-frequency \ufb01nancial data. Physical Review\nE, 96(1):012303, 2017.\n\n[22] Johann Pfanzagl. Parametric statistical theory. Walter de Gruyter, 2011.\n\n[23] Bernhard Scholkopf and Alexander J Smola. Learning with kernels: support vector machines,\n\nregularization, optimization, and beyond. MIT press, 2001.\n\n[24] Alex Smola, Arthur Gretton, Le Song, and Bernhard Sch\u00f6lkopf. A hilbert space embedding\nfor distributions. In International Conference on Algorithmic Learning Theory, pages 13\u201331.\nSpringer, 2007.\n\n[25] Bharath Sriperumbudur, Kenji Fukumizu, and Gert Lanckriet. On the relation between univer-\nsality, characteristic kernels and rkhs embedding of measures. In Proceedings of the Thirteenth\nInternational Conference on Arti\ufb01cial Intelligence and Statistics, pages 773\u2013780, 2010.\n\n[26] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1.\n\nMIT press Cambridge, 1998.\n\n[27] Shuai Xiao, Mehrdad Farajtabar, Xiaojing Ye, Junchi Yan, Xiaokang Yang, Le Song, and\nHongyuan Zha. Wasserstein learning of deep generative point process models. In Advances in\nNeural Information Processing Systems, pages 3250\u20133259, 2017.\n\n[28] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy\ninverse reinforcement learning. In AAAI, volume 8, pages 1433\u20131438. Chicago, IL, USA, 2008.\n\n11\n\n\f", "award": [], "sourceid": 6874, "authors": [{"given_name": "Shuang", "family_name": "Li", "institution": "Georgia Institute of Technology"}, {"given_name": "Shuai", "family_name": "Xiao", "institution": "Ant Financial"}, {"given_name": "Shixiang", "family_name": "Zhu", "institution": "Georgia Institute of Technology"}, {"given_name": "Nan", "family_name": "Du", "institution": "Google Brain"}, {"given_name": "Yao", "family_name": "Xie", "institution": "Georgia Institute of Technology"}, {"given_name": "Le", "family_name": "Song", "institution": "Ant Financial & Georgia Institute of Technology"}]}