{"title": "Logistic Normal Priors for Unsupervised Probabilistic Grammar Induction", "book": "Advances in Neural Information Processing Systems", "page_first": 321, "page_last": 328, "abstract": null, "full_text": "Logistic Normal Priors for Unsup ervised Probabilistic Grammar Induction\nShay B. Cohen Kevin Gimp el Noah A. Smith Language Technologies Institute School of Computer Science Carnegie Mellon University {scohen,kgimpel,nasmith}@cs.cmu.edu\n\nAbstract\nWe explore a new Bayesian model for probabilistic grammars, a family of distributions over discrete structures that includes hidden Markov models and probabilistic context-free grammars. Our model extends the correlated topic model framework to probabilistic grammars, exploiting the logistic normal distribution as a prior over the grammar parameters. We derive a variational EM algorithm for that model, and then experiment with the task of unsupervised grammar induction for natural language dependency parsing. We show that our model achieves superior results over previous models that use different priors.\n\n1\n\nIntro duction\n\nUnsupervised learning of structured variables in data is a difficult problem that has received considerable recent attention. In this paper, we consider learning probabilistic grammars, a class of structure models that includes Markov models, hidden Markov models (HMMs) and probabilistic context-free grammars (PCFGs). Central to natural language processing (NLP), probabilistic grammars are recursive generative models over discrete graphical structures, built out of conditional multinomial distributions, that make independence assumptions to permit efficient exact probabilistic inference. There has been an increased interest in the use of Bayesian methods as applied to probabilistic grammars for NLP, including part-of-speech tagging [10, 20], phrase-structure parsing [7, 11, 16], and combinations of models [8]. In Bayesian-minded work with probabilistic grammars, a common thread is the use of a Dirichlet prior for the underlying multinomials, because as the conjugate prior for the multinomial, it bestows computational feasibility. The Dirichlet prior can also be used to encourage the desired property of sparsity in the learned grammar [11]. A related widely known example is the latent Dirichlet al location (LDA) model for topic modeling in document collections [5], in which each document's topic distribution is treated as a hidden variable, as is the topic distribution from which each word is drawn.1 Blei and Lafferty [4] showed empirical improvements over LDA using a logistic normal distribution that permits different topics to correlate with each other, resulting in a correlated topic model (CTM). Here we aim to learn analogous correlations such as: a word that is likely to take one kind of argument (e.g., singular nouns) may be likely to take others as well (e.g., plural or proper nouns). By permitting such correlations via the distribution over the\n1 A certain variant of LDA can be seen as a Bayesian version of a zero-order HMM, where the unigram state (topic) distribution is sampled first for each sequence (document).\n\n\f\nk k k K k K y x N\n\nFigure 1: A graphical model for the logistic normal probabilistic grammar. y is the derivation, x is the observed string. parameters, we hope to break independence assumptions typically made about the behavior of different part-of-speech tags. In this paper, we present a model, in the Bayesian setting, which extends CTM for probabilistic grammars. We also derive an inference algorithm for that model, which is ultimately used to provide a point estimate for the grammar, permitting us to perform fast and exact inference. This is required if the learned grammar is to be used as a component in an application. The rest of the paper is organized as follows. 2 gives a general form for probabilistic grammars built out of multinomial distributions. 3 describes our model and an efficient variational inference algorithm. 4 presents a probabilistic context-free dependency grammar often used in unsupervised natural language learning. Experimental results showing the competitiveness of our method for estimating that grammar are presented in 5.\n\n2\n\nProbabilistic Grammars\n\nA probabilistic grammar defines a probability distribution over a certain kind of structured ob ject (a derivation of the underlying symbolic grammar) explained through step-by-step stochastic process. HMMs, for example, can be understood as a random walk through a probabilistic finite-state network, with an output symbol sampled at each state. PCFGs generate phrase-structure trees by recursively rewriting nonterminal symbols as sequences of \"child\" symbols (each itself either a nonterminal symbol or a terminal symbol analogous to the emissions of an HMM). Each step or emission of an HMM and each rewriting operation of a PCFG is conditionally independent of the others given a single structural element (one HMM or PCFG state); this Markov property permits efficient inference. In general, a probabilistic grammar defines the joint probability of a string x and a grammatical derivation y: p(x, y | ) = kK iNk\n=1 =1\n\nkki,i ,\n\nf\n\n(x,y)\n\n= exp\n\nkK iNk\n=1 =1\n\nfk,i (x, y) log k,i\n\n(1)\n\nwhere fk,i is a function that \"counts\" the number of times the k th distribution's ith event occurs in the derivation. The parameters are a collection of K multinomials 1 , ..., K , the k th of which includes Nk events. Note that there may be many derivations y for a given string x--perhaps even infinitely many in some kinds of grammars. HMMs and vanilla PCFGs are the best known probabilistic grammars, but there are others. For example, in 5 we experiment with the \"dependency model with valence,\" a probabilistic grammar for dependency parsing first proposed in [14].\n\n3\n\nLogistic Normal Prior on Probabilistic Grammars\n\nA natural choice for a prior over the parameters of a probabilistic grammar is a Dirichlet prior. The Dirichlet family is conjugate to the multinomial family, which makes the inference more elegant and less computationally intensive. In addition, a Dirichlet prior can encourage sparse solutions, a property which is important with probabilistic grammars [11].\n\n\f\nHowever, in [4], Blei and Lafferty noticed that the Dirichlet distribution is limited in its expressive power when modeling a corpus of documents, since it is less flexible about capturing relationships between possible topics. To solve this modeling issue, they extended the LDA model to use a logistic normal distribution [2] yielding correlated topic models. The logistic normal distribution maps a d-dimensional multivariate Gaussian to a distribution d on the d-dimensional probability simplex, Sd = { z1 , ..., zd Rd : zi 0, i=1 zi = 1}, by exponentiating the normally-distributed variables and normalizing. Here we take a step analogous to Blei and Lafferty, aiming to capture correlations between the grammar's parameters. Our hierarchical generative model, which we call a logisticnormal probabilistic grammar, generates a sentence and derivation tree x, y as follows (see also Fig. 1): 1. Generate k N(k , k ) for k = 1, ..., K . N k ) for k = 1, ..., K and i = 1, ..., Nk . 2. Set k,i = exp(k,i ) i =1 exp(k,i 3. Generate x and y from p(x, y | ) (i.e., sample from the probabilistic grammar). We now turn to derive a variational inference algorithm for the model.2 Variational Bayesian inference seeks an approximate posterior function q ( , y) which maximizes a lower bound (the negated variational free energy) on the log-likelihood [12], a bound which is achieved using Jensen's inequality: K log p(x, y | , ) i=1 Eq [log p( i | i , i )] + Eq [log p(x, y | )] + H (q ) (2) We make a mean-field assumption, and assume that the posterior has the following form: K Nk q ( , y) = (3) q (y) q (k,i | k,i , k,i ) ~ ~2 k=1 i=1 ~ ~2 where q (k,i | k,i , k,i ) is a Gaussian N(k,i , k,i ). ~ ~2 Unfolding the expectation with respect to q (y) in the second term in Eq. 2, while recalling that is a deterministic function of , we have that: K = Nk y Eq [log p(x, y | )] = Eq() q (y)fk,i (x, y) log k,i k=1 i=1 ~\nfk , i\n\nK Eq()\nk=1\n\n Nk ~ fk,i k,i - log i=1\n\nNk\ni\n\n=1\n\nexp k,i\n\n(4)\n\n~ where fk,i is the expected number of occurrences of the ith event in distribution k , under q (y).3 The logarithm term in Eq. 4 is problematic, so we follow [4] in approximating it with ~ ~ a first-order Taylor expansion, introducing K more variational parameters 1 , ..., K : N Nk 1 k ~ log k - 1 + (5) log i =1 exp k,i i =1 exp k,i ~ k We now have Eq [log p(x, y | )] Eq() = K\nk=1\n\n Nk ~ Nk 1 ~ k,i - log k + 1 - ~k i=1 fk,i i =1 exp k,i ~ ~ Nk ~ 1 ~ Nk k,i ~2 ~ i=1 fk,i k,i - log k + 1 - ~ i =1 exp k,i + 2 k\nk=1 k,i\n\nK\n\n(6)\n\n=\n2\n\nK\n\nk=1\n\nNk ~ ~ i=1 fk,i k,i\n\n(7)\n\nWe note that variational inference algorithms have been successfully applied to grammar learning tasks, for example, in [16] and [15]. 3 With probabilistic grammars, this quantity can be computed using a summing dynamic programming algorithm like the forward-backward or inside-outside algorithm.\n\n\f\n~ ~ Note the shorthand k,i to denote an expression involving , , and . ~~ The final form of our bound is:4 K + log p(x, y | , ) Eq [log p( k | k , k )] k=1 K\nk=1\n\n+ Nk ~ ~ fk,i k,i H (q ) i=1\n\n(8)\n\nSince, we are interested in EM-style algorithm, we will alternate between finding the maximizing q ( ) and the maximizing q (y). Maximization with respect to q ( ) is not hard, because q ( ) is parametrized (see Appendix A). The following lemma shows that fortunately, finding the maximizing q (y), which we did not parametrize originally, is not hard either: Lemma 1. Let r(y | x, e ) denote the conditional distribution over y given x defined as: r(y | x, e ) =\n~ ~\n\n1 ~ Z ( )\n\nK\n\nk=1\n\nNk\n\ni=1\n\n~ exp k,i fk,i (x, y)\n\n(9)\n\n~ ~ where Z ( ) is a normalization constant. Then q (y) = r(y | x, e ) maximizes the bound in Eq. 8.\n\nProof. First note that H (q ) = H (q ( | , )) + H (q (y)). This means that the terms we are ~~ ~ interested in maximizing from Eq. 8 are the following, after writing down fk,i explicitly: L = argmax\nq (y )\n\ny\n\nq (y)\n\nK\nk=1\n\n~ i=1 fk,i (x, y)k,i Nk\n\n+\n\nH (q (y))\n\n(10)\n\nHowever, note that: L = argmin DKL (q (y) r(y | x, e ))\nq (y ) ~\n\n(11)\n\nwhere DKL denotes the KL divergence. To see that, combine the definition of KL divergence K Nk ~ ~ ~ ~ with the fact that k=1 i=1 fk,i (x, y)k,i - log Z ( ) = log r(y | x, e ) where log Z ( ) does not depend on q (y). Eq. 11 is minimized when q = r. Interestingly, from the above lemma, the minimizing q (y) has the same form as the probabilistic grammar in discussion, only without having sum-to-one constraints on (leading to the required normalization constant). As in classic EM with probabilistic grammars, we never need to represent q (y) explicitly; we need only ~, which can be calculated as expected f ~ feature values under r(y | x, e ) using dynamic programming. As noted, we are interested in a point estimate of . To achieve this, we will use the above variational method within an EM algorithm that estimates and in empirical Bayes fashion, then estimates as , the mean of the learned prior. In the E-step, we maximize the bound with respect to the variational parameters (, , , and ~) using coordinate ~~~ f ascent. We optimize each of these separately in turn, cycling through, using appropriate optimization algorithms for each (conjugate gradient for , Newton's method for , a closed ~ ~ ~ form for , and dynamic programming to solve for ~). In the M-step, we apply maximum f likelihood estimation with respect to and given sufficient statistics gathered from the variational parameters in the E-step. The full algorithm is given in Appendix A.\n\n4\n\nProbabilistic Dep endency Grammar Mo del\n\nDependency grammar [19] refers to linguistic theories that posit graphical representations of sentences in which words are vertices and the syntax is a tree. Such grammars can be context-free or context-sensitive in power, and they can be made probabilistic [9]. Dependency syntax is widely used in information extraction, machine translation, question\n4\n\nA tighter bound was proposed in [1], but we follow [4] for simplicity.\n\n\f\nx = NNP VBD JJ NNP ; y =\n\nNNP Patrick\n\nVBD spoke\n\nJJ NNP little French\n\nFigure 2: An example of a dependency tree (derivation y). NNP denotes a proper noun, VBD a past-tense verb, and JJ an adjective, following the Penn Treebank conventions. answering, and other natural language processing applications. Here, we are interested in unsupervised dependency parsing using the \"dependency model with valence\" [14]. The model is a probabilistic head automaton grammar [3] with a \"split\" form that renders inference cubic in the length of the sentence [6]. Let x = x1 , x2 , ..., xn be a sentence (here, as in prior work, represented as a sequence of part-of-speech tags). x0 is a special \"wall\" symbol, $, on the left of every sentence. A tree y is defined by a pair of functions yleft and yright (both {0, 1, 2, ..., n} 2{1,2,...,n} ) that map each word to its sets of left and right dependents, respectively. Here, the graph is constrained to be a projective tree rooted at x0 = $: each word except $ has a single parent, and there are no cycles or crossing dependencies. yleft (0) is taken to be empty, and yright (0) contains the sentence's single head. Let y(i) denote the subtree rooted at position i. The probability P (y(i) | xi , ) of generating this subtree, given its head word xi , is defined recursively: D P (y(i) | xi , ) = (12) {left ,right } s (stop | xi , D , [yD (i) = ]) j yD (i) s (stop | xi , D , firsty (j )) c (xj | xi , D ) P (y(j ) | xj , ) where firsty (j ) is a predicate defined to be true iff xj is the closest child (on either side) to its parent xi . The probability of the entire tree is given by p(x, y | ) = P (y(0) | $, ). The parameters are the multinomial distributions s ( | , , ) and c ( | , ). To follow the general setting of Eq. 1, we index these distributions as 1 , ..., K . Figure 2 shows a dependency tree and its probability under this model.\n\n5\n\nExp eriments\n\nData Following the setting in [13], we experimented using part-of-speech sequences from the Wal l Street Journal Penn Treebank [17], stripped of words and punctuation. We follow standard parsing conventions and train on sections 221,5 tune on section 22, and report final results on section 23. Evaluation After learning a point estimate , we predict y for unseen test data (by parsing with the probabilistic grammar) and report the fraction of words whose predicted parent matches the gold standard corpus, known as attachment accuracy. Two parsing methods were considered: the most probable \"Viterbi\" parse (argmaxy p(y | x, )) and the minimum Bayes risk (MBR) parse (argminy Ep(y |x,) [ (y; x, y )]) with dependency attachment error as the loss function. Settings Our experiment compares four methods for estimating the probabilistic grammar's parameters: EM Maximum likelihood estimate of using the EM algorithm to optimize p(x | ) [14]. EM-MAP Maximum a posteriori estimate of using the EM algorithm and a fixed symmetric Dirichlet prior with > 1 to optimize p(x, | ). Tune to maximize the likelihood of an unannotated development dataset, using grid search over [1.1, 30].\n5 Training in the unsupervised setting for this data set can be expensive, and requires running a cubic-time dynamic programming algorithm iteratively, so we follow common practice in restricting the training set (but not development or test sets) to sentences of length ten or fewer words. Short sentences are also less structurally ambiguous and may therefore be easier to learn from.\n\n\f\nVB-Dirichlet Use variational Bayes inference to estimate the posterior distribution p( | x, ), which is a Dirichlet. Tune the symmetric Dirichlet prior's parameter to maximize the likelihood of an unannotated development dataset, using grid search over [0.0001, 30]. Use the mean of the posterior Dirichlet as a point estimate for . VB-EM-Dirichlet Use variational Bayes EM to optimize p(x | ) with respect to . Use the mean of the learned Dirichlet as a point estimate for (similar to [5]). VB-EM-Log-Normal Use variational Bayes EM to optimize p(x | , ) with respect to and . Use the (exponentiated) mean of this Gaussian as a point estimate for . Initialization is known to be important for EM as well as for the other algorithms we experiment with, since it involves non-convex optimization. We used the successful initializer from [14], which estimates using soft counts on the training data where, in an n-length 1 sentence, (a) each word is counted as the sentence's head n times, and (b) each word xi attaches to xj proportional to |i - j |, normalized to a single attachment per word. This initializer is used with EM, EM-MAP, VB-Dirichlet, and VB-EM-Dirichlet. In the case of VB-EM-Log-Normal, it is used as an initializer both for and inside the E-step. In all experiments reported here, we run the iterative estimation algorithm until the likelihood of a held-out, unannotated dataset stops increasing. For learning with the logistic normal prior, we consider two initializations of the covariance matrices k . The first is the Nk Nk identity matrix. We then tried to bias the solution by injecting prior knowledge about the part-of-speech tags. Injecting a bias to parameter estimation of the DMV model has proved to be useful [18]. To do that, we mapped the tag set (34 tags) to twelve disjoint tag families.6 The covariance matrices for all dependency distributions were initialized with 1 on the diagonal, 0.5 between tags which belong to the same family, and 0 otherwise. These results are given in Table 1 with the annotation \"families.\" Results Table 1 shows experimental results. We report attachment accuracy on three subsets of the corpus: sentences of length 10 (typically reported in prior work and most similar to the training dataset), length 20, and the full corpus. The Bayesian methods all outperform the common baseline (in which we attach each word to the word on its right), but the logistic normal prior performs considerably better than the other two methods as well. The learned covariance matrices were very sparse when using the identity matrix to initialize. The diagonal values showed considerable variation, suggesting the importance of variance alone. When using the \"tag families\" initialization for the covariance, there were 151 elements across the covariance matrices which were not identically 0 (out of more than 1,000), pointing to a learned relationship between parameters. In this case, most covariance matrices for c dependencies were diagonal, while many of the covariance matrices for the stopping probabilities (s ) had significant correlations.\n\n6\n\nConclusion\n\nWe have considered a Bayesian model for probabilistic grammars, which is based on the logistic normal prior. Experimentally, several different approaches for grammar induction were compared based on different priors. We found that a logistic normal prior outperforms earlier approaches, presumably because it can capitalize on similarity between part-of-speech tags, as different tags tend to appear as arguments in similar syntactic contexts. We achieved state-of-the-art unsupervised dependency parsing results.\n6 These are simply coarser tags: adjective, adverb, conjunction, foreign, interjection, noun, number, particle, preposition, pronoun, proper, verb. The coarse tags were chosen manually to fit seven treebanks in different languages.\n\n\f\nAttach-Right EM EM-MAP, = 1.1 VB-Dirichlet, = 0.25 VB-EM-Dirichlet (0 ) VB-EM-Log-Normal, k = I VB-EM-Log-Normal, families\n\nattachment accuracy (%) Viterbi decoding MBR decoding |x| 10 |x| 20 all |x| 10 |x| 20 38.4 33.4 31.7 38.4 33.4 45.8 39.1 34.2 46.1 39.9 45.9 39.5 34.9 46.2 40.6 46.9 40.0 35.7 47.1 41.1 45.9 39.4 34.9 46.1 40.6 56.6 43.3 37.4 59.1 45.9 59.3 45.1 39.0 59.4 45.9\n\nall 31.7 35.9 36.7 37.6 36.9 39.9 40.5\n\nTable 1: Attachment accuracy of different learning methods on unseen test data from the Penn Treebank of varying levels of difficulty imposed through a length filter. Attach-Right attaches each word to the word on its right and the last word to $. EM and EM-MAP with a Dirichlet prior ( > 1) are reproductions of earlier results [14, 18].\n\nAcknowledgments\nThe authors would like to thank the anonymous reviewers, John Lafferty, and Matthew Harrison for their useful feedback and comments. This work was made possible by an IBM faculty award, NSF grants IIS-0713265 and IIS-0836431 to the third author and computational resources provided by Yahoo.\n\nReferences\n[1] A. Ahmed and E. Xing. On tight approximate inference of the logistic normal topic admixture model. In Proc. of AISTATS, 2007. [2] J. Aitchison and S. M. Shen. Logistic-normal distributions: some properties and uses. Biometrika, 67:261272, 1980. [3] H. Alshawi and A. L. Buchsbaum. Head automata and bilingual tiling: Translation with minimal representations. In Proc. of ACL, 1996. [4] D. Blei and J. D. Lafferty. Correlated topic models. In Proc. of NIPS, 2006. [5] D. Blei, A. Ng, and M. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:9931022, 2003. [6] J. Eisner. Bilexical grammars and a cubic-time probabilistic parser. In Proc. of IWPT, 1997. [7] J. Eisner. Transformational priors over grammars. In Proc. of EMNLP, 2002. [8] J. R. Finkel, C. D. Manning, and A. Y. Ng. Solving the problem of cascading errors: Approximate Bayesian inference for linguistic annotation pipelines. In Proc. of EMNLP, 2006. [9] H. Gaifman. Dependency systems and phrase-structure systems. Information and Control, 8, 1965. [10] S. Goldwater and T. L. Griffiths. A fully Bayesian approach to unsupervised part-ofspeech tagging. In Proc. of ACL, 2007. [11] M. Johnson, T. L. Griffiths, and S. Goldwater. Bayesian inference for PCFGs via Markov chain Monte Carlo. In Proc. of NAACL, 2007. [12] M. I. Jordan, Z. Ghahramani, T. S. Jaakola, and L. K. Saul. An introduction to variational methods for graphical models. Machine Learning, 37(2):183233, 1999. [13] D. Klein and C. D. Manning. A generative constituent-context model for improved grammar induction. In Proc. of ACL, 2002. [14] D. Klein and C. D. Manning. Corpus-based induction of syntactic structure: Models of dependency and constituency. In Proc. of ACL, 2004.\n\n\f\n[15] K. Kurihara and T. Sato. Variational Bayesian grammar induction for natural language. In Proc. of ICGI, 2006. [16] P. Liang, S. Petrov, M. Jordan, and D. Klein. The infinite PCFG using hierarchical Dirichlet processes. In Proc. of EMNLP, 2007. [17] M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz. Building a large annotated corpus of English: The Penn treebank. Computational Linguistics, 19:313330, 1993. [18] N. A. Smith and J. Eisner. Annealing structural bias in multilingual weighted grammar induction. In Proc. of COLING-ACL, 2006. e [19] L. Tesni`re. Elment de Syntaxe Structurale. Klincksieck, 1959. e [20] K. Toutanova and M. Johnson. A Bayesian LDA-based model for semi-supervised part-of-speech tagging. In Proc. of NIPS, 2007.\n\nA\n\nVB-EM for Logistic-Normal Probabilistic Grammars\n\nThe algorithm for variational inference with probabilistic grammars using logistic normal ~l,(t) prior follows.7 Since the updates for k are fast, we perform them after each optimization routine in the E-step (suppressed for clarity). There are variational parameters for each training example, indexed by . We denote by B the variational bound in Eq. 8. Our stopping criterion relies on the likelihood of a held-out set (5) using a point estimate of the model.\nO\n\nInput: initial parameters (0) , (0) , training data x, and development data x utput: learned parameters , t1; rep eat E-step (for each training example = 1, ..., M ): rep eat ,(t) optimize for k , k = 1, ..., K : use conjugate gradient descent \" ith w \" \"~ ` PNk \" ~ (t-1) -1 (t-1) L ~ ~ fk,i /k exp k,i ~ ) )( - k ) - fk,i + ~ = - ( = k\n ,i ~ k k i i 1\n\n+\n\nk,i ~2\n\n/2\n\n\n\n;\n\n, k = 1,\".., K : use Newton's method for each coordinate (with k,i > 0) with . ~ PNk ~ ~ = - ~2 ~ ~2 fk,i \" exp(k,i + k,i /2)/2k + 1/2k,i ; i =1 \" \" ~ ,(t) , k: ,(t) PNk exp ,(t) + ( ,(t) )2 /2 ; ~ update k ~k,i ~k,i k i=1 \" \" PNk ,(t) ,(t) ,(t) 1 ~ ,(t) ~ ,(t) ~ ,(t) ~ ~ update k , k: k,i k,i - log k + 1 - ~ ,(t) i =1 exp k,i + (k,i )2 /2 ; ~ optimize k ~\nL k,i ~2 (t-1) -k,ii /2 k ,(t) compute expected counts ~k , k = 1, ..., K : use an inside-outside algorithm to re-estimate f ~; ,(t) ~ expected counts fk,i in weighted grammar q (y ) with weights e until B does not change ; M-step: Estimate (t) and (t) using the following maximum likelihood closed form solution: P (t) ,(t) 1 k,i M M k,i =1 ~ h i \" \"P (t) ,(t) ,(t) (t) (t) (t) P ,(t) (t) P ,(t) M 1 k M ~ ~ ~ ,(t) )2 ,i i,j + M k,i k,j - k,j M k,i - k,i M k,j k =1 k,i k,j + ( =1 ~ =1 ~ i,j\n\n,(t)\n\nwhere i,j = 1 if i = j and 0 otherwise. until likelihood of held-out data, p(x | E [(t) ]), decreases ; t t + 1; return (t) , (t)\n\n7\n\nAn implementation of the algorithm is available at http://www.ark.cs.cmu.edu/DAGEEM.\n\n\f\n", "award": [], "sourceid": 3559, "authors": [{"given_name": "Shay", "family_name": "Cohen", "institution": null}, {"given_name": "Kevin", "family_name": "Gimpel", "institution": null}, {"given_name": "Noah", "family_name": "Smith", "institution": null}]}