{"title": "Continuous Speech Recognition by Linked Predictive Neural Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 199, "page_last": 205, "abstract": null, "full_text": "Continuous Speech Recognition by \nLinked Predictive Neural Networks \n\nJoe Tebelskis, Alex Waibel, Bojan Petek, and Otto Schmidbauer \n\nSchool of Computer Science \nCarnegie Mellon University \n\nPittsburgh, PA 15213 \n\nAbstract \n\nWe present a large vocabulary, continuous speech recognition system based \non Linked Predictive Neural Networks (LPNN's). The system uses neu(cid:173)\nral networks as predictors of speech frames, yielding distortion measures \nwhich are used by the One Stage DTW algorithm to perform continuous \nspeech recognition. The system, already deployed in a Speech to Speech \nTranslation system, currently achieves 95%, 58%, and 39% word accuracy \non tasks with perplexity 5, 111, and 402 respectively, outperforming sev(cid:173)\neral simple HMMs that we tested. We also found that the accuracy and \nspeed of the LPNN can be slightly improved by the judicious use of hidden \ncontrol inputs. We conclude by discussing the strengths and weaknesses \nof the predictive approach. \n\n1 \n\nINTRODUCTION \n\nNeural networks are proving to be useful for difficult tasks such as speech recogni(cid:173)\ntion, because they can easily be trained to compute smooth, nonlinear, nonpara(cid:173)\nmetric functions from any input space to output space. In speech recognition, the \nfunction most often computed by networks is classification, in which spectral frames \nare mapped into a finite set of classes, such as phonemes. In theory, classification \nnetworks approximate the optimal Bayesian discriminant function [1], and in prac(cid:173)\ntice they have yielded very high accuracy [2, 3, 4]. However, integrating a phoneme \nclassifier into a speech recognition system is nontrivial, since classification decisions \ntend to be binary, and binary phoneme-level errors tend to confound word-level \nhypotheses. To circumvent this problem, neural network training must be carefully \nintegrated into word level training [1, 5]. An alternative function which can be com-\n\n199 \n\n\f200 \n\nTebelskis, Waibel, Petek, and Schmidbauer \n\nputed by networks is prediction, where spectral frames are mapped into predicted \nspectral frames. This provides a simple way to get non-binary distortion measures, \nwith straightforward integration into a speech recognition system. Predictive net(cid:173)\nworks have been used successfully for small vocabulary [6, 7] and large vocabulary \n[8, 9] speech recognition systems. In this paper we describe our prediction-based \nLPNN system [9], which performs large vocabulary continuous speech recognition, \nand which has already been deployed within a Speech to Speech Translation system \n[10]. We present our experimental results, and discuss the strengths and weaknesses \nof the predictive approach. \n\n2 LINKED PREDICTIVE NEURAL NETWORKS \n\nThe LPNN system is based on canonical phoneme models, which can be logically \nconcatenated in any order (using a \"linkage pattern\") to create templates for dif(cid:173)\nferent words; this makes the LPNN suitable for large vocabulary recognition. \n\nEach canonical phoneme is modeled by a short sequence of neural networks. The \nnumber of nets in the sequence, N >= 1, corresponds to the granularity of the \nphoneme model. These phone modeling networks are nonlinear, multilayered, feed(cid:173)\nforward, and \"predictive\" in the sense that, given a short section of speech, the \nnetworks are required to extrapolate the raw speech signal, rather than to classify \nit. Thus, each predictive network produces a time-varying model of the speech \nsignal which will be accurate in regions corresponding to the phoneme for which \nthat network has been trained, but inaccurate in other regions (which are better \nmodeled by other networks). Phonemes are thus \"recognized\" indirectly, by virtue \nof the relative accuracies of the different predictive networks in various sections of \nspeech. Note, however, that phonemes are not classified at the frame level. Instead, \ncontinuous scores (prediction errors) are accumulated for various word candidates, \nand a decision is made only at the word level, where it is finally appropriate. \n\n2.1 TRAINING AND TESTING ALGORITHMS \n\nThe purpose of the training procedure is both (a) to train the networks to become \nbetter predictors, and (b) to cause the networks to specialize on different phonemes. \nGiven a known training utterance, the training procedure consists of three steps: \n\n1. Forward Pass: All the networks make their predictions across the speech sam(cid:173)\n\nple, and we compute the Euclidean distance matrix of prediction errors between \npredicted and actual speech frames. (See Figure 1.) \n\n2. Alignment Step: We compute the optimal time-alignment path between the \ninput speech and corresponding predictor nets, using Dynamic Time Warping. \n3. Backward Pass: Prediction error is backpropagated into the networks according \n\nto the segmentation given by the alignment path. (See Figure 2.) \n\nHence backpropagation causes the nets to become better predictors, and the align(cid:173)\nment path induces specialization of the networks for different phonemes. \n\nTesting is performed using the One Stage algorithm [11], which is a classical exten(cid:173)\nsion of the Dynamic Time Warping algorithm for continuous speech. \n\n\fContinuous Speech Recognition by Linked ftedictive Neural Networks \n\n201 \n\n\u2022 \n\n\u2022 \n\n\u2022 \u2022 \u2022 - - - - prediction errors \n\n..--+--+-----if---~ ......\u2022................ . ... \n+--+---+-+--+-----4: \u2022......\u2022.................... \nt--t---t--+--t---t--~ .... . .\u2022......... . . . ........ \n\n\u00ab \nCtJ \n\u00ab \n\nphoneme \"a\" \npredictors \n\nphoneme \"b\" \npredictors \n\nFigure 1: The forward pass during training. Canonical phonemes are modeled by \nsequences of N predictive networks, shown as triangles (here N=3). Words are \nrepresented by \"linkage patterns\" over these canonical phoneme models (shown in \nthe area above the triangles), according to the phonetic spelling of the words. Here \nwe are training on the word \"ABA\". In the forward pass, prediction errors (shown \nas black circles) are computed for all predictors, for each frame of the input speech. \nAs these prediction errors are routed through the linkage pattern, they fill a distance \nmatrix (upper right). \n\n+--+---1--1---1--+---1-: : :: : :A(igr)~~~(p.~th\u00b7 \u00b7 .... \n\n\u00ab \nCtJ \n\u00ab \\~~-\\tI------+-\n\n~;:;:1tt:;:;:;:;:::;::;:;::;::;::;:;:;:;:;::;:::;::;J;;tt \n\nFigure 2: The backward pass during training. After the DTW alignment path has \nbeen computed, error is backpropagated into the various predictors responsible for \neach point along the alignment path. The back propagated error signal at each such \npoint is the vector difference between the predicted and actual frame. This teaches \nthe networks to become better predictors, and also causes the networks to specialize \non different phonemes. \n\n\f202 \n\nTebelskis, Waibel, Petek, and Schmidbauer \n\n3 RECOGNITION EXPERIMENTS \n\nWe have evaluated the LPNN system on a database of continuous speech recorded \nat CMU. The database consists of 204 English sentences using a vocabulary of 402 \nwords, comprising 12 dialogs in the domain of conference registration. Training \nand testing versions of this database were recorded in a quiet office by multiple \nspeakers for speaker-dependent experiments. Recordings were digitized at a sam(cid:173)\npling rate of 16 KHz. A Hamming window and an FFT were computed, to produce \n16 melscale spectral coefficients every 10 msec. In our experiments we used 40 \ncontext-independent phoneme models (including one for silence), each of which had \na 6-state phoneme topology similar to the one used in the SPICOS system [12]. \n\ntV>o tt..1s; \n\"*,,,theOl .. d p/l