{"title": "English Alphabet Recognition with Telephone Speech", "book": "Advances in Neural Information Processing Systems", "page_first": 199, "page_last": 206, "abstract": null, "full_text": "English Alphabet Recognition \n\nwith Telephone Speech \n\nMark Fanty, Ronald A. Cole and Krist Roginski \n\nCenter for Spoken Language Understanding \n\nOregon Graduate Institute of Science and Technology \n19600 N.W. Von Neumann Dr., Beaverton, OR 97006 \n\nAbstract \n\nA recognition system is reported which recognizes names spelled over the \ntelephone with brief pauses between letters. The system uses separate \nneural networks to locate segment boundaries and classify letters. The \nletter scores are then used to search a database of names to find the best \nscoring name. The speaker-independent classification rate for spoken let(cid:173)\nters is 89%. The system retrieves the correct name, spelled with pauses \nbetween letters, 91 % of the time from a database of 50,000 names. \n\n1 \n\nINTRODUCTION \n\nThe English alphabet is difficult to recognize automatically because many letters \nsound alike; e.g., BID, PIT, VIZ and F IS. When spoken over the telephone, the \ninformation needed to discriminate among several of these pairs, such as F IS, PIT, \nBID and VIZ, is further reduced due to the limited bandwidth of the channel \nSpeaker-independent recognition of spelled names over the telephone is difficult \ndue to variability caused by channel distortions, different handsets, and a variety \nof background noises. Finally, when dealing with a large population of speakers, \ndialect and foreign accents alter letter pronunciations. An R from a Boston speaker \nmay not contain an [r]. \n\nHuman classification performance on telephone speech underscores the difficulty \nof the problem. We presented each of ten listeners with 3,197 spoken letters in \nrandom order for identification. The letters were taken from 100 telephone calls \n199 \n\n\f200 \n\nFanty, Cole, and Roginski \n\nin which the English alphabet was recited with pauses between letters, and 100 \ndifferent telephone calls with first or last names spelled with pauses between letters. \nOur subjects averaged 93% correct classification of the letters, with performance \nranging from 90% to 95%. This compares to error rates of about 1 % for high quality \nmicrophone speech [DALY 87]. \n\nOver the past three years, our group at OGI has produced a series of letter clas(cid:173)\nsification and name retrieval systems. These systems combine speech knowledge \nand neural network classification to achieve accurate spoken letter recognition \n[COLE 90, FANTY 91]. Our initial work focused on speaker-independent recogni(cid:173)\ntion of isolated letters using high quality microphone speech. By accurately locating \nsegment boundaries and carefully designing feature measurements to discriminate \namong letters, we achieved 96% classification of letters. \n\n, \n\nWe extended isolated letter recognition to recognition of words spelled with brief \npauses between the letters, again using high quality speech [FANTY 91, COLE 91]. \nThis task is more difficult than recognition of isolated letters because there are \n\"pauses\" within letters such as the closures in \"X\" \"H\" and \"W \" which must be \ndistinguished from the pauses that separate letters, and because speakers do not \nalways pause between letters when asked to do so. In the system, a neural network \nsegments speech into a sequence of broad phonetic categories. Rules are applied \nto the segmentation to locate letter boundaries, and the hypothesized letters are \nre-classified using a second neural network . The letter scores from this network are \nused to retrieve the best scoring name from a database of 50,000 last names. First \nchoice name retrieval was 95.3%, with 99% of the spelled names in the top three \nchoices. Letter recognition accuracy was 90%. \n\n\" \n\nDuring the past year, with support from US WEST Advanced Technologies, we \nhave extended our approach to recognition of names spelled over the telephone . \nThis report describes the recognition system, some experiments that motivated its \ndesign, and its current performance. \n\n1.1 SYSTEM OVERVIEW \n\nData Capture and Signal Processing. Telephone speech is sampled at 8 kHz \nat 14-bit resolution. Signal processing routines perform a seventh order PLP (Per(cid:173)\nceptual Linear Predictive) analysis [HERMANSKY 90] every 3 msec using a 10 \nmsec window. This analysis yields eight coefficients per frame, including energy. \n\nPhonetic Classification. Frame-based phonetic classification provides a se(cid:173)\nquence of phonetic labels that can be used to locate and classify letters. Clas(cid:173)\nsification is performed by a fully-connected three-layer feed-forward network that \nassigns 22 phonetic category scores to each 3 msec time frame. The 22 labels pro(cid:173)\nvide an intermediate level of description, in which some phonetic categories, such \nas [b]-[d], [p]-[t]-[k] and [m]-[n] are combined; these fine phonetic distinctions are \nperformed during letter classification, described below. The input to the network \nconsists of 120 features representing PLP coefficients in a 432 msec window centered \non the frame to be classified. \n\nThe frame-by-frame outputs of the phonetic classifier are converted to a sequence \nof phonetic segments corresponding to a sequence of hypothesized letters. This is \n\n\fEnglish Alphabet Recognition with Telephone Speech \n\n201 \n\ndone with a Viterbi search that uses duration and phoneme sequence constraints \nprovided by letter models. For example, the letter model for MN consists of optional \nglottalization (MN-q), followed by the vowel [eh] (MN-eh), followed by the nasal \nmurmur (MN-mn). Because background noise is often classified as [f]-[s) or [m]-[n), \na noise \"letter\" model was added which consists of either of these phonemes. \n\nLetter Classification. Once letter segmentation is performed, a set of 178 fea(cid:173)\ntures is computed for each letter and used by a fully-connected feed-forward network \nwith one hidden layer to reclassify the letter. Feature measurements are based on \nthe phonetic boundaries provided by the segmentation. At present, the features \nconsist of segment durations, PLP coefficients for thirds of the consonant (fricative \nor stop) before the first sonorant, PLP for sevenths of the first sonorant, PLP for \nthe 200 msecs after the sonorant, PLP slices 6 and 10 msec after the sonorant onset, \nPLP slices 6 and 30 msec before any internal sonorant boundary (e.g. [eh]/[m)), \nzero crossing and amplitude profiles from 180 msec before the sonorant to 180 msec \nafter the sonorant. The outputs of the classifier are the 26 letters plus the category \n\"not a letter.\" \n\nName Retrieval. The output of the classifier is a score between 0 and 1 for each \nletter. These scores are treated as probabilities and the most likely name is retrieved \nfrom the database of 50,000 last names. The database is stored in an efficient tree \nstructure. Letter deletions and insertions are allowed with a penalty. \n\n2 SYSTEM DEVELOPMENT \n\n2.1 DATA COLLECTION \n\nCallers were solicited through local newspaper and television coverage, and notices \non computer bulletin boards and news groups. Callers had the choice of using a \nlocal phone number or toll-free 800-number. \n\nA Gradient Technology Desklab attached to a UNIX workstation was programmed \nto answer the phone and record the answers to pre-recorded questions. The first \nthree thousand callers were given the following instructions, designed to generate \nspoken and spelled names, city names, and yes/no responses: (1) What city are \nyou calling from? (2) What is your last name? (3) Please spell your last name. (4) \nPlease spell your last name with short pauses between letters. (5) Does your last \nname contain the letter \"A\" as in apple? (6) What is your first name? (7) Please \nspell your first name with short pauses between letters. (8) What city and state did \nyou grow up in? (9) Would you like to receive more information about the results \nof this project? \n\nIn order to achieve sufficient coverage of rare letters, the final 1000 speakers were \nasked to recite the entire English alphabet with brief pauses between letters. \n\nThe system described here was trained on 800 speakers and tested on 400 speakers. \nThe training set contains 400 English alphabets and 800 first and last names spelled \nwith pauses between letters. The test set consists of 100 alphabets and 300 last \nnames spelled with pauses between letters. \n\n\f202 \n\nFanty, Cole, and Roginski \n\nA subset of the data was phonetically labeled to train and evaluate the neural \nnetwork segmenter. Time-aligned phonetic labels were assigned to 300 first and \nlast names and 100 alphabets, using the following labels: cl bcl dcl kcl pcl tcl q \naa ax: ay b ch d ah eh ey f iy jh kim n ow p r s t uw v w y z h#. This label \nset represents a subset of the TIMIT [LAMEL 86] labels sufficient to describe the \nEnglish alphabet. \n\n2.2 FRAME-BASED CLASSIFICATION \n\nExplicit location of segment boundaries is an important feature of our approach. \nConsider, for example, the letters Band D. They are distinguished by information \nat the onset of the letter; the spectrum of the release burst of [b] and [d], and the \nformant transitions during the first 10 or 15 msec of the vowel [iy]. By precisely \nlocating the burst onset and vowel onset, feature measurements can be designed to \noptimize discrimination. Moreover, the duration of the initial consonant segment \ncan be used to discriminate B from P, and D from T. \nA large number of experiments were performed to improve segmentation accuracy. \n[ROGINSKI 91]. These experiments focused on (a) determining the appropriate \nset of phonetic categories, (b) determining the set of features that yield the most \naccurate classification of these categories, and (c) determining the best strategy for \nsampling speech frames within the phonetic categories. \n\nPhonetic Categories. Given our recognition strategy of first locating segment \nboundaries and then classifying letters, it makes little sense to attempt to discrim(cid:173)\ninate [b]-[d], [p]-[t]-[k] or [m]-[n] at this stage. Experiments confirmed that using \nthe complete set of phonetic categories found in the English alphabet did not pro(cid:173)\nduce the most accurate frame-based phonetic classification. The actual choice of \ncategories was guided initially by perceptual confusions in the listening experiment, \nand was refined through a series of experiments in which different combinations of \nacoustically similar categories were merged. \n\nFeatures Used for Classification. A series of experiments was performed which \ncovaried the amount of acoustic context provided to the network and the number of \nhidden units in the network. The results are shown in Figure 1. A network with 432 \nmsec of spectral information, centered on the frame to be classified, and 40 hidden \nunits was chosen as the best compromise. \n\nSampling of Speech Frames. The training and test sets contained about 1.7 \nmillion 3 msec frames of speech; too many to train on all of them The manner in \nwhich speech frames were sampled was found to have a large effect of performance. \nIt was necessary to sample more speech frames from less frequently occurring cate(cid:173)\ngories and those with short durations (e.g., [b]). \n\nThe location within segments of the speech frames selected was found to have a \nprofound effect on the accuracy of boundary location. Accurate boundary placement \nrequired the correct proportion of speech frames sampled near segment boundaries. \nFor example, in order to achieve accurate location of stop bursts, it was necessary \nto sample a high proportion of speech frames just prior to the burst (within the \n\n\fEnglish Alphabet Recognition with Telephone Speech \n\n203 \n\n60 hidden nodes \n\n40 hidden nodes \n\n20 hidden nodes \n\ng \n\ni \nj \n\nc: \n0 \n\n~ t:: \n8 \nc ~ \n~ \nIII a... \n\no \n\n100 \n\n200 \n\n300 \n\n400 \n\n500 \n\n600 \n\nContext window in milliseconds \n\nFigure 1: Performance of the phonetic classifier as a function of PLP context and \nnumber of hidden units. \n\n\f204 \n\nFanty, Cole, and Roginski \n\nclosure category). Figure 2 shows the improvement in the placement of the [b]j[iy] \nboundary after sampling more training frames near that boundary. \n\n2.3 LETTER CLASSIFICATION \n\nIn order to avoid segmenting training data for letter classification by hand, an \nautomatic procedure was used. Each utterance was listened to and the letter names \nwere transcribed manually. Segmentation was performed as described above, except \nthe Viterbi search was forced to match the transcribed letter sequence. This resulted \nin very accurate segmentation. \n\nOne concern with this procedure was that artificially good segmentation for the \ntraining data could hurt performance on the test set, where there are bound to be \nmore segmentation errors (since the letter sequence is not known). The letter clas(cid:173)\nsifier should be able to recover from segmentation errors (e.g. a B being segmented \nas V with a long [v] before the burst). To do so, the network must be trained with \nerrorful segmentation. \n\nThe solution is to perform two segmentations. The forced segmentation finds the \nletter boundaries so the correct identity is known. A second, unforced, segmentation \nis performed and these phonetic boundaries are used to generate features used to \ntrain the classifier. \n\nAny \"letters\" found by the unforced search which correspond to noise or silence \nfrom the forced search are used as training data for the \"not a letter\" category. So \nthere are two ways noise can be eliminated: It can match the noise model of the \nsegmenter during the Viterbi search, or it can match a letter during segmentation, \nbut be reclassified as \"not a letter\" by the letter classifier. Both are necessary in \nthe current system. \n\n3 PERFORMANCE \n\nFrame-Based Phonetic Classification. The phonetic classifier was trained on \nselected speech frames from 200 speakers. About 450 speech frames were selected \nfrom 50 different occurrences of each phonetic category. Phonetic segmentation \nperformance on 50 alphabets and 150 last names was evaluated by comparing the \nfirst-choice of the classifier at each time frame to the label provided by a human \nexpert. The frame-by-frame agreement was 80% before the Viterbi search and 90% \nafter the Viterbi search. \n\nLetter Classification and N arne Retrieval. The training set consists of 400 \nalphabets spelled by 400 callers plus first and last names spelled by 400 callers, all \nwith pauses between the letters. \n\nWhen tested on 100 alphabets from new speakers, the letter classification was 89% \nwith less than 1 % insertions. When tested on 300 last names from new speakers, \nthe letter classification was 87% with 1.5% insertions. \n\nFor the 300 callers spelling their last name, 90.7% of the names were correctly \nretrieved from a list of 50,000 common last names. 95.7% of the names were in the \n\n\fEnglish Alphabet Recognition with Telephone Speech \n\n205 \n\nfI) \nQ) \n(,) \n~ \nQ) \n'-\n~ T\"\" \n(,) \n(,) \n\n0 \n\n0 -0 \n\n'-\nQ) \nJ:J \nE \n~ z \n\n<= -87-6-5-4-3-2-1 0 1 234 5 6 7>= 8 \n\nOffset from hand labels \n\nLO \n\no \n\nLO \nT\"\" \n\nfI) \nQ) \n(,) \n~ \nQ) \n'-\n~ T\"\" \n(,) \n(,) \n\n0 \n\n0 -0 \n\n'-\nQ) \nJ:J \nE \n~ z \n\nLO \n\no \n\n<= -87-6-5-4-3-2-1 0 1 2 3 4 5 6 7>= 8 \n\nOffset from hand labels \n\nFigure 2: Test set improvement in the placement of the [b]j[iy] boundary after \nsampling more training frames near that boundary. The top histogram shows the \ndifference between hand-labeled boundaries and the system's boundaries in 3 msec \nframes before adding extra boundary frames. The bottom histogram shows the \ndifference after adding the boundary frames. \n\n\f206 \n\nFanry, Cole, and Roginski \n\ntop three. \n\n4 DISCUSSION \n\nThe recognition system described in this paper classifies letters of the English al(cid:173)\nphabet produced by any speaker over telephone lines at 89% accuracy for spelled \nalphabets and retrieves names from a list of 50,000 with 91 % first choice accuracy. \nThe system has a number of characteristic features. We represent speech using an \nauditory model-Perceptual Linear Predictive (PLP) analysis. We perform explicit \nsegmentation of the speech signal into phonetic categories. Explicit segmentation \nallows us to use segment durations to discriminate letters, and to extract features \nfrom specific regions of the signal. Finally, speech knowledge is used to design a \nset of features that work best for English letters. We are currently analyzing errors \nmade by our system. The great advantage of our approach is that individual errors \ncan be analyzed, and individual features can be added to improve performance. \n\nAcknowledgements \n\nResearch supported by US WEST Advanced Technologies, APPLE Computer Inc., \nNSF, ONR, Digital Equipment Corporation and Oregon Advanced Computing In(cid:173)\nstitute. \n\nReferences \n\n[COLE 91] R. A. Cole, M. Fanty, M. Gopalakrishnan, and R. D. T. Janssen. \nSpeaker-independent name retrieval from spellings using a database of 50,000 \nnames. \nIn Proceedings of the IEEE International Conference on Acoustics, \nSpeech, and Signal Processing, 1991. \n\n[COLE 90] R. A. Cole, M. Fanty, Y. Muthusamy, and M. Gopalakrishnan. Speaker(cid:173)\nindependent recognition of spoken English letters. In Proceedings of the Inter(cid:173)\nnational Joint Conference on Neural Networks, San Diego, CA, 1990. \n\n[DALY 87] N. Daly. Recognition of words from their spellings: Integration of mul(cid:173)\n\ntiple knowledge sources. Master's thesis, Massachusetts Institute of Technology, \nMay, 1987. \n\n[FANTY 91] M. Fanty and R. A. Cole. Spoken letter recognition. In R. P. Lipp(cid:173)\n\nman, J. Moody, and D. S. Touretzky, editors, Advances in Neural Information \nProcessing Systems 3. San Mateo, CA: Morgan Kaufmann, 1991. \n\n[HERMANSKY 90] H. Hermansky. Perceptual Linear Predictive (PLP) analysis of \n\nspeech. J. Acoust. Soc. Am., 87(4):1738-1752, 1990. \n\n[LAMEL 86] L. Lamel, R. Kassel, and S. Seneff. Speech database development: De(cid:173)\n\nsign and analysis of the acoustic-phonetic corpus. In Proceedings of the DARPA \nSpeech Recognition Workshop, pages 100-110, 1986. \n\n[ROGINSKI 91] Krist Roginski. A neural network phonetic classifier for telephone \n\nspoken letter recognition. Master's thesis, Oregon Graduate Institute, 1991. \n\n\fPART IV \n\nLANGUAGE \n\n\f\f", "award": [], "sourceid": 550, "authors": [{"given_name": "Mark", "family_name": "Fanty", "institution": null}, {"given_name": "Ronald", "family_name": "Cole", "institution": null}, {"given_name": "Krist", "family_name": "Roginski", "institution": null}]}