{"title": "Combining Visual and Acoustic Speech Signals with a Neural Network Improves Intelligibility", "book": "Advances in Neural Information Processing Systems", "page_first": 232, "page_last": 239, "abstract": null, "full_text": "232 \n\nSejnowski, Yuhas, Goldstein and Jenkins \n\nCombining  Visual  and  Acoustic  Speech  Signals \nwith  a  Neural  Network  Improves  Intelligibility \n\nT .J. Sejnowski \nThe Salk Institute \nand \nDepartment of Biology \nThe University of \nCalifornia at San  Diego \nSan Diego,  CA 92037 \n\nB.P. Yuhas \n\nM.H.  Goldstein, Jr. \nDepartment of Electrical \n\nand Computer \nEngineering \n\nThe Johns Hopkins \n\nUniversity \n\nBaltimore,  MD  21218 \n\nABSTRACT \n\nR.E. Jenkins \nThe Applied  Physics \nLaboratory \nThe Johns Hopkins \nUniversity \nLaurel,  MD  20707 \n\nAcoustic speech recognition degrades in the presence of noise.  Com(cid:173)\npensatory  information  is  available  from  the  visual  speech  signals \naround  the  speaker's  mouth.  Previous  attempts  at  using  these \nvisual speech signals to improve automatic speech  recognition sys(cid:173)\ntems have combined the acoustic and visual speech information at a \nsymbolic level  using heuristic rules.  In this paper, we  demonstrate \nan  alternative  approach  to fusing  the  visual  and  acoustic  speech \ninformation  by  training  feedforward  neural  networks  to  map  the \nvisual signal onto the corresponding short-term spectral amplitude \nenvelope  (STSAE)  of the  acoustic  signal.  This  information  can \nbe  directly  combined  with  the  degraded  acoustic  STSAE.  Signif(cid:173)\nicant  improvements  are  demonstrated  in  vowel  recognition  from \nnoise-degraded  acoustic signals.  These results are compared to the \nperformance of humans, as well  as other pattern matching and es(cid:173)\ntimation algorithms. \n\nINTRODUCTION \n\n1 \nCurrent automatic speech recognition systems rely almost exclusively on the acous(cid:173)\ntic speech signal, and as a consequence, these systems often perform poorly in noisy \n\n\fCombining Visual and Acoustic Speech Signals \n\n233 \n\nenvironments.  To compensate for  noise-degradation of the acoustic signal,  one  can \neither attempt to remove the noise from the acoustic :;ignal or supplement the acous(cid:173)\ntic signal with other sources  of speech  information.  One such  source  is  the  visible \nmovements  of the  mouth.  For  humans,  visual  speech  signals  can  improve  speech \nperception when the acoustic signal is degraded by noise (Sumby and Pollack, 1954) \nand  can  serve  as  a  source  of speech  information  when  the  acoustic  signal  is  com(cid:173)\npletely  absent  through lipreading.  How  can  these  visual speech signals  be used  to \nimprove  the  automatic recognition of speech? \n\nOne  speech recognition  system that has extensively  used  the visual  speech  signals \nwas  developed  by  Eric  Petajan  (1987).  For  a  limited  vocabulary,  Petajan demon(cid:173)\nstrated that the visual speech signals can be used to significantly improve automatic \nspeech  recognition  compared  to the acoustic  recognition  alone.  The system relied \nupon a codebook of images that were  used to translate incoming images into corre(cid:173)\nsponding  symbols.  These symbol  strings were  then  compared  to stored sequences \nrepresenting different words in the vocabulary.  This categorical treatment of speech \nsignals  is  required  because of the  computational  limitations  of currently  available \ndigital serial hardware. \n\nThis  paper  proposes  an  alternative  method  for  processing  visual  speech  signals \nbased on analog computation in a distributed network architecture.  By  using many \ninterconnected processors working in  parallel large amounts of data can be handled \nconcurrently.  In  addition  to speeding  up  the computation,  this  approach does  not \nrequire  segmentation  in  the  early stages of processing;  rather,  analog signals from \nthe  visual  and  auditory  pathways  flow  through  networks  in  real  time  and  can  be \ncombined  directly. \nResults are presented from  a series of experiments that use neural networks to pro(cid:173)\ncess  the  visual  speech  signals  of two  talkers.  In  these  preliminary  experiments, \nthe  results  are limited  to static images of vowels.  We  demonstrate  that  these  net(cid:173)\nworks  are  able  to extract speech information from the visual images,  and that this \ninformation can be used  to improve  automatic vowel  recognition. \n\n2  VISUAL AND  ACOUSTIC  SPEECH SIGNALS \nThe acoustic speech signal can be modeled as the response of the vocal tract filter to \na sound source (Fant,  1960).  The resonances of the vocal tract are called formants. \nThey often appear as peaks in the short-term power spectrum, and are sufficient  to \nidentify  the  individual  vowels  (Peterson  and  Barney,  1953).  The  overall  shape  of \nthe short-time spectra is  important for  general speech perception  (Cole,  1980). \n\nThe  configuration  of the  articulators  define  the  shape  of the  vocal  tract  and  the \ncorresponding resonance characteristics of the filter.  While some of the articulators \nare  visible  on  the face  of the speaker  (e.g.,  the  lips,  teeth  and  sometimes  the  tip \nof the  tongue),  others  are  not.  The  contribution  of the  visible  articulators  to the \nacoustic signal results in  speech sounds that are much  more susceptible to acoustic \nnoise  distortion  than  are  the contributions from  the  hidden  articulators  (Petajan, \n1987),  and  therefore,  the  visual  speech  signal  tends  to  complement  the  acoustic \n\n\f234 \n\nSejnowski, Yuhas, Goldstein and Jenkins \n\nsignal.  For  example,  the  visibly  distinct  speech  sounds  Ibl  and  Ikl  are  among \nthe first  pairs to be  confused  when  presented  acoustically  in  the  presence of noise. \nBecause of this complementary structure, the perception of speech in noise is greatly \nimproved  when  both speech  signals  are  present.  How  and  at  what  level  are  these \ntwo speech signals being combined? \nIn  previous  attempts at  using  the  visual  speech  signals,  the  information from  the \nvisual  signal  was  incorporated  into  the  recognition  system  after  the  signals  were \ncategorized (Petajan, 1987).  In the approach taken here,  visual signals will be used \nto resolve  ambiguities  in the  acoustic  signal  before  either is  categorized.  By  com(cid:173)\nbining these two sources of information at an early stage of processing, it is possible \nto reduce  the number of erroneous  decisions  made  and  increase  the  amount  of in(cid:173)\nformation  passed to later stages of processing (Summerfield,  1987).  The additional \ninformation  provided  by  the visual signal can serve  to constrain the possible inter(cid:173)\npretations of an ambiguous acoustic signal, or it can serve as  an alternative source \nof speech  information  when  the  acoustical  signal  is  heavily  noise-corrupted.  In ei(cid:173)\nther  case,  a  massive  amount  of computation  must  be  performed  on  the  raw  data. \nNew  massively-parallel  architectures  based  on  neural  networks  and  new  training \nprocedures have  made  this approach feasible. \n\nINTERPRETING THE VISUAL SIGNALS \n\n3 \nIn our approach,  the visual signal was mapped directly into an  acoustic representa(cid:173)\ntion closely  related to the vocal tract's transfer function  (Summerfield,  1987).  This \nrepresentation  allowed  the visual  signal  to  be fused  with  the  acoustic signal  prior \nto any symbolic encoding. \nThe  visual  signals  provide  only  a  partial  description  of  the  vocal  tract  transfer \nfunction  and  that description is  usually  ambiguous.  For  a given  visual  signal there \nare  many  possible  configurations  of the  full  vocal  tract,  and  consequently  many \npossible  corresponding acoustic signals.  The goal  was  to define  a  good estimate of \nthat acoustic signal from the visual signal and then use  that estimate in conjunction \nwith any  residual acoustic information. \nThe speech  signals  used  in  these  experiments  were  obtained from  a  male  speaker \nwho was video taped while seated facing the camera, under well-lit conditions.  The \nvisual and acoustic signals were then transferred and stored on laser disc (Bernstein \nand Eberhardt,  1986), which  allowed  the  access of individual video frames  and the \ncorresponding sound track.  The NTSC video standard is based upon 30 frames per \nsecond  and  words  are  preserved  as  a  series of frames  on  the  laser  disc.  A  data set \nwas  constructed of 12  examples of 9 different  vowels  (Yuhas  et  al.,  1989). \nA  reduced  area-of-interest  in  the  image  was  automatically  defined  and  centered \naround the mouth.  The resulting sub-image was sampled to produce  a topograph(cid:173)\nically  accurate  image  of 20  x  25  pixels  that  would  serve  to  represent  the  visual \nspeech  signal.  While  not  the  most  efficient  encoding  one  could  use,  it  is  faithful \nto the  parallel  approach  to  computation  advocated  here  and  represents  what  one \nmight observe  through  an  array of sensors. \n\n\fCombining Visual and Acoustic Speech Signals \n\n235 \n\nAlong with each video frame on the laser disc there is  33 ms of acoustic speech.  The \nrepresentation chosen for  the acoustic output structure was  the short-time spectral \namplitude envelope (STSAE) of the acoustic signal, because it is essential to speech \nrecognition and also closely related to the vocal tract's transfer function.  It can be \ncalculated  from  the short-term power spectrum of the acoustic signal.  The speech \nsignal was sampled  and cepstral analysis was  used to produced  a smooth envelope \nof the original power spectrum that could  be sampled at 32 frequencies. \n\nFigure 1:  Typical lip  images presented to the network. \n\nThree-layered feedforward  networks with non-linear units were used  to perform the \nmapping.  A  lip  image  was  presented  across  500  input  units,  and  an  estimated \nSTSAE  was  produced  across  32  output  units.  Networks  with  five  hidden  units \nwere  found  to  provide  the  necessary  bandwidth  while  minimizing  the  effects  of \nover-learning.  The standard backpropagation  technique  was  used  to  compute  the \nerror  gradients  for  training  the  network.  However,  instead  of using  a  fixed-step \nsteepest-descent  algorithm  for  updating  the  weights,  the  error  gradient  was  used \nin  a  conjugate-gradient  algorithm.  The weights  were  changed  only  after  all  of the \ntraining patterns were  presented. \n\n4 \n\nINTEGRATING THE VISUAL AND ACOUSTIC SPEECH \nSIGNALS \n\nTo evaluate the spectral estimates,  a feedforward  network was  trained to recognize \nvowels  from  their  STSAE's.  With  no  noise  present,  the  trained  network  could \ncorrectly  categorized  100% of the 54  STSAE's in  its training set:  thus serving as  a \nperfect recognizer for this data.  The vowel recognizer was then presented with speech \ninformation  through  two  channels,  as  shown  in  Fig.  2.  The path on  the  bottom \nrepresents the information obtained from the acoustic signal, while the path on the \ntop provides information obtained from  the corresponding visual speech signal. \n\nTo assess the performance of the recognizer  in  noise,  clean spectral envelopes were \nsystematically  degraded  by  noise  and  then  presented  to  the  recognizer.  In  this \nparticular  condition,  no  visual  input  was  given  to  the  network.  The  noise  was \nintroduced by  adding a normalized random vector  to the STSAE.  Noise  corrupted \nvectors  were  produced  at  3  dB  intervals  from  -12  dB  to  24  dB.  At  each  step  6 \ndifferent  vectors  were  produced,  and  the  performance  reported  was  the  average. \nFig.  3 shows the recognition rates as  a function  of the speech-to- noise  ratio.  At a \nspeech-to-noise ratio of -12  dB,  the recognizer  was  operating at chance or  11.1%. \n\nNext,  a  network  trained  to estimate  the spectral envelopes from  images  was  used \n\n\f236 \n\nSejnowski, Yuhas, Goldstein and Jenkins \n\nto provide an  independent STSAE input into the recognizer  (along the  top of Fig. \n2).  This network  was  not trained on  any  of the data that was  used  in  training the \nvowel recognizer.  The task remained  to combine these two  STSAE's. \n\nSTSAE  estimated \nfrom  the visual signal \n\nVisual \nSpeech \nSignal \n\nNeural  ~ \n\n> l...-N_e_t\",_or_k--JF==~: \n\n_ \n\n~ ~ ==\u00bb  Recognizer \n~ Combined \n\nSTSAE's \n\nAcoustic \nSpeech \nSignal \n\n>1f==\u00bb b STSAE  of  the \n\nAcoustic signal \n\nNoise \n\nFigure 2:  A vowel recognizer that integrates the acoustic and visual speech signals. \n\nWe  considered three different ways of combining the estimates obtained from visual \nsignals  with  the  noised  degraded  acoustic  envelopes.  The  first  approach  was  to \nsimply  average  the  two  envelopes,  which  proved  to  be  less  than  optimal.  The \nrecognizer was able to identify 55.6% ofthe STSAE estimated from the visual signal, \nbut when the visual estimate was combined with the noise degraded acoustic signal \nthe recognizer was only  capable of 35% at  a SIN  of -12  dB.  Similarly,  at  very  high \nsignal-to-noise ratios, the combined input produced poorer results than the acoustic \nsignal  alone  provided.  To  correct  for  this,  the  two  inputs  needed  to  be  weighted \naccording  to  the  relative  amount  of information  available  from  each  source.  A \nweighting factor  was  introduced which  was  a function  of speech-to-noise: \n\na  SViltJll1  +  (1  - a) SAcotJ,tic \n\n(1) \n\nThe optimal value for  the parameter a  was  found  empirically to vary linearly with \nthe  speech-to-noise  ratio  in  dB.  The  value  for  a  ranged  from  approximately  0.8 \nat SIN  of -12dB  to 0.0  at 24  dB.  The results obtained from  using  the  a  weighted \naverage are shown in  Fig.  3. \nThe  third  method  used  to fuse  the  two  STSAE's  was  with  a second-order  neural \nnetwork  (Rumelhart  et  al.  1986).  Sigma-pi networks were  trained  to take  in  noise(cid:173)\ndegraded acoustic envelopes and estimated envelopes from the corresponding visual \nspeech  signal.  The networks were  able  to recreate  the  noise-free  acoustic envelope \nwith greater accuracy than any of the other methods, as measured by mean squared \nerror.  This increased accuracy did not however translate into improved recognition \nrates. \n\n\fCombining Visual and Acoustic Speech Signals \n\n237 \n\n100 \n\n80 \n\n60 \n\n40 \n\n20 \n\n~ \n'-\" \n\n.... \n\n0 \n<l) \nL.. \nL.. \n\n0 u \n\n0 \n-15 \n\n-9 \n\n-3 \n\n3 \n\n9 \nSIN  (dB) \n\n15 \n\n21 \n\n27 \n\nFigure 3:  The visual contribution to speech recognition  in  noise.  The lower  curve \nshows  the  performance  of the  recognizer  under  varying  signal-to-noise  conditions \nusing only  the acoustic  channel.  The top curve shows the final  improvement  when \nthe two  channels were combined using the a  weighted average. \n\n5  COMPARING PERFORMANCE \nThe performance of the network was compared to more traditional signal-processing \ntechniques. \n\n5.1  K-NEAREST NEIGHBORS \n\nIn  this first  comparison,  an estimate of the STSAE was obtained using  a  k-nearest \nneighbors  approach.  The images  in  the  training set  were  stored  along  with  their \ncorresponding  STSAE  calculated  from  the  acoustic  signal.  These  images  served \nas  the  data  base  of stored  templates.  Individual  images  from  the  test  set  were \ncorrelated against  all  of the stored  images and  the  closest  k  images  were  selected. \nThe  acoustic  STSAE  corresponding  to  the  k  selected  images  were  then  averaged \nto produce an estimate of the STSAE  corresponding  to the  test image.  Using this \nprocedure for  various values of k,  average MSE was calculated for  the test set.  This \nprocedure was  then repeated with  the test  and training set reversed. \n\nFor values ofk between 2 and 6 the k-nearest neighbor estimator was able to produce \nSTSAE  estimates  with  approximately  the same  accuracy  as  the  neural  networks. \nThose  networks  evaluated  after  500  training  epochs  produced  estimates  with  9% \nmore error than the KNN  approach,  while  those  weights corresponding to the net(cid:173)\nworks'  best performance, as defined above,  produced estimates with 5%  less  error. \n\n5.1.1  PRINCIPAL  COMPONENT ANALYSIS \n\nA second  method of comparison was  to obtain an STSAE estimate using  a  combi(cid:173)\nnation  of optimal linear  techniques.  The first  step was  to encode the images using \na  Hotelling  transform,  which  produces  an  optimal  encoding  of an  image  with  re(cid:173)\nspect to a least-mean-squared error.  The encoded image  Yi  was computed from the \n\n\f238 \n\nSejnowski, Yuhas, Goldstein and Jenkins \n\nnormalized image  %i  using \n\n(2) \n\nwhere  m1:  was  the  mean  image.  A  was  a  transformation  matrix whose  rows  were \nthe five  largest  eigenvectors of the covariance matrix of the images.  The vector  Yi \nrepresents  the image  as do the hidden units of the neural network. \nThe second  step  was  to find  a  mapping from  the  encoded  image  vector  Yi  to the \ncorresponding  short-term spectral envelope  Si  using  a  linear  least-squares fit.  For \nthe  Yi'S  calculated  above,  a  B  was  found  that  provided  the  best  estimate  of the \ndesired  Si: \n\n(3) \n\nIf we  think of the matrix A  as  corresponding to the weights from the input layer to \nthe hidden  units, then  B  maps  the hidden  units to the output units. \nThe networks  trained  to  produce  STSAE  estimates were  far  superior  to those  ob(cid:173)\ntained  using  the  coefficients  of A  and  B.  This  was  true not  only  for  the  training \ndata from  which  A  and  B  were  calculated,  but  also  for  the  test  data set.  When \ncompared  to networks  trained for  500  epochs,  the  networks  produced  estimates of \nthe  STSAE's that were  46%  better on  the training set  and  12%  better on  the  test \nset. \n\n6  CONCLUSION \nHumans  are  capable  of combining  information  received  through  distinct  sensory \nchannels  with  great  speed  and  ease.  The  combined  use  of the  visual  and  acous(cid:173)\ntic speech  signals  is just one example of integrating information  across  modalities. \nSumby  and  Pollack  (1954)  have  shown  that  the  relative  improvement  provided  by \nthe  visual  signal  varies  with  the  signal-to-noise  ratio  of the  acoustic  signal.  By \ncombining the speech information available from  the two speech signals before cat(cid:173)\negorizing,  we  obtained performance  that was  comparable  to that demonstrated by \nhumans. \nWe  have shown that visual and acoustic speech information can be effectively fused \nwithout  requiring  categorical  preprocessing.  The low-level  integration  of the  two \nspeech signals  was  particularly useful when  the signal-to-noise ratio ranged from 3 \ndB  to  15  dB,  where  the  combined  signals  were  recognized  with  a greater  accuracy \nthan either of the two  component  signals  alone.  In  contrast,  an  independent  cate(cid:173)\ngorical decisions on each channel would have  required additional information in the \nform of ad hoc rules to produce the same level of performance. \nLip  reading research  has  traditionally focused  on  the identification  and  evaluation \nof visual features  (Montgomery  and  Jackson,  1983).  Reducing  the original speech \nsignals  to  a  finite  set  of predefined  parameters  or  to  discrete  symbols  can  waste \na  tremendous  amount  of information.  For  an  automatic  recognition  system  this \ninformation may prove to be useful  at a later stage of processing.  In our approach, \nspeech information in the visual signal is accessed without requiring discrete feature \nanalysis or making categorical decisions. \n\n\fCom bining Visual and Acoustic Speech Signals \n\n239 \n\nThis line of research has consequences for  other problems, such as  target identifica(cid:173)\ntion  based on  multiple  sensors.  For example,  the same problems  arise  in  designing \nsystems that combine radar and infrared data.  Mapping into a common representa(cid:173)\ntion  using  neural network models  could  also  be  applied  to these problem domains. \nThe key  insight  is  to combine  this  information  at  a  stage  prior  to  categorization. \nNeural network learning procedures allow systems to be constructed for  performing \nthe mappings as long as sufficient  data are  available  to train the  network. \n\nAcknowledgements \n\nThis research was supported by grant AFOSR-86-0256 from the Air  Force Office  of \nScientific Research and  by  the Applied  Physics  Laboratory's IRAD. \n\nReferences \n\nBernstein, L.E.  and Eberhardt, S.P.  (1986).  Johns  Hopkins  Lipreading  Corpus  I-II, \nJohns Hopkins  University,  Baltimore,  MD \nCole,  R.A.  (1980).  (Ed.)  Perception  and  Production  of Fluent  Speech,  Lawrence \nErlbaum Assoc,  Publishers,  Hillsdale,  NJ \nFant, G.  (1960).  Acoustic  Theory  of Speech  Production.  Mouton & Co., Publishers, \nThe Hague,  Netherlands \nMontgomery,  A.  and  Jackson,  P.L.  (1983).  Physical  Characteristics  of  the  lips \nunderlying vowel  lipreading.  J. Acoust.  Soc.  Am.  73,  2134-2144. \nPetajan,  E.D.  (1987).  An  improved  Automatic  Lipreading  System  To  Enhance \nSpeech Recognition.  Bell Laboratories  Technical Report No.  11251-871012-111TM. \n\nPeterson,  G.E.  and  Barney,  H.L.  (1952).  Control Methods Used  in  a Study of the \nVowels.  J. Acoust.  Soc.  Am.  24,  175-184. \nRumelhart, D.E., Hinton, G.E. and Williams, R.J. (1986).  Learning internal repre(cid:173)\nsentations by  error  propagation.  In:  D.E.  Rumelhart  and J.L.  McClelland.  (Eds.) \nParallel Distributed Processing  in  the  Microstructure  of Cognition:  Vol  1.  Founda(cid:173)\ntions MIT Press,  Cambridge,  MA \nSumby,  W.H.  and  Pollack,  I. (1954).  Visual  Contribution to Speech  Intelligibility \nin  Noise.  J. Acoust.  Soc.  Am.  26,  212-215. \nSummerfield,  Q.(1987).  Some  preliminaries  to a  comprehensive  account  of audio(cid:173)\nvisual  speech  perception.  In:  B.  Dodd  and  R.  Campbell  (Eds. )  Hearing  by  Eye: \nThe  Pschology  of Lip-Reading,  Lawrence-Erlbaum Assoc,  Hillsdale,  NJ. \n\nYuhas,  B.P., Goldstein, M.H. Jr.  and Sejnowski, T.J. (1989).  Integration of Acous(cid:173)\ntic  and  Visual Speech  Signals  Using  Neural  Networks.  IEEE Comm Magazine  27, \nNovember 65-71. \n\n\f", "award": [], "sourceid": 290, "authors": [{"given_name": "Terrence", "family_name": "Sejnowski", "institution": null}, {"given_name": "Ben", "family_name": "Yuhas", "institution": null}, {"given_name": "Moise", "family_name": "Goldstein", "institution": null}, {"given_name": "Robert", "family_name": "Jenkins", "institution": null}]}