{"title": "Pattern Playback in the 90s", "book": "Advances in Neural Information Processing Systems", "page_first": 827, "page_last": 834, "abstract": null, "full_text": "Pattern Playback in the '90s \n\nMalcolm Slaney \n\nInterval Research Corporation \n\n180 l-C Page Mill Road, \n\nPalo Alto, CA 94304 \nmalcolm@interval.com \n\nAbstract \n\nDeciding the appropriate representation to use for modeling human \nauditory processing is a critical issue in auditory science. While engi(cid:173)\nneers have successfully performed many single-speaker tasks with LPC \nand spectrogram methods, more difficult problems will need a richer \nrepresentation. This paper describes a powerful auditory representation \nknown as the correlogram and shows how this non-linear representation \ncan be converted back into sound, with no loss of perceptually impor(cid:173)\ntant information. The correlogram is interesting because it is a neuro(cid:173)\nphysiologically plausible representation of sound. This paper shows \nimproved methods for spectrogram inversion (conventional pattern \nplayback), inversion of a cochlear model, and inversion of the correlo(cid:173)\ngram representation. \n\n1 INTRODUCTIONl \n\nMy interest in auditory models and perceptual displays [2] is motivated by the problem of \nsound understanding, especially the separation of speech from noisy backgrounds and \ninterfering speakers. The correlogram and related representations are a pattern space \nwithin which sounds can be \"understood\" and \"separated\" [3][4]. I am therefore interested \nin resynthesizing sounds from these representations as a way to test and evaluate sound \nseparation algorithms, and as a way to apply sound separation to problems such as speech \nenhancement. The conversion of sound to a correlogram involves the intermediate repre(cid:173)\nsentation of a cochleagram, as shown in Figure 1, so cochlear-model inversion is \naddressed as one piece of the overall problem. \n\n1. Much of this work was performed by Malcolm Slaney, Daniel Naar and Rich(cid:173)\nard F. Lyon while all three were employed at Apple Computer. The mathematical \ndetails of this work were presented at the 1994ICASSP[I]. \n\n\f828 \n\nMalcolm Slaney \n\nWaveform \n\nj 1r--*T'T~\"\"\"I-~!\"'T!-~~-'-ooI-'1 ::::S:ec::ti :n:: h \n\nCochleagram \n\nCorrelogram \n\n.... t--'~~'--t.:lfga..~ l!!!Jnme \n\n------------~ \n\n------------~ \n\nFigure 1. Three stages in low-level auditory perception are shown here. Sound waves are con(cid:173)\nverted into a detailed representation with broad spectral bands, known as cochleagrams. The \ncorrelogram then summarizes the periodicities in the cochleagram with short-time autocorrela(cid:173)\ntion. The result is a perceptual movie synchronized to the acoustic signal. The two inversion \nproblems addressed in this work are indicated with arrows from right to left \n\nThere are three factors which can be used to judge the quality of an auditory model: psy(cid:173)\nchoacoustic comparisons, neurophysiological plausibility, and does it represent the per(cid:173)\nceptually relevant information? First, the correlogram has been shown to simply and \naccurately predict human pitch perception [5]. The neurophysiological basis for the corre(cid:173)\nlogram has not been found, but there are neural circuits performing the same calculation in \nthe mustached bat's echolocation system [6]. Finally, from an information representation \npoint of view, does the correlogram preserve the salient information? The results of this \npaper show that no information has been lost. Since the psychoacoustic, neurophysiologi(cid:173)\ncal, and information representation measures are all positive, perhaps the correlogram is \nthe basis of most auditory processing. \n\nThe inversion techniques described here are important because they allow us to readily \nevaluate the results of sound separation models that \"zero out\" unwanted portions of the \nsignal in the correlogram domain. This work extends the convex projection approach of \nIrino [7] and Yang [8] by considering a different cochlear model, and by including the cor(cid:173)\nrelogram inversion. The convex projection approach is well suited to \"filling in\" missing \ninformation. While this paper only describes the process for one particular auditory \nmodel, the techniques are equally useful for other models. \n\nThis paper describes three aspects of the problem: cochleagram inversion, conversion of \nthe correlogram into spectrograms, and spectrogram inversion. A number of reconstruc(cid:173)\ntion options are explored in this paper. Some are fast, while other techniques use time-con(cid:173)\nsuming iterations to produce reconstructions perceptually equivalent to the original sound. \nFast versions of these algorithms could allow us to separate a speaker's voice from the \nbackground noise in real time. \n\n2 COCHLEAGRAM INVERSION \n\nFigure 2 shows a block diagram of the cochlear model [9] that is used in this work. The \nbasis of the model is a bank of filters, implemented as a cascade of low-pass filters, that \nsplits the input signal into broad spectral bands. The output from each filter in the bank is \ncalled a channel. The energy in each channel is detected and used to adjust the channel \ngain, implementing a simple model of auditory sensitivity adaptation, or automatic gain \ncontrol (AGe). The half-wave rectifier (HWR) detection nonlinearity provides a wave(cid:173)\nform for each channel that roughly represents the instantaneous neural firing rate at each \nposition along the cochlea. \n\nE \n~---+ \n> \n~ \n\nCochlear \nFilterbank \n\nDetector \n(HWR) \n\nAdaptation \n\nor \nAGC \n\nFigure 2. Three stages of the simple cochlear model used in this paper are shown above. \n\n\fPattern Playback in the '90s \n\n829 \n\nThe cochleagram is converted back into sound by reversing the three steps shown in Fig(cid:173)\nure 2. First the AGe is divided out, then the negative portions of each cochlear channel are \nrecovered by using the fact that each channel is spectrally limited. Finally, the cochlear fil(cid:173)\nters are inverted by running the filters backwards, and then correcting the resulting spec(cid:173)\ntral slope. \n\nThe AGe stage in this cochlear model is controlled by its own output. It is a combination \nof a multiplicative gain and a simple first-order filter to track the history of the output sig(cid:173)\nnal. Since the controlling signal is directly available, the AGe can be inverted by tracking \nthe output history and then dividing instead of multiplying. The performance of this algo(cid:173)\nrithm is described by Naar [10] and will not be addressed here. It is worth noting that AGe \ninversion becomes more difficult as the level of the input signal is raised, resulting in more \ncompression in the forward path. \n\nThe next stage in the inversion process can be done in one of two ways. After AGC inver(cid:173)\nsion, both the positive values of the signal and the spectral extant of the signal are known. \nProjections onto convex sets [11], in this case defined by the positive values of the detec(cid:173)\ntor output and the spectral extant of the cochlear filters, can be used to find the original \nsignal. This is shown in the left half of Figure 3. Alternatively, the spectral projection filter \ncan be combined with the next stage of processing to make the algorithm more efficient. \nThe increased efficiency is due to better match between the spectral projection and the \ncochlear filterbank, and due to the simplified computations within each iteration. This is \nshown in the right half of Figure 3. The result is an algorithm that produces nearly perfect \nresults with no iterations at all. \n\nFilter \nBank \n\nInversion \n\nSpectral \nProjection \n\nTemporal \nProjection \n\nE \n-go \n~Qi \n0> \nL... __ ..... CI)~ \n\nC Other --110.. Inversion \nhannels-\n\nAGC \n\nInversion \n\n~ \n~ \n~ Q) \n~ \n8 \no \n\nFigure 3. There are two ways to use convex projections to recover the information lost \nby the detectors. The conventional approach is shown on the left. The right figure \nshows a more efficient approach where the spectral projection has been combined with \nthe filterbank inversion \n\nFinally, the multiple outputs from the cochlear filterbank are converted back into a single \nwaveform by correcting the phase and summing all channels. In the ideal case, each \ncochlear channel contains a unique portion of the spectral energy, but with a bit of phase \ndelay and amplitude change. For example, if we run the signal through the same filter the \nspectral content does not change much but both the phase delay and amplitude change will \nbe doubled. More interestingly, if we run the signal through the filter backwards, the for(cid:173)\nward and backward phase changes cancel out. After this phase correction, we can sum all \nchannels and get back the Original waveform, with a bit of spectral coloration. The spec(cid:173)\ntral coloration or tilt can be fixed with a simple filter. A more efficient approach to correct \nthe spectral tilt is to scale each channel by an appropriate weight before summing, as \nshown in Figure 4. The result is a perfect reconstruction, over those frequencies where the \ncochlear filters are non-zero. \n\nFigure 5 shows results from the cochleagram inversion procedure. An impulse is shown on \nthe left, before and after 10 iterations of the HWR inversion (using the algorithm on the \nright half of Figure 3). With no iterations the result is nearly perfect, except for a bit of \nnoise near the center. The overall curvature of the baseline is due to the fact that informa-\n\n\f830 \n\n~ \n\nc:UI \n~'S \n.s5 \n.... 0-\niIO \n\nTime \n\nReversed \nFilter for \nInversion \n\nIIR or FIR \nFilter to \nCorrect \n\nTilt \n\nE \n-0 .... \n~ \n;:'0> \n0> \nCJ)~ \n\n~ \n\nc:,a \nas;:, \n\u20ac,e. \n0>;:, \nitO \n\nMalcolm Slaney \n\nTime \n\nReversed \nFilter for \nInversion \n\nE \n-0 .... c:.g \n;:'0> \n0> \nCJ)~ \n\nFigure 4. Two approaches are shown here to invert the filterbank. The left diagram shows the \nnormal approach, the right figure shows a more efficient approach where the spectral-tilt filter \nis converted to a simple multiplication. \n\ntion near DC has been lost as it travels through the auditory system and there is no way to \nrecover it with the information that we have. A more interesting example is shown on the \nright. Here the word \"tap\" 1 has been reconstructed, with and without the AGC inversion. \nWith the AGe inversion the result is nearly identical to the original. The auditory system \nis very sensitive to onsets and quickly adapts to steady state sounds like vowels. It is inter(cid:173)\nesting to compare this to the reconstruction withoutAGC inversion. Without the AGC, the \nresult is similar to what the ear hears, the onsets are more prominent and the vowels are \ndeemphasized. This is shown in the right half of Figure 5. \n\nImpulse inversion \nwith no iterations \n\nImpulse iteration \nwith 10 iterations \n\n\"Tap\" reconstruction \"Tap\" Reconstruction \nwith AGC Inversion without AGC Inversion \n\nFigure 5. The cochlear reconstructions of an impulse and the word \"tap\" are shown here. The \nfirst and second reconstructions show an impulse reconstruction with and without iterations. \nThe third and fourth waveforms are the word \"tap\" with and without the AGe inversion. \n\n3 CORRELOGRAM INVERSION \n\nThe correlogram is an efficient way to capture the short-time periodicities in the auditory \nsignal. Many mechanical measurements of the cochlea have shown that the response is \nhighly non-linear. As the signal level changes there are large variations in the bandwidth \nand center frequency of the cochlear response. With these kinds of changes, it is difficult \nto imagine a system that can make sense of the spectral profile. This is especially true for \ndecisions like pitch determination and sound separation. \n\nBut through all these changes in the cochlear filters, the timing information in the signal is \npreserved. The spectral profile, as measured by the cochlea, might change, but the rate of \nglottal pulses is preserved. Thus I believe the auditory system is based on a representation \nof sound that makes short-time periodicities apparent. One such representation is the cor(cid:173)\nrelogram. The correlogram measures the temporal correlation within each channel, either \nusing FFfs which are most efficient in computer implementations, or neural delay lines \nmuch like those found in the binaural system of the owl. \n\n1. The syllable \"tap\", samples 14000 through 17000 of the \"trainldr5/fcdfll \nsxl06/sx106.adc\" utterance on the TIMIT Speech Database, is used in all voiced \nexamples in this paper. \n\n\fPattern Playback in the '90s \n\n831 \n\nThe process of inverting the correlogram is simplified by noting that each autocorrelation \nis related by the Fourier transform to a power spectrum. By combining many power spec(cid:173)\ntrums into a picture, the result is a spectrogram. This process is shown in Figure 6. In this \nway, a separate spectrogram is created for each channel. There are known techniques for \nconverting a spectrogram, which has amplitude information but no phase information, \nback into the original waveform. The process of converting from a spectrogram back into \na waveform is described in Section 4. The correlogram inversion process consists of \ninverting many spectrograms to form an estimate of a cochleagram. The cochleagram is \ninverted using the techniques described in Section 2. \n\nFrame 42 of Correlogram \n\nFrame 43 of Correlogram \n\nIFFTto get \nof spectrogram \n\nSpectrogram \n\nInversion ., \n\nOne line of Cochleagram \n\nTime \n\nTime \n\nFigure 6. Correlogram inversion is possible by noting that each row of the correlogram con(cid:173)\ntains the same information as a spectrogram of the same row of cochleagram output. By con(cid:173)\nverting the correlogram into many spectrograms, the spectrogram inversion techniques \ndescribed in Section 4 can be used. The lower horizontal stripe in the spectrogram is due to the \nnarrow passband of the cochlear channel. Half-wave rectification of the cochlear filter output \ncauses the upper horizontal stripes. \n\nOne important improvement to the basic method is possible due to the special characteris(cid:173)\ntics of the correlogram. The essence of the spectrogram inversion problem is to recover \nthe phase information that has been thrown away. This is an iterative procedure and would \nbe costly ifit had to be performed on each channel. Fortunately, there is quite a bit of over(cid:173)\nlap between cochlear channels. Thus the phase recovered from one channel can be used to \ninitialize the spectrogram inversion for the next channel. A difficulty with spectrogram \ninversion is that the absolute phase is lost. By using the phase from one channel to initial(cid:173)\nize the next, a more consistent set of cochlear channel outputs is recovered. \n\n4 SPECTROGRAM INVERSION \n\nWhile spectrograms are not an accurate model of human perception, an implementation of \na correlogram includes the calculation of many spectrograms. Mathematically, an autocor(cid:173)\nrelation calculation is similar to a spectrogram or short-time power spectrum. One column \nof a conventional spectrogram is related to an autocorrelation of a portion of the original \nwaveform ~y a Fourier transform (see Figure 6). Unfortunately, the final representation of \nboth spectrograms and autocorrelations is missing the phase information. The main task of \na spectrogram inversion algorithm is to recover a consistent estimate ofthe missing phase. \nThis process is not magical, it can only recover a signal that has the same magnitude spec(cid:173)\ntrum as the original spectrogram. But the consistency constraint on the time evolution of \nthe signal power spectrum also constrains the time evolution of the spectral phase. \n\n\f832 \n\nMalcolm Slaney \n\nThe basic procedure in spectrogram inversion [12] consists of iterating between the time \nand the frequency domains. Starting from the frequency domain, the magnitude but not \nthe phase is known. As an initial guess, any phase value can be used. The individual power \nspectra are inverse Fourier transformed and then summed to arrive at a single waveform. \nIf the original spectrogram used overlapping windows of data, the information from adja(cid:173)\ncent windows either constructively or destructively interferes to estimate a waveform. A \nspectrogram of this new data is calculated, and the phase is now retained. We know the \noriginal magnitude was correct. Thus we can estimate a better spectrogram by combining \nthe original magnitude information with the new phase information. It can be shown that \neach iteration will reduce the error. \n\nFigure 7 shows an outline of steps that can be used to improve the consistency of phase \nestimates during the first iteration. As each portion of the waveform is added to the esti(cid:173)\nmated signal, it is possible to add a linear phase so that each waveform lines up with the \nproceedings segments. The algorithm described in the paragraph above assumes an initial \nphase of zero. A more likely phase guess is to choose a phase that is consistent with the \nexisting data. The result with no iterations is a waveform that is often closer to the original \nthan that calculated assuming zero initial phase and ten iterations. \n\nThe total computational cost is minimized by combining these improvements with the ini(cid:173)\ntial phase estimates from adjacent channels of the correlogram. Thus when inverting the \nfirst channel of the correlogram, a cross-correlation is used to pick the initial phase and a \nfew more iterations insure a consistent result. After the first channel, the phase of the pro(cid:173)\nceeding channel is used to initialize the spectrogram inversion and only a few iterations \nare necessary to fine tune the waveform. \n\nReconstructions from \nsegments 1 through N \n\no \n\n400 \n\nCross Correlation \n\nRotated Segment N+ 1 \n\nhi\\;\\; I \n\n400 \n\no \n\nSegmentN+1 \n\nb;&/\\; I \n\n400 \n\no \n\nNew Reconstruction \n\n300 \n\nMaximize fit by \nchOOSing peak \n\no \n\nFigure 7. A procedure for adjusting the phase of new segments when inverting a spectrogram is \nshown above. As each new segment (bottom left) is converted from a power spectrum into a \nwaveform, a linear phase is added to maximize the fit with the existing segments (top left.) The \namount of rotation is determined by a cross correlation (middle). Adding the new segment with \nthe proper rotation (top right) produces the new waveform (bottom right.) \n\n\fPattern Playback in the '90s \n\n833 \n\n5 PUTTING IT TOGETHER \n\nThis paper has described two steps to convert a correlogram into a sound. These steps are \ndetailed below: \n\nI) \n\nFor each row of the correlogram: \na) \n\nb) \n\nc) \n\nConvert the autocorrelation data into power spectrum (Section \n3). \nUse spectrogram inversion (Section 4) to convert the spectro(cid:173)\ngrams into an estimate of cochlear channel output. \nAssemble the results of spectrogram inversion into an estimate \nof the cochleagram. \n\n2) \n\nInvert the cochleagram using the techniques described in Section 2. \n\nThis process is diagrammed in Figures I and 6. \n\n6 RESULTS \n\nFigure 8 shows the results of the complete reconstruction process for a 200Hz impulse \ntrain and the word \"tap.\" In both cases, no iterations were performed for either the spectro(cid:173)\ngram or filterbank inversion. More iterations reduce the spectral error, but do not make the \ngraphs look better or change the perceptual quality much. It is worth noting that the \"tap\" \nreconstruction from a correlogram looks similar to the cochleagram reconstruction with(cid:173)\nout the AGC (see Figure 5.) Reducing the level of the input signal, thus reducing the \namount of compression performed by the AGC, results in a correlogram reconstruction \nsimilar to the original waveform. \n\n0.02 r---------. \n\n0.05f\"'\"'\"!~----\"\"\"\" \n\n0.01 \no \n-0.01 \n\n-0.02\"'--~~~~~ \n50 100 150 200 \n\no \n\n-0.05 \" ' - - - - - - -.... \n\n1000 2000 3000 \n\no \n\nFigure 8. Reconstructions from the correlogram representation of an impulse train and the word \n\"tap\" are shown above. Reducing the input signal level, thus minimizing the effect of errors \nwhen inverting the AGe, produces results identical to the original \"tap.\" \n\nIt is important to note that the algorithms described in this paper are designed to minimize \nthe error in the mean-square sense. This is a convenient mathematical definition, but it \ndoesn't always correlate with human perception. A trivial example of this is possible by \ncomparing a waveform and a copy of the waveform delayed by lOms. Using the mean(cid:173)\nsquared error, the numerical error is very high yet the two waveforms are perceptually \nequivalent. Despite this, the results of these algorithms based on mean-square error do \nsound good. \n\n7 CONCLUSIONS \n\nThis paper has described several techniques that allow several stages of an auditory model \nto be converted back into sound. By converting each row of the correlogram into a spec(cid:173)\ntrogram, the spectrogram inversion techniques of Section 4 can be used. The special char(cid:173)\nacteristics of a correlogram described in Section 3 are used to make the calculation more \nefficient. Finally, the cochlear filterbank can be inverted to recover the original waveform. \nThe results are waveforms, perceptually identical to the original waveforms. \n\n\f834 \n\nMalcolm Slaney \n\nThese techniques will be especially useful as part of a sound separation system. I do not \nbelieve that our auditory system resynthesizes partial waveforms from the auditory scene. \nYet, all research systems generate separated sounds so that we can more easily perceive \ntheir success. More work is still needed to fine-tune these algorithm and to investigate the \nability to reconstruct sounds from partial correlograms. \n\nAcknowledgments \n\nI am grateful for the inspiration provided by Frank Cooper's work in the early 1950's on \npattern playback[13][14]. His work demonstrated that it was possible to convert a spectro(cid:173)\ngram, painted onto clear plastic, into sound. \nThis work in this paper was performed with Daniel Naar and Richard F. Lyon. We are \ngrateful for the help we have received from Richard Duda (San Jose State), Shihab \nShamma (U. of Maryland), Jim Boyles (The MathWorks) and Michele Covell (Interval \nResearch). \n\nReferences \n\n[1] Malcolm Slaney, D. Naar, R. F. Lyon, \"Auditory model inversion for sound separa(cid:173)\ntion,\" Proc. of IEEE ICASSP, Volume II, pp. 77-80, 1994. \n[2] M. Slaney and R. F. Lyon, \"On the importance of time-A temporal representation of \nsound,\" in Visual Representations of Speech Signals, eds. M. Cooke, S. Beet, and M. \nCrawford, J. Wiley and Sons, Sussex, England, 1993. \n[3] R. F. Lyon, \"A computational model of binaural localization and separation,\" Proc. of \nIEEE ICASSP, 1148-1151, 1983. \n[4] M. Weintraub, \"The GRASP sound separation system,\" Proc. of IEEE ICASSP, pp. \n18A.6.1-18A.6.4, 1984. \n[5] D. Hennes, \"Pitch analysis,\" in Visual Representations of Speech Signals, eds. M. \nCooke, S. Beet, and M. Crawford, J. Wiley and Sons, Sussex, England, 1993. \n[6] N. Suga, \"Cortical computational maps for auditory imaging,\" Neural Networks, 3, 3-\n21, 1990. \n[7] T. lrino, H. Kawahara, \"Signal reconstruction from modified auditory wavelet trans(cid:173)\nfonn,\" IEEE Trans. on Signal Processing, 41,3549-3554, Dec. 1993. \n[8] x. Yang, K. Wang, and S. Sharnma, \"Auditory representations of acoustic signals,\" \nIEEE Trans. on Information Theory, 38, 824-839, 1992. \n[9] R. F. Lyon, \"A computational model of filtering, detection, and compression in the \ncochlea,\" Proc. of the IEEE ICASSP, 1282-1285,1982. \n[10] D. Naar, \"Sound resynthesis from a correlogram,\" San Jose State University, Depart(cid:173)\nment of Electrical Engineering, Technical Report #3, May 1993. \n[11] R. W. Papoulis, \"A new algorithm in spectral analysis and band-limited extrapola(cid:173)\ntion,\" IEEE Trans. Circuits Sys., vol. 22, 735, 1975. \n[12] D. Griffin and J. Lim, \"Signal estimation from modified short-time Fourier trans(cid:173)\nfonn,\" IEEE Trans. on Acoustics, Speech, and Signal Processing, 32, 236-242, 1984. \n[13] F. S. Cooper, \"Some Instrumental Aids to Research on Speech,\" Report on the Fourth \nAnnual Round Table Meeting on Linguistics and Language Teaching, Georgetown Uni(cid:173)\nversity Press, 46-53, 1953. \n[14] F. S. Cooper, \"Acoustics in human communications: Evolving ideas about the nature \nof speech,\" J. Acoust. Soc. Am., 68(1),18-21, July 1980. \n\n\f", "award": [], "sourceid": 978, "authors": [{"given_name": "Malcolm", "family_name": "Slaney", "institution": null}]}