{"title": "Unsupervised feature learning for audio classification using convolutional deep belief networks", "book": "Advances in Neural Information Processing Systems", "page_first": 1096, "page_last": 1104, "abstract": "In recent years, deep learning approaches have gained significant interest as a way of building hierarchical representations from unlabeled data. However, to our knowledge, these deep learning approaches have not been extensively studied for auditory data. In this paper, we apply convolutional deep belief networks to audio data and empirically evaluate them on various audio classification tasks. For the case of speech data, we show that the learned features correspond to phones/phonemes. In addition, our feature representations trained from unlabeled audio data show very good performance for multiple audio classification tasks. We hope that this paper will inspire more research on deep learning approaches applied to a wide range of audio recognition tasks.", "full_text": "Unsupervised feature learning for audio classi\ufb01cation\n\nusing convolutional deep belief networks\n\nHonglak Lee\n\nYan Largman\nPeter Pham\nComputer Science Department\n\nStanford University\nStanford, CA 94305\n\nAndrew Y. Ng\n\nAbstract\n\nIn recent years, deep learning approaches have gained signi\ufb01cant interest as a\nway of building hierarchical representations from unlabeled data. However, to\nour knowledge, these deep learning approaches have not been extensively stud-\nied for auditory data.\nIn this paper, we apply convolutional deep belief net-\nworks to audio data and empirically evaluate them on various audio classi\ufb01cation\ntasks. In the case of speech data, we show that the learned features correspond to\nphones/phonemes. In addition, our feature representations learned from unlabeled\naudio data show very good performance for multiple audio classi\ufb01cation tasks.\nWe hope that this paper will inspire more research on deep learning approaches\napplied to a wide range of audio recognition tasks.\n\nIntroduction\n\n1\nUnderstanding how to recognize complex, high-dimensional audio data is one of the greatest chal-\nlenges of our time. Previous work [1, 2] revealed that learning a sparse representation of auditory\nsignals leads to \ufb01lters that closely correspond to those of neurons in early audio processing in mam-\nmals. For example, when sparse coding models are applied to natural sounds or speech, the learned\nrepresentations (basis vectors) showed a striking resemblance to the cochlear \ufb01lters in the auditory\ncortex. In related work, Grosse et al. [3] proposed an ef\ufb01cient sparse coding algorithm for auditory\nsignals and demonstrated its usefulness in audio classi\ufb01cation tasks.\nHowever, the proposed methods have been applied to learn relatively shallow, one-layer representa-\ntions. Learning more complex, higher-level representation is still a non-trivial, challenging problem.\nRecently, many promising approaches have been proposed to learn the processing steps of the \u201csec-\nond stage and beyond\u201d [4, 5, 6, 7, 8]. These \u201cdeep learning\u201d algorithms try to learn simple features\nin the lower layers and more complex features in the higher layers. However, to the best of our\nknowledge, these \u201cdeep learning\u201d approaches have not been extensively applied to auditory data.\nThe deep belief network [4] is a generative probabilistic model composed of one visible (observed)\nlayer and many hidden layers. Each hidden layer unit learns a statistical relationship between the\nunits in the lower layer; the higher layer representations tend to become more complex. The deep\nbelief network can be ef\ufb01ciently trained using greedy layerwise training, in which the hidden layers\nare trained one at a time in a bottom-up fashion [4]. Recently, convolutional deep belief networks [9]\nhave been developed to scale up the algorithm to high-dimensional data. Similar to deep belief\nnetworks, convolutional deep belief networks can be trained in a greedy, bottom-up fashion. By\napplying these networks to images, Lee et al. (2009) showed good performance in several visual\nrecognition tasks [9].\nIn this paper, we will apply convolutional deep belief networks to unlabeled auditory data (such as\nspeech and music) and evaluate the learned feature representations on several audio classi\ufb01cation\ntasks. In the case of speech data, we show that the learned features correspond to phones/phonemes.\nIn addition, our feature representations outperform other baseline features (spectrogram and MFCC)\n\n1\n\n\ffor multiple audio classi\ufb01cation tasks. In particular, our method compares favorably with other state-\nof-the-art algorithms for the speaker identi\ufb01cation task. For the phone classi\ufb01cation task, MFCC\nfeatures can be augmented with our features to improve accuracy. We also show for certain tasks\nthat the second-layer features produce higher accuracy than the \ufb01rst-layer features, which justi\ufb01es\nthe use of deep learning approaches for audio classi\ufb01cation. Finally, we show that our features give\nbetter performance in comparison to other baseline features for music classi\ufb01cation tasks. In our\nexperiments, the learned features often performed much better than other baseline features when\nthere was only a small number of labeled training examples. To the best of our knowledge, we are\nthe \ufb01rst to apply deep learning algorithms to a range of audio classi\ufb01cation tasks. We hope that this\npaper will inspire more research on deep learning approaches applied to audio recognition tasks.\n\n2 Algorithms\n2.1 Convolutional deep belief networks\n\nWe \ufb01rst brie\ufb02y review convolutional restricted Boltzmann machines (CRBMs) [9, 10, 11] as building\nblocks for convolutional deep belief networks (CDBNs). We will follow the formulation of [9] and\nadapt it to a one dimensional setting. For the purpose of this explanation, we assume that all inputs\nto the algorithm are single-channel time-series data with nV frames (an nV dimensional vector);\nhowever, the formulation can be straightforwardly extended to the case of multiple channels.\nThe CRBM is an extension of the \u201cregular\u201d RBM [4] to a convolutional setting, in which the weights\nbetween the hidden units and the visible units are shared among all locations in the hidden layer.\nThe CRBM consists of two layers: an input (visible) layer V and a hidden layer H. The hidden\nunits are binary-valued, and the visible units are binary-valued or real-valued.\nConsider the input layer consisting of an nV dimensional array of binary units. To construct the\nhidden layer, consider K nW -dimensional \ufb01lter weights W K (also referred to as \u201cbases\u201d throughout\nthis paper). The hidden layer consists of K \u201cgroups\u201d of nH-dimensional arrays (where nH (cid:44)\nnV \u2212 nW + 1) with units in group k sharing the weights W k. There is also a shared bias bk for each\ngroup and a shared bias c for the visible units. The energy function can then be de\ufb01ned as:\n\nE(v, h) = \u2212 K(cid:88)\nnH(cid:88)\nnW(cid:88)\ni \u2212 K(cid:88)\nnV(cid:88)\n\nnH(cid:88)\n\nv2\n\nk=1\n\nr=1\n\nj=1\n\n1\n2\n\nhk\nj W k\n\nnW(cid:88)\n\nE(v, h) =\n\nnV(cid:88)\n\ni=1\n\nbk\n\nr vj+r\u22121 \u2212 K(cid:88)\nnH(cid:88)\nr vj+r\u22121 \u2212 K(cid:88)\n\nhk\nj W k\n\nk=1\n\nj=1\n\nbk\n\nj \u2212 c\nhk\nnH(cid:88)\n\nnV(cid:88)\n\ni\n\nk=1\n\nj=1\n\nr=1\n\nk=1\n\nj=1\n\ni=1\n\nSimilarly, the energy function of CRBM with real-valued visible units can be de\ufb01ned as:\n\nvi.\n\n(1)\n\nj \u2212 c\nhk\n\nvi.\n\n(2)\n\nThe joint and conditional probability distributions are de\ufb01ned as follows:\n\n1\nZ\n\nexp(\u2212E(v, h))\n\nP (hk\n\nP (v, h) =\nj = 1|v) = sigmoid(( \u02dcW k \u2217v v)j + bk)\n\nP (vi = 1|h) = sigmoid((cid:88)\nP (vi|h) = N ormal((cid:88)\n\n(W k \u2217f hk)i + c)\n\nk\n\n(for binary visible units)\n\n(W k \u2217f hk)i + c, 1)\n\n(for real visible units),\n\n(3)\n\n(4)\n\n(5)\n\n(6)\n\nk\n\nwhere \u2217v is a \u201cvalid\u201d convolution, \u2217f is a \u201cfull\u201d convolution,1 and \u02dcW k\nnW \u2212j+1. Since all units\nin one layer are conditionally independent given the other layer, inference in the network can be\nef\ufb01ciently performed using block Gibbs sampling. Lee et al. [9] further developed a convolutional\nRBM with \u201cprobabilistic max-pooling,\u201d where the maxima over small neighborhoods of hidden\nunits are computed in a probabilistically sound way. (See [9] for more details.) In this paper, we use\nCRBMs with probabilistic max-pooling as building blocks for convolutional deep belief networks.\n\n(cid:44) W k\n\nj\n\n1Given an m-dimensional vector and an n-dimensional kernel (where m > n), valid convolution gives a\n\n(m \u2212 n + 1)-dimensional vector, and full convolution gives a (m + n \u2212 1)-dimensional vector.\n\n2\n\n\fFor training the convolutional RBMs, computing the exact gradient for the log-likelihood term is in-\ntractable. However, contrastive divergence [12] can be used to approximate the gradient effectively.\nSince a typical CRBM is highly overcomplete, a sparsity penalty term is added to the log-likelihood\nobjective [8, 9]. More speci\ufb01cally, the training objective can be written as\n\nminimizeW,b,c\n\nLlikelihood(W, b, c) + Lsparsity(W, b, c),\n\n(7)\nwhere Llikelihood is a negative log-likelihood that measures how well the CRBM approximates the\ninput data distribution, and Lsparsity is a penalty term that constrains the hidden units to having\nsparse average activations. This sparsity regularization can be viewed as limiting the \u201ccapacity\u201d\nof the network, and it often results in more easily interpretable feature representations. Once the\nparameters for all the layers are trained, we stack the CRBMs to form a convolutional deep belief\nnetwork. For inference, we use feed-forward approximation.\n2.2 Application to audio data\nFor the application of CDBNs to audio data, we \ufb01rst convert time-domain signals into spectro-\ngrams. However, the dimensionality of the spectrograms is large (e.g., 160 channels). We apply\nPCA whitening to the spectrograms and create lower dimensional representations. Thus, the data\nwe feed into the CDBN consists of nc channels of one-dimensional vectors of length nV , where nc is\nthe number of PCA components in our representation. Similarly, the \ufb01rst-layer bases are comprised\nof nc channels of one-dimensional \ufb01lters of length nW .\n3 Unsupervised feature learning\n3.1 Training on unlabeled TIMIT data\nWe trained the \ufb01rst and second-layer CDBN representations using a large, unlabeled speech dataset.\nFirst, we extracted the spectrogram from each utterance of the TIMIT training data [13]. The spec-\ntrogram had a 20 ms window size with 10 ms overlaps. The spectrogram was further processed using\nPCA whitening (with 80 components) to reduce the dimensionality. We then trained 300 \ufb01rst-layer\nbases with a \ufb01lter length (nW ) of 6 and a max-pooling ratio (local neighborhood size) of 3. We\nfurther trained 300 second-layer bases using the max-pooled \ufb01rst-layer activations as input, again\nwith a \ufb01lter length of 6 and a max-pooling ratio of 3.\n3.2 Visualization\nIn this section, we illustrate what the network \u201clearns\u201d through visualization. We visualize the \ufb01rst-\nlayer bases by multiplying the inverse of the PCA whitening on each \ufb01rst-layer basis (Figure 1).\nEach second-layer basis is visualized as a weighted linear combination of the \ufb01rst-layer bases.\n\nFigure 1: Visualization of randomly selected \ufb01rst-layer CDBN bases trained on the TIMIT data.\nEach column represents a \u201ctemporal receptive \ufb01eld\u201d of a \ufb01rst-layer basis in the spectrogram space.\nThe frequency channels are ordered from the lowest frequency (bottom) to the highest frequency\n(top). All \ufb01gures in the paper are best viewed in color.\n3.2.1 Phonemes and the CDBN features\nIn Figure 2, we show how our bases relate to phonemes by comparing visualizations of each\nphoneme with the bases that are most activated by that phoneme.\nFor each phoneme, we show \ufb01ve spectrograms of sound clips of that phoneme (top \ufb01ve columns in\neach phoneme group), and the \ufb01ve \ufb01rst-layer bases with the highest average activations on the given\nphoneme (bottom \ufb01ve columns in each phoneme group). Many of the \ufb01rst-layer bases closely match\nthe shapes of phonemes. There are prominent horizontal bands in the lower frequencies of the \ufb01rst-\nlayer bases that respond most to vowels (for example, \u201cah\u201d and \u201coy\u201d). The bases that respond most\n\n3\n\nhigh freq. low freq.\fFigure 2: Visualization of the four different phonemes and their corresponding \ufb01rst-layer CDBN\nbases. For each phoneme: (top) the spectrograms of the \ufb01ve randomly selected phones; (bottom)\n\ufb01ve \ufb01rst-layer bases with the highest average activations on the given phoneme.\n\nto fricatives (for example, \u201cs\u201d) typically take the form of widely distributed areas of energy in the\nhigh frequencies of the spectrogram. Both of these patterns re\ufb02ect the structure of the corresponding\nphoneme spectrograms.\nCloser inspection of the bases provides slight evidence that the \ufb01rst-layer bases also capture more\n\ufb01ne-grained details. For example, the \ufb01rst and third \u201coy\u201d bases re\ufb02ect the upward-slanting pattern\nin the phoneme spectrograms. The top \u201cel\u201d bases mirror the intensity patterns of the corresponding\nphoneme spectrograms: a high intensity region appears in the lowest frequencies, and another region\nof lesser intensity appears a bit higher up.\n\n3.2.2 Speaker gender information and the CDBN features\n\nIn Figure 3, we show an analysis of two-layer CDBN feature representations with respect to the gen-\nder classi\ufb01cation task (Section 4.2). Note that the network was trained on unlabeled data; therefore,\nno information about speaker gender was given during training.\n\nFigure 3: (Left) \ufb01ve spectrogram samples of \u201cae\u201d phoneme from female (top)/male (bottom) speak-\ners.\n(Middle) Visualization of the \ufb01ve \ufb01rst-layer bases that most differentially activate for fe-\nmale/male speakers. (Right) Visualization of the \ufb01ve second-layer bases that most differentially\nactivate for female/male speakers.\n\nFor comparison with the CDBN features, randomly selected spectrograms of female (top left \ufb01ve\ncolumns) and male (bottom left \ufb01ve columns) pronunciations of the \u201cae\u201d phoneme from the TIMIT\ndataset are shown. Spectrograms for the female pronunciations are qualitatively distinguishable by a\n\ufb01ner horizontal banding pattern in low frequencies, whereas male pronunciations have more blurred\n\n4\n\nExample phones (\"ah\")Example phones (\"oy\")Example phones (\"el\")Example phones (\"s\")First layer basesFirst layer basesFirst layer basesFirst layer basesExample phones (female)First layer bases (\"female\")Second layer bases (\"female\")Example phones (male)First layer bases (\"male\")Second layer bases (\"male\")\fpatterns. This gender difference in the vowel pronunciation patterns is typical across the TIMIT\ndata.\nOnly the bases that are most biased to activate on either male or female speech are shown. The bases\nthat are most active on female speech encode the horizontal band pattern that is prominent in the\nspectrograms of female pronunciations. On the other hand, the male-biased bases have more blurred\npatterns, which again visually matches the corresponding spectrograms.\n\n4 Application to speech recognition tasks\nIn this section, we demonstrate that the CDBN feature representations learned from the unlabeled\nspeech corpus can be useful for multiple speech recognition tasks, such as speaker identi\ufb01cation,\ngender classi\ufb01cation, and phone classi\ufb01cation. In most of our experiments, we followed the self-\ntaught learning framework [14]. The motivation for self-taught learning comes from situations\nwhere we are given only a small amount of labeled data and a large amount of unlabeled data;2\ntherefore, one of our main interests was to evaluate the different feature representations given a small\nnumber of labeled training examples (as often assumed in self-taught learning or semi-supervised\nlearning settings). More speci\ufb01cally, we trained the CDBN on unlabeled TIMIT data (as described\nin Section 3.1); then we used the CDBN features for classi\ufb01cation on labeled training/test data3 that\nwere randomly selected from the TIMIT corpus.4\n\n4.1 Speaker identi\ufb01cation\n\nWe evaluated the usefulness of the learned CDBN representations for the speaker identi\ufb01cation task.\nThe subset of the TIMIT corpus that we used for speaker identi\ufb01cation has 168 speakers and 10\nutterances (sentences) per speaker, resulting in a total of 1680 utterances. We performed 168-way\nclassi\ufb01cation on this set. For each number of utterances per speaker, we randomly selected training\nutterances and testing utterances and measured the classi\ufb01cation accuracy; we report the results\naveraged over 10 random trials.5 To construct training and test data for the classi\ufb01cation task,\nwe extracted a spectrogram from each utterance in the TIMIT corpus. We denote this spectrogram\nrepresentation as \u201cRAW\u201d features. We computed the \ufb01rst and second-layer CDBN features using the\nspectrogram as input. We also computed MFCC features, widely-used standard features for generic\nspeech recognition tasks. As a result, we obtained spectrogram/MFCC/CDBN representations for\neach utterance with multiple (typically, several hundred) frames. In our experiments, we used simple\nsummary statistics (for each channel) such as average, max, or standard deviation over all the frames.\nWe evaluated the features using standard supervised classi\ufb01ers, such as SVM, GDA, and KNN.\nThe choices of summary statistics and hyperparameters for the classi\ufb01ers were done using cross-\nvalidation. We report the average classi\ufb01cation accuracy (over 10 random trials) with a varying\nnumber of training examples.\nTable 1 shows the average classi\ufb01cation accuracy for each feature representation. The results\nshow that the \ufb01rst and second CDBN representations both outperform baseline features (RAW and\nMFCC). The numbers compare MFCC and CDBN features with as many of the same factors (such as\npreprocessing and classi\ufb01cation algorithms) as possible. Further, to make a fair comparison between\nCDBN features and MFCC, we used the best performing implementation6 among several standard\nimplementations for MFCC. Our results suggest that without special preprocessing or postprocess-\n\n2In self-taught learning, the labeled data and unlabeled data don\u2019t need to share the same labels or the same\n\ngenerative distributions.\n\n3There are two disjoint TIMIT data sets. We drew unlabeled data from the larger of the two for unsupervised\nfeature learning, and we drew labeled data from the other data set to create our training and test set for the\nclassi\ufb01cation tasks.\n\n4In the case of phone classi\ufb01cation, we followed the standard protocol (e.g., [15]) rather than self-taught\n\nlearning framework to evaluate our algorithm in comparison to other methods.\n\n5Details: There were some exceptions to this; for the case of eight training utterances, we followed\nReynolds (1995) [16]; more speci\ufb01cally, we used eight training utterances (2 sa sentences, 3 si sentences and\n\ufb01rst 3 sx sentences); the two testing utterances were the remaining 2 sx sentences. We used cross validation\nfor selecting hyperparameters for classi\ufb01cation, except for the case of 1 utterance per speaker, where we used a\nrandomly selected validation sentence per speaker.\n\n6We used Dan Ellis\u2019 implementation available at: http://labrosa.ee.columbia.edu/matlab/\n\nrastamat.\n\n5\n\n\fTable 1: Test classi\ufb01cation accuracy for speaker identi\ufb01cation using summary statistics\nRAW MFCC CDBN L1 CDBN L2 CDBN L1+L2\n#training utterances per speaker\n46.7% 54.4%\n43.5% 69.9%\n67.9% 76.5%\n80.6% 82.6%\n90.4% 92.0%\n\n74.5%\n76.7%\n91.3%\n93.7%\n97.9%\n\n72.8%\n76.7%\n91.8%\n93.8%\n97.0%\n\n62.8%\n66.2%\n84.3%\n89.6%\n95.2%\n\n1\n2\n3\n5\n8\n\nTable 2: Test classi\ufb01cation accuracy for speaker identi\ufb01cation using all frames\n\n#training utterances per speaker MFCC ([16]\u2019s method) CDBN MFCC ([16]) + CDBN\n\n1\n2\n3\n5\n8\n\n40.2%\n87.9%\n95.9%\n99.2%\n99.7%\n\n90.0%\n97.9%\n98.7%\n99.2%\n99.7%\n\n90.7%\n98.7%\n99.2%\n99.6%\n100.0%\n\ning (besides the summary statistics which were needed to reduce the number of features), the CDBN\nfeatures outperform MFCC features, especially in a setting with a very limited number of labeled\nexamples.\nWe further experimented to determine if the CDBN features can achieve competitive performance in\ncomparison to other more sophisticated, state-of-the-art methods. For each feature representation,\nwe used the classi\ufb01er that achieved the highest performance. More speci\ufb01cally, for the MFCC fea-\ntures we replicated Reynolds (1995)\u2019s method,7 and for the CDBN features we used a SVM based\nensemble method.8 As shown in Table 2, the CDBN features consistently outperformed MFCC fea-\ntures when the number of training examples was small. We also combined both methods by taking a\nlinear combination of the two classi\ufb01er outputs (before taking the \ufb01nal classi\ufb01cation prediction from\neach algorithm).9 The resulting combined classi\ufb01er performed the best, achieving 100% accuracy\nfor the case of 8 training utterances per speaker.\n\n4.2 Speaker gender classi\ufb01cation\n\nWe also evaluated the same CDBN features which were learned for the speaker identi\ufb01cation task on\nthe gender classi\ufb01cation task. We report the classi\ufb01cation accuracy for various quantities of training\nexamples (utterances) per gender. For each number of training examples, we randomly sampled\ntraining examples and 200 testing examples; we report the test classi\ufb01cation accuracy averaged\nover 20 trials. As shown in Table 3, both the \ufb01rst and second CDBN features outperformed the\nbaseline features, especially when the number of training examples were small. The second-layer\nCDBN features consistently performed better than the \ufb01rst-layer CDBN features. This suggests that\nthe second-layer representation learned more invariant features that are relevant for speaker gender\nclassi\ufb01cation, justifying the use of \u201cdeep\u201d architectures.\n\n4.3 Phone classi\ufb01cation\n\nFinally, we evaluated our learned representation on phone classi\ufb01cation tasks. For this experiment,\nwe treated each phone segment as an individual example and computed the spectrogram (RAW) and\nMFCC features for each phone segment. Similarly, we computed the \ufb01rst-layer CDBN representa-\ntions. Following the standard protocol [15], we report the 39 way phone classi\ufb01cation accuracy on\nthe test data (TIMIT core test set) for various numbers of training sentences. For each number of\ntraining examples, we report the average classi\ufb01cation accuracy over 5 random trials. The summary\n\n7Details: In [16], MFCC features (with multiple frames) were computed for each utterance; then a Gaussian\nmixture model was trained for each speaker (treating each individual MFCC frame as a input example to the\nGMM. For the a given test utterance, the prediction was made by determining the GMM model that had the\nhighest test log-likelihood.\n\n8In detail, we treated each single-frame CDBN features as an individual example. Then, we trained a multi-\nclass linear SVM for these individual frames. For testing, we computed SVM prediction score for each speaker,\nand then aggregated predictions from all the frames. Overall, the highest scoring speaker was selected for the\nprediction.\n\n9The constant for the linear combination was \ufb01xed across all the numbers of training utterances, and it was\n\nselected using cross validation.\n\n6\n\n\fTable 3: Test accuracy for gender classi\ufb01cation problem\n\n#training utterances per gender\n\n1\n2\n3\n5\n7\n10\n\nRAW MFCC CDBN L1 CDBN L2 CDBN L1+L2\n68.4% 58.5%\n76.7% 78.7%\n79.5% 84.1%\n84.4% 86.9%\n89.2% 89.0%\n91.3% 89.8%\n\n85.8%\n92.5%\n94.2%\n95.8%\n96.6%\n96.7%\n\n83.6%\n92.3%\n94.2%\n95.6%\n96.5%\n96.6%\n\n78.5%\n86.0%\n88.9%\n93.1%\n94.2%\n94.7%\n\nTable 4: Test accuracy for phone classi\ufb01cation problem\n\n#training utterances\n\n100\n200\n500\n1000\n2000\n3696\n\nRAW MFCC MFCC ([15]\u2019s method) CDBN L1 MFCC+CDBN L1 ([15])\n36.9% 58.3%\n37.8% 61.5%\n38.7% 64.9%\n39.0% 67.2%\n39.2% 69.2%\n39.4% 70.8%\n\n67.2%\n71.0%\n75.1%\n77.1%\n79.2%\n80.3%\n\n66.6%\n70.3%\n74.1%\n76.3%\n78.4%\n79.6%\n\n53.7%\n56.7%\n59.7%\n61.6%\n63.1%\n64.4%\n\nresults are shown in Table 4. In this experiment, the \ufb01rst-layer CDBN features performed better\nthan spectrogram features, but they did not outperform the MFCC features. However, by combining\nMFCC features and CDBN features, we could achieve about 0.7% accuracy improvement consis-\ntently over all the numbers of training utterances. In the realm of phone classi\ufb01cation, in which\nsigni\ufb01cant research effort is often needed to achieve even improvements well under a percent, this\nis a signi\ufb01cant improvement. [17, 18, 19, 20]\nThis suggests that the \ufb01rst-layer CDBN features learned somewhat informative features for phone\nclassi\ufb01cation tasks in an unsupervised way. In contrast to the gender classi\ufb01cation task, the second-\nlayer CDBN features did not offer much improvement over the \ufb01rst-layer CDBN features. This\nresult is not unexpected considering the fact that the time-scale of most phonemes roughly corre-\nsponds to the time-scale of the \ufb01rst-layer CDBN features.\n\n5 Application to music classi\ufb01cation tasks\nIn this section, we assess the applicability of CDBN features to various music classi\ufb01cation tasks.\n\nTable 5: Test accuracy for 5-way music genre classi\ufb01cation\n\nTrain examples\n\n1\n2\n3\n5\n\nRAW MFCC CDBN L1 CDBN L2 CDBN L1+L2\n51.6% 54.0%\n57.0% 62.1%\n59.7% 65.3%\n65.8% 68.3%\n\n66.1%\n69.7%\n70.0%\n73.1%\n\n62.5%\n67.9%\n66.7%\n69.2%\n\n64.3%\n69.5%\n69.5%\n72.7%\n\n5.1 Music genre classi\ufb01cation\n\nFor the task of music genre classi\ufb01cation, we trained the \ufb01rst and second-layer CDBN representa-\ntions on an unlabeled collection of music data.10 First, we computed the spectrogram (20 ms window\nsize with 10 ms overlaps) representation for individual songs. The spectrogram was PCA-whitened\nand then fed into the CDBN as input data. We trained 300 \ufb01rst-layer bases with a \ufb01lter length of 10\nand a max-pooling ratio of 3. In addition, we trained 300 second-layer bases with a \ufb01lter length of\n10 and a max-pooling ratio of 3.\nWe evaluated the learned CDBN representation for 5-way genre classi\ufb01cation tasks. The training\nand test songs for the classi\ufb01cation tasks were randomly sampled from 5 genres (classical, electric,\njazz, pop, and rock) and did not overlap with the unlabeled data. We randomly sampled 3-second\nsegments from each song and treated each segment as an individual training or testing example. We\nreport the classi\ufb01cation accuracy for various numbers of training examples. For each number of\ntraining examples, we averaged over 20 random trials. The results are shown in Table 5. In this task,\nthe \ufb01rst-layer CDBN features performed the best overall.\n\n10Available from http://ismir2004.ismir.net/ISMIR_Contest.html.\n\n7\n\n\f5.2 Music artist classi\ufb01cation\nFurthermore, we evaluated whether the CDBN features are useful in identifying individual artists.11\nFollowing the same procedure as in Section 5.1, we trained the \ufb01rst and second-layer CDBN rep-\nresentations from an unlabeled collection of classical music data. Some representative bases are\nshown in Figure 4. Then we evaluated the learned CDBN representation for 4-way artist identi\ufb01-\ncation tasks. The disjoint sets of training and test songs for the classi\ufb01cation tasks were randomly\nsampled from the songs of four artists. The unlabeled data and the labeled data did not include the\nsame artists. We randomly sampled 3-second segments from each song and treated each segment as\nan individual example. We report the classi\ufb01cation accuracy for various quantities of training ex-\namples. For each number of training examples, we averaged over 20 random trials. The results are\nshown in Table 6. The results show that both the \ufb01rst and second-layer CDBN features performed\nbetter than the baseline features, and that either using the second-layer features only or combining\nthe \ufb01rst and the second-layer features yielded the best results. This suggests that the second-layer\nCDBN representation might have captured somewhat useful, higher-level features than the \ufb01rst-layer\nCDBN representation.\n\nFigure 4: Visualization of randomly selected \ufb01rst-layer CDBN bases trained on classical music data.\n\nTable 6: Test accuracy for 4-way artist identi\ufb01cation\n\nTrain examples\n\n1\n2\n3\n5\n\nRAW MFCC CDBN L1 CDBN L2 CDBN L1+L2\n56.0% 63.7%\n69.4% 66.1%\n73.9% 67.9%\n79.4% 71.6%\n\n69.2%\n76.3%\n78.7%\n81.4%\n\n67.6%\n76.1%\n78.0%\n80.9%\n\n67.7%\n74.2%\n75.8%\n81.9%\n\n6 Discussion and conclusion\nModern speech datasets are much larger than the TIMIT dataset. While the challenge of larger\ndatasets often lies in considering harder tasks, our objective in using the TIMIT data was to restrict\nthe amount of labeled data our algorithm had to learn from. It remains an interesting problem to\napply deep learning to larger datasets and more challenging tasks.\nIn this paper, we applied convolutional deep belief networks to audio data and evaluated on various\naudio classi\ufb01cation tasks. By leveraging a large amount of unlabeled data, our learned features\noften equaled or surpassed MFCC features, which are hand-tailored to audio data. Furthermore,\neven when our features did not outperform MFCC, we could achieve higher classi\ufb01cation accuracy\nby combining both. Also, our results show that a single CDBN feature representation can achieve\nhigh performance on multiple audio recognition tasks. We hope that our approach will inspire more\nresearch on automatically learning deep feature hierarchies for audio data.\nAcknowledgment\nWe thank Yoshua Bengio, Dan Jurafsky, Yun-Hsuan Sung, Pedro Moreno, Roger Grosse for helpful\ndiscussions. We also thank anonymous reviewers for their constructive comments. This work was\nsupported in part by the National Science Foundation under grant EFRI-0835878, and in part by the\nOf\ufb01ce of Naval Research under MURI N000140710747.\n\n11In our experiments, we found that artist identi\ufb01cation task was more dif\ufb01cult than the speaker identi\ufb01cation\n\ntask because the local sound patterns can be highly variable even for the same artist.\n\n8\n\nhigh freq. low freq.\fReferences\n[1] E. C. Smith and M. S. Lewicki. Ef\ufb01cient auditory coding. Nature, 439:978\u2013982, 2006.\n[2] B. A. Olshausen and D. J. Field. Emergence of simple-cell receptive \ufb01eld properties by learning\n\na sparse code for natural images. Nature, 381:607\u2013609, 1996.\n\n[3] R. Grosse, R. Raina, H. Kwong, and A.Y. Ng. Shift-invariant sparse coding for audio classi\ufb01-\n\ncation. In UAI, 2007.\n\n[4] G. E. Hinton, S. Osindero, and Y.-W. Teh. A fast learning algorithm for deep belief nets.\n\nNeural Computation, 18(7):1527\u20131554, 2006.\n\n[5] M. Ranzato, C. Poultney, S. Chopra, and Y. LeCun. Ef\ufb01cient learning of sparse representations\n\nwith an energy-based model. In NIPS, 2006.\n\n[6] Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. Greedy layer-wise training of deep\n\nnetworks. In NIPS, 2006.\n\n[7] H. Larochelle, D. Erhan, A. Courville, J. Bergstra, and Y. Bengio. An empirical evaluation of\n\ndeep architectures on problems with many factors of variation. In ICML, 2007.\n\n[8] H. Lee, C. Ekanadham, and A. Y. Ng. Sparse deep belief network model for visual area V2. In\n\nNIPS, 2008.\n\n[9] H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng. Convolutional deep belief networks for\n\nscalable unsupervised learning of hierarchical representations. In ICML, 2009.\n\n[10] G. Desjardins and Y. Bengio. Empirical evaluation of convolutional RBMs for vision. Techni-\n\ncal report, 2008.\n\n[11] M. Norouzi, M. Ranjbar, and G. Mori. Stacks of convolutional restricted boltzmann machines\n\nfor shift-invariant feature learning. In CVPR, 2009.\n\n[12] G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural\n\nComputation, 14:1771\u20131800, 2002.\n\n[13] W. Fisher, G. Doddington, and K. Goudie-Marshall. The darpa speech recognition research\n\ndatabase: Speci\ufb01cations and status. In DARPA Speech Recognition Workshop, 1986.\n\n[14] R. Raina, A. Battle, H. Lee, B. Packer, and A. Y. Ng. Self-taught learning: Transfer learning\n\nfrom unlabeled data. In ICML, 2007.\n\n[15] P. Clarkson and P. J. Moreno. On the use of support vector machines for phonetic classi\ufb01cation.\n\nIn ICASSP99, pages 585\u2013588, 1999.\n\n[16] D. A. Reynolds. Speaker identi\ufb01cation and veri\ufb01cation using gaussian mixture speaker models.\n\nSpeech Commun., 17(1-2):91\u2013108, 1995.\n\n[17] F. Sha and L. K. Saul. Large margin gaussian mixture modeling for phonetic classication and\n\nrecognition. In ICASSP\u201906, 2006.\n\n[18] Y.-H. Sung, C. Boulis, C. Manning, and D. Jurafsky. Regularization, adaptation, and non-\nIn\n\nindependent features improve hidden conditional random \ufb01elds for phone classi\ufb01cation.\nIEEE ASRU, 2007.\n\n[19] S. Petrov, A. Pauls, and D. Klein. Learning structured models for phone recognition.\n\nEMNLP-CoNLL, 2007.\n\nIn\n\n[20] D. Yu, L. Deng, and A. Acero. Hidden conditional random \ufb01eld with distribution constraints\n\nfor phone classi\ufb01cation. In Interspeech, 2009.\n\n9\n\n\f", "award": [], "sourceid": 1171, "authors": [{"given_name": "Honglak", "family_name": "Lee", "institution": null}, {"given_name": "Peter", "family_name": "Pham", "institution": null}, {"given_name": "Yan", "family_name": "Largman", "institution": null}, {"given_name": "Andrew", "family_name": "Ng", "institution": null}]}