Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes

Part of Advances in Neural Information Processing Systems 20 (NIPS 2007)

Authors

Geoffrey E. Hinton, Russ R. Salakhutdinov

Abstract

We show how to use unlabeled data and a deep belief net (DBN) to learn a good covariance kernel for a Gaussian process. We first learn a deep generative model of the unlabeled data using the fast, greedy algorithm introduced by Hinton et.al. If the data is high-dimensional and highly-structured, a Gaussian kernel applied to the top layer of features in the DBN works much better than a similar kernel applied to the raw input. Performance at both regression and classification can then be further improved by using backpropagation through the DBN to discriminatively fine-tune the covariance kernel.