{"title": "Inferring Motor Programs from Images of Handwritten Digits", "book": "Advances in Neural Information Processing Systems", "page_first": 515, "page_last": 522, "abstract": null, "full_text": "Inferring Motor Programs from Images of\n\nHandwritten Digits\n\nDepartment of Computer Science, University of Toronto\n\n10 King\u2019s College Road, Toronto, M5S 3G5 Canada\n\nGeoffrey Hinton and Vinod Nair\n\n{hinton,vnair}@cs.toronto.edu\n\nAbstract\n\nWe describe a generative model for handwritten digits that uses two pairs\nof opposing springs whose stiffnesses are controlled by a motor program.\nWe show how neural networks can be trained to infer the motor programs\nrequired to accurately reconstruct the MNIST digits. The inferred motor\nprograms can be used directly for digit classi\ufb01cation, but they can also be\nused in other ways. By adding noise to the motor program inferred from\nan MNIST image we can generate a large set of very different images of\nthe same class, thus enlarging the training set available to other methods.\nWe can also use the motor programs as additional, highly informative\noutputs which reduce over\ufb01tting when training a feed-forward classi\ufb01er.\n\n1 Overview\nThe idea that patterns can be recognized by \ufb01guring out how they were generated has been\naround for at least half a century [1, 2] and one of the \ufb01rst proposed applications was the\nrecognition of handwriting using a generative model that involved pairs of opposing springs\n[3, 4]. The \u201canalysis-by-synthesis\u201d approach is attractive because the true generative model\nshould provide the most natural way to characterize a class of patterns. The handwritten 2\u2019s\nin \ufb01gure 1, for example, are very variable when viewed as pixels but they have very similar\nmotor programs. Despite its obvious merits, analysis-by-synthesis has had few successes,\npartly because it is computationally expensive to invert non-linear generative models and\npartly because the underlying parameters of the generative model are unknown for most\nlarge data sets. For example, the only source of information about how the MNIST digits\nwere drawn is the images themselves.\n\nWe describe a simple generative model in which a pen is controlled by two pairs of op-\nposing springs whose stiffnesses are speci\ufb01ed by a motor program.\nIf the sequence of\nstiffnesses is speci\ufb01ed correctly, the model can produce images which look very like the\nMNIST digits. Using a separate network for each digit class, we show that backpropaga-\ntion can be used to learn a \u201crecognition\u201d network that maps images to the motor programs\nrequired to produce them. An interesting aspect of this learning is that the network creates\nits own training data, so it does not require the training images to be labelled with motor\nprograms. Each recognition network starts with a single example of a motor program and\ngrows an \u201cisland of competence\u201d around this example, progressively extending the region\nover which it can map small changes in the image to the corresponding small changes in\nthe motor program (see \ufb01gure 2).\n\n\fFigure 1: An MNIST image of a 2 and the additional images that can be generated by infer-\nring the motor program and then adding random noise to it. The pixels are very different,\nbut they are all clearly twos.\n\nFairly good digit recognition can be achieved by using the 10 recognition networks to \ufb01nd\n10 motor programs for a test image and then scoring each motor program by its squared\nerror in reconstructing the image. The 10 scores are then fed into a softmax classi\ufb01er.\nRecognition can be improved by using PCA to model the distribution of motor trajectories\nfor each class and using the distance of a motor trajectory from the relevant PCA hyperplane\nas an additional score.\n\nEach recognition network is solving a dif\ufb01cult global search problem in which the correct\nmotor program must be found by a single, \u201copen-loop\u201d pass through the network. More\naccurate recognition can be achieved by using this open-loop global search to initialize an\niterative, closed-loop local search which uses the error in the reconstructed image to re-\nvise the motor program. This requires reconstruction errors in pixel space to be mapped to\ncorrections in the space of spring stiffnesses. We cannot backpropagate errors through the\ngenerative model because it is just a hand-coded computer program. So we learn \u201cgenera-\ntive\u201d networks, one per digit class, that emulate the generator. After learning, backpropa-\ngation through these generative networks is used to convert pixel reconstruction errors into\nstiffness corrections.\n\nOur \ufb01nal system gives 1.82% error on the MNIST test set which is similar to the 1.7%\nachieved by a very different generative approach [5] but worse than the 1.53% produced\nby the best backpropagation networks or the 1.4% produced by support vector machines\n[6]. It is much worse than the 0.4% produced by convolutional neural networks that use\ncleverly enhanced training sets [7]. Recognition of test images is quite slow because it uses\nten different recognition networks followed by iterative local search. There is, however, a\nmuch more ef\ufb01cient way to make use of our ability to extract motor programs. They can\nbe treated as additional output labels when using backpropagation to train a single, multi-\nlayer, discriminative neural network. These additional labels act as a very informative\nregularizer that reduces the error rate from 1.53% to 1.27% in a network with two hidden\nlayers of 500 units each. This is a new method of improving performance that can be used\nin conjunction with other tricks such as preprocessing the images, enhancing the training\nset or using convolutional neural nets [8, 7].\n\n2 A simple generative model for drawing digits\n\nThe generative model uses two pairs of opposing springs at right angles. One end of each\nspring is attached to a frictionless horizontal or vertical rail that is 39 pixels from the center\nof the image. The other end is attached to a \u201cpen\u201d that has signi\ufb01cant mass. The springs\nthemselves are weightless and have zero rest length. The pen starts at the equilibrium\nposition de\ufb01ned by the initial stiffnesses of the four springs. It then follows a trajectory\nthat is determined by the stiffness of each spring at each of the 16 subsequent time steps\nin the motor program. The mass is large compared with the rate at which the stiffnesses\nchange, so the system is typically far from equilibrium as it follows the smooth trajectory.\nOn each time step, the momentum is multiplied by 0.9 to simulate viscosity. A coarse-grain\ntrajectory is computed by using one step of forward integration for each time step in the\nmotor program, so it contains 17 points. The code is at www.cs.toronto.edu/\u223c hinton/code.\n\n\fFigure 2: The training data for each class-speci\ufb01c recognition network is produced by\nadding noise to motor programs that are inferred from MNIST images using the current\nparameters of the recognition network. To initiate this process, the biases of the output\nunits are set by hand so that they represent a prototypical motor program for the class.\n\nGiven a coarse-grain trajectory, we need a way of assigning an intensity to each pixel. We\ntried various methods until we hand-evolved one that was able to reproduce the MNIST im-\nages fairly accurately, but we suspect that many other methods would be just as good. For\neach point on the coarse trajectory, we share two units of ink between the the four closest\npixels using bilinear interpolation. We also use linear interpolation to add three \ufb01ne-grain\ntrajectory points between every pair of coarse-grain points. These \ufb01ne-grain points also\ncontribute ink to the pixels using bilinear interpolation, but the amount of ink they con-\ntribute is zero if they are less than one pixel apart and rises linearly to the same amount as\nthe coarse-grain points if they are more than two pixels apart. This generates a thin skeleton\nwith a fairly uniform ink density. To \ufb02esh-out the skeleton, we use two \u201cink parameters\u201d,\na, b, to specify a 3 \u00d7 3 kernel of the form b(1 + a)[ a\n12 ] which\nis convolved with the image four times. Finally, the pixel intensities are clipped to lie in\nthe interval [0,1]. The matlab code is at www.cs.toronto.edu/\u223c hinton/code. The values of\n2a and b/1.5 are additional, logistic outputs of the recognition networks1.\n\n6 , 1 \u2212 a, a\n6 ;\n\n12 , a\n\n6 , a\n\n12 , a\n\n6 , a\n\n12 ; a\n\na\n\n3 Training the recognition networks\nThe obvious way to learn a recognition network is to use a training set in which the inputs\nare images and the target outputs are the motor programs that were used to generate those\nimages. If we knew the distribution over motor programs for a given digit class, we could\neasily produce such a set by running the generator. Unfortunately, the distribution over\nmotor programs is exactly what we want to learn from the data, so we need a way to train\n\n1We can add all sorts of parameters to the hand-coded generative model and then get the recogni-\ntion networks to learn to extract the appropriate values for each image. The global mass and viscosity\nas well as the spacing of the rails that hold the springs can be learned. We can even implement af\ufb01ne-\nlike transformations by attaching the four springs to endpoints whose eight coordinates are given by\nthe recognition networks. These extra parameters make the learning slower and, for the normalized\ndigits, they do not improve discrimination, probably because they help the wrong digit models as\nmuch as the right one.\n\n\fthe recognition network without knowing this distribution in advance. Generating scribbles\nfrom random motor programs will not work because the capacity of the network will be\nwasted on irrelevant images that are far from the real data.\n\nFigure 2 shows how a single, prototype motor program can be used to initialize a learning\nprocess that creates its own training data. The prototype consists of a sequence of 4 \u00d7 17\nspring stiffnesses that are used to set the biases on 68 of the 70 logistic output units of\nthe recognition net.\nIf the weights coming from the 400 hidden units are initially very\nsmall, the recognition net will then output a motor program that is a close approximation\nto the prototype, whatever the input image. Some random noise is then added to this motor\nprogram and it is used to generate a training image. So initially, all of the generated training\nimages are very similar to the one produced by the prototype. The recognition net will\ntherefore devote its capacity to modeling the way in which small changes in these images\nmap to small changes in the motor program. Images in the MNIST training set that are\nclose to the prototype will then be given their correct motor programs. This will tend to\nstretch the distribution of motor programs produced by the network along the directions that\ncorrespond to the manifold on which the digits lie. As time goes by, the generated training\nset will expand along the manifold for that digit class until all of the MNIST training images\nof that class are well modelled by the recognition network.\n\nIt takes about 10 hours in matlab on a 3 GHz Xeon to train each recognition network.\nWe use minibatches of size 100, momentum of 0.9, and adaptive learning rates on each\nconnection that increase additively when the sign of the gradient agrees with the sign of\nthe previous weight change and decrease multiplicatively when the signs disagree [9]. The\nnet is generating its own training data, so the objective function is always changing which\nmakes it inadvisable to use optimization methods that go as far as they can in a carefully\nchosen direction. Figures 3 and 4 show some examples of how well the recognition nets\nperform after training. Nearly all models achieve an average squared pixel error of less\nthan 15 per image on their validation set (pixel intensities are between 0 and 1 with a\npreponderance of extreme values). The inferred motor programs are clearly good enough\nto capture the diverse handwriting styles in the data. They are not good enough, however, to\ngive classi\ufb01cation performance comparable to the state-of-the-art on the MNIST database.\nSo we added a series of enhancements to the basic system to improve the classi\ufb01cation\naccuracy.\n\n4 Enhancements to the basic system\nExtra strokes in ones and sevens. One limitation of the basic system is that it draws digits\nusing only a single stroke (i.e. the trajectory is a single, unbroken curve). But when people\ndraw digits, they often add extra strokes to them. Two of the most common examples are\nthe dash at the bottom of ones, and the dash through the middle of sevens (see examples in\n\ufb01gure 5). About 2.2% of ones and 13% of sevens in the MNIST training set are dashed and\nnot modelling the dashes reduces classi\ufb01cation accuracy signi\ufb01cantly. We model dashed\nones and sevens by augmenting their basic motor programs with another motor program to\ndraw the dash. For example, a dashed seven is generated by \ufb01rst drawing an ordinary seven\nusing the motor program computed by the seven model, and then drawing the dash with a\nmotor program computed by a separate neural network that models only dashes.\n\nDashes in ones and sevens are modeled with two different networks. Their training pro-\nceeds the same way as with the other models, except now there are only 50 hidden units and\nthe training set contains only the dashed cases of the digit. (Separating these cases from\nthe rest of the MNIST training set is easy because they can be quickly spotted by looking\nat the difference between the images and their reconstructions by the dashless digit model.)\nThe net takes the entire image of a digit as input, and computes the motor program for\njust the dash. When reconstructing an unlabelled image as say, a seven, we compute both\n\n\fFigure 3: Examples of validation set images reconstructed by their corresponding model.\nIn each case the original image is on the left and the reconstruction is on the right. Super-\nimposed on the original image is the pen trajectory.\n\nthe dashed and dashless versions of seven and pick the one with the lower squared pixel\nerror to be that image\u2019s reconstruction as a seven. Figure 5 shows examples of images\nreconstructed using the extra stroke.\nLocal search. When reconstructing an image in its own class, a digit model often produces\na sensible, overall approximation of the image. However, some of the \ufb01ner details of the\nreconstruction may be slightly wrong and need to be \ufb01xed up by an iterative local search\nthat adjusts the motor program to reduce the reconstruction error. We \ufb01rst approximate the\ngraphics model with a neural network that contains a single hidden layer of 500 logistic\nunits. We train one such generative network for each of the ten digits and for the dashed\nversion of ones and sevens (for a total of 12 nets). The motor programs used for training\nare obtained by adding noise to the motor programs inferred from the training data by\nthe relevant, fully trained recognition network. The images produced from these motor\nprograms by the graphics model are used as the targets for the supervised learning of each\ngenerative network. Given these targets, the weight updates are computed in the same way\nas for the recognition networks.\n\nFigure 4: To model 4\u2019s we use a single smooth trajectory, but turn off the ink for timesteps\n9 and 10. For images in which the pen does not need to leave the paper, the recognition net\n\ufb01nds a trajectory in which points 8 and 11 are close together so that points 9 and 10 are not\nneeded. For 5\u2019s we leave the top until last and turn off the ink for timesteps 13 and 14.\n\n\fFigure 5: Examples of dashed ones and sevens reconstructed using a second stroke. The\npen trajectory for the dash is shown in blue, superimposed on the original image.\n\nInitial squared pixel error = 33.8\n\n10 iterations, error = 15.2\n\n20 iterations, error = 10.5\n\n30 iterations, error = 9.3\n\nFigure 6: An example of how local search improves the detailed registration of the trajec-\ntory found by the correct model. After 30 iterations, the squared pixel error is less than a\nthird of its initial value.\n\nOnce the generative network is trained, we can use it to iteratively improve the initial motor\nprogram computed by the recognition network for an image. The main steps in one iteration\nare: 1) compute the error between the image and the reconstruction generated from the\ncurrent motor program by the graphics model; 2) backpropagate the reconstruction error\nthrough the generative network to calculate its gradient with respect to the motor program;\n3) compute a new motor program by taking a step along the direction of steepest descent\nplus 0.5 times the previous step. Figure 6 shows an example of how local search improves\nthe reconstruction by the correct model. Local search is usually less effective at improving\nthe \ufb01ts of the wrong models, so it eliminates about 20% of the classi\ufb01cation errors on the\nvalidation set.\nPCA model of the image residuals. The sum of squared pixel errors is not the best way\nof comparing an image with its reconstruction, because it treats the residual pixel errors\nas independent and zero-mean Gaussian distributed, which they are not. By modelling the\nstructure in the residual vectors, we can get a better estimate of the conditional probability\nof the image given the motor program. For each digit class, we construct a PCA model of\nthe image residual vectors for the training images. Then, given a test image, we project\nthe image residual vector produced by each inferred motor program onto the relevant PCA\nhyperplane and compute the squared distance between the residual and its projection. This\ngives ten scores for the image that measure the quality of its reconstructions by the digit\nmodels. We don\u2019t discard the old sum of squared pixel errors as they are still useful for\nclassifying most images correctly.\nInstead, all twenty scores are used as inputs to the\nclassi\ufb01er, which decides how to combine both types of scores to achieve high classi\ufb01cation\naccuracy.\nPCA model of trajectories. Classifying an image by comparing its reconstruction errors\nfor the different digit models tacitly relies on the assumption that the incorrect models will\nreconstruct the image poorly. Since the models have only been trained on images in their\n\n\fSquared error = 24.9, Shape prior score = 31.5\n\nSquared error = 15.0, Shape prior score = 104.2\n\nFigure 7: Reconstruction of a two image by the two model (left box) and by the three model\n(right box), with the pen trajectory superimposed on the original image. The three model\nsharply bends the bottom of its trajectory to better explain the ink, but the trajectory prior\nfor three penalizes it with a high score. The two model has a higher squared error, but a\nmuch lower prior score, which allows the classi\ufb01er to correctly label the image.\n\nown class, they often do reconstruct images from other classes poorly, but occasionally\nthey \ufb01t an image from another class well. For example, \ufb01gure 7 shows how the three model\nreconstructs a two image better than the two model by generating a highly contorted three.\nThis problem becomes even more pronounced with local search which sometimes contorts\nthe wrong model to \ufb01t the image really well. The solution is to learn a PCA model of the\ntrajectories that a digit model infers from images in its own class. Given a test image, the\ntrajectory computed by each digit model is scored by its squared distance from the relevant\nPCA hyperplane. These 10 \u201cprior\u201d scores are then given to the classi\ufb01er along with the\n20 \u201clikelihood\u201d scores described above. The prior scores eliminate many classi\ufb01cation\nmistakes such as the one in \ufb01gure 7.\n\n5 Classi\ufb01cation results\nTo classify a test image, we apply multinomial logistic regression to the 30 scores \u2013 i.e.\nwe use a neural network with no hidden units, 10 softmax output units and a cross-entropy\nerror. The net is trained by gradient descent using the scores for the validation set images.\nTo illustrate the gain in classi\ufb01cation accuracy achieved by the enhancements explained\nabove, table 1 gives the percent error on the validation set as each enhancement is added to\nthe system. Together, the enhancements almost halve the number of mistakes.\n\nEnhancements Validation set\n\n% error\n\nTest set\n% error\n\nNone\n\n1\n1, 2\n1, 2, 3\n1, 2, 3, 4\n\n4.43\n3.84\n3.01\n2.67\n2.28\n\n1.82\n\nTable 1: The gain in classi\ufb01cation accuracy on the validation set as the following enhance-\nments are added: 1) extra stroke for dashed ones and sevens, 2) local search, 3) PCA model\nof image residual, and 4) PCA trajectory prior. To avoid using the test set for model selec-\ntion, the performance on the of\ufb01cial test set was only measured for the \ufb01nal system.\n\n6 Discussion\nAfter training a single neural network to output both the class label and the motor program\nfor all classes (as described in section 1) we tried ignoring the label output and classifying\n\n\fthe test images by using the cost, under 10 different PCA models, of the trajectory de\ufb01ned\nby the inferred motor program. Each PCA model was \ufb01tted to the trajectories extracted\nfrom the training images for a given class. This gave 1.80% errors which is as good as the\n1.82% we got using the 10 separate recognition networks and local search. This is quite\nsurprising because the motor programs produced by the single network were simpli\ufb01ed\nto make them all have the same dimensionality and they produced signi\ufb01cantly poorer\nreconstructions. By only using the 10 digit-speci\ufb01c recognition nets to create the motor\nprograms for the training data, we get much faster recognition of test data because at test\ntime we can use a single recognition network for all classes.\nIt also means we do not\nneed to trade-off prior scores against image residual scores because there is only one image\nresidual.\n\nThe ability to extract motor programs could also be used to enhance the training set. [7]\nshows that error rates can be halved by using smooth vector distortion \ufb01elds to create extra\ntraining data. They argue that these \ufb01elds simulate \u201cuncontrolled oscillations of the hand\nmuscles dampened by inertia\u201d. Motor noise may be better modelled by adding noise to\nan actual motor program as shown in \ufb01gure 1. Notice that this produces a wide variety of\nnon-blurry images and it can also change the topology.\n\nThe techniques we have used for extracting motor programs from digit images may be ap-\nplicable to speech. There are excellent generative models that can produce almost perfect\nspeech if they are given the right formant parameters [10]. Using one of these generative\nmodels we may be able to train a large number of specialized recognition networks to ex-\ntract formant parameters from speech without requiring labeled training data. Once this has\nbeen done, labeled data would be available for training a single feed-forward network that\ncould recover accurate formant parameters which could be used for real-time recognition.\nAcknowledgements We thank Steve Isard, David MacKay and Allan Jepson for helpful discus-\nsions. This research was funded by NSERC, CFI and OIT. GEH is a fellow of the Canadian Institute\nfor Advanced Research and holds a Canada Research Chair in machine learning.\n\nReferences\n[1] D. M. MacKay. Mindlike behaviour in artefacts. British Journal for Philosophy of Science,\n\n2:105\u2013121, 1951.\n\n[2] M. Halle and K. Stevens. Speech recognition: A model and a program for research.\n\nTransactions on Information Theory, IT-8 (2):155\u2013159, 1962.\n\nIRE\n\n[3] Murray Eden. Handwriting and pattern recognition. IRE Transactions on Information Theory,\n\nIT-8 (2):160\u2013166, 1962.\n\n[4] J.M. Hollerbach. An oscillation theory of handwriting. Biological Cybernetics, 39:139\u2013156,\n\n1981.\n\n[5] G. Mayraz and G. E. Hinton. Recognizing hand-written digits using hierarchical products of\nexperts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24:189\u2013197, 2001.\n[6] D. Decoste and B. Schoelkopf. Training invariant support vector machines. Machine Learning,\n\n46:161\u2013190, 2002.\n\n[7] Patrice Y. Simard, Dave Steinkraus, and John Platt. Best practice for convolutional neural net-\nworks applied to visual document analysis. In International Conference on Document Analysis\nand Recogntion (ICDAR), IEEE Computer Society, Los Alamitos, pages 958\u2013962, 2003.\n\n[8] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document\n\nrecognition. Proceedings of the IEEE, 86(11):2278\u20132324, November 1998.\n\n[9] A. Jacobs R. Increased Rates of Convergence Through Learning Rate Adaptation. Technical\n\nReport: UM-CS-1987-117. University of Massachusetts, Amherst, MA, 1987.\n\n[10] W. Holmes, J. Holmes, and M. Judd. Extension of the bandwith of the jsru parallel-formant\nsynthesizer for high quality synthesis of male and female speech. In Proceedings of ICASSP 90\n(1), pages 313\u2013316, 1990.\n\n\f", "award": [], "sourceid": 2937, "authors": [{"given_name": "Vinod", "family_name": "Nair", "institution": null}, {"given_name": "Geoffrey", "family_name": "Hinton", "institution": null}]}