{"title": "Temporal Difference Learning of Position Evaluation in the Game of Go", "book": "Advances in Neural Information Processing Systems", "page_first": 817, "page_last": 824, "abstract": null, "full_text": "Temporal Difference Learning of \n\nPosition Evaluation in the Game of Go \n\nNicol N. Schraudolph \nschraudo~salk.edu \n\nPeter Dayan \n\ndayan~salk.edu \n\nTerrence J. Sejnowski \nterry~salk.edu \n\nComputational Neurobiology Laboratory \nThe Salk Institute for Biological Studies \n\nSan Diego, CA 92186-5800 \n\nAbstract \n\nThe game of Go has a high branching factor that defeats the tree \nsearch approach used in computer chess, and long-range spa(cid:173)\ntiotemporal interactions that make position evaluation extremely \ndifficult. Development of conventional Go programs is hampered \nby their knowledge-intensive nature. We demonstrate a viable \nalternative by training networks to evaluate Go positions via tem(cid:173)\nporal difference (TD) learning. \nOur approach is based on network architectures that reflect the \nspatial organization of both input and reinforcement signals on \nthe Go board, and training protocols that provide exposure to \ncompetent (though unlabelled) play. These techniques yield far \nbetter performance than undifferentiated networks trained by self(cid:173)\nplay alone. A network with less than 500 weights learned within \n3,000 games of 9x9 Go a position evaluation function that enables \na primitive one-ply search to defeat a commercial Go program at \na low playing level. \n\n1 INTRODUCTION \n\nGo was developed three to four millenia ago in China; it is the oldest and one of the \nmost popular board games in the world. Like chess, it is a deterministic, perfect \ninformation, zero-sum game of strategy between two players. They alternate in \n\n817 \n\n\f818 \n\nSchraudolph, Dayan, and Sejnowski \n\nplacing black and white stones on the intersections of a 19x19 grid (smaller for \nbeginners) with the objective of surrounding more board area (territory) with their \nstones than the opponent. Adjacent stones of the same color form groups; an empty \nintersection adjacent to a group is called a liberty of that group. A group is captured \nand removed from the board when its last liberty is occupied by the opponent. To \nprevent loops, it is illegal to make a move which recreates a prior board position. A \nplayer may pass at any time; the game ends when both players pass in succession. \nUnlike most other games of strategy, Go has remained an elusive skill for com puters \nto acquire -\nindeed it has been recognized as a /I grand challenge\" of Artificial \nIntelligence (Rivest, 1993). The game tree search approach used extensively in \ncomputer chess is infeasible: the game tree of Go has an average branching factor \nof around 200, but even beginners may routinely look ahead up to 60 plies in \nsome situations. Humans appear to rely mostly on static evaluation of board \npositions, aided by highly selective yet deep local lookahead. Conventional Go \nprograms are carefully (and protractedly) tuned expert systems (Fotland, 1993). \nThey are fundamentally limited by their need for human assistance in compiling \nand integrating domain knowledge, and still play barely above the level of a \nhuman beginner -\na machine learning approach may thus offer considerable \nadvantages. (Brugmann, 1993) has shown that a knowledge-free optimization \napproach to Go can work in principle: he obtained respectable (though inefficient) \nplay by selecting moves through simulated annealing (Kirkpatrick et al., 1983) over \npossible continuations of the game. \nThe pattern recognition component inherent in Go is amenable to connectionist \nmethods. Supervised backpropagation networks have been applied to the game \n(Stoutamire, 1991; Enderton, 1991) but face a bottleneck in the scarcity of hand(cid:173)\nlabelled training data. We propose an alternative approach based on the TD(A) \npredictive learning algorithm (Sutton, 1984; Sutton, 1988; Barto et al., 1983), which \nhas been successfully applied to the game of backgammon by (Tesauro, 1992). \nHis TD-Gammon program uses a backpropagation network to map preselected \nfeatures of the board position to an output reflecting the probability that the player \nto move would win. It was trained by TD(O) while playing only itself, yet learned \nan evaluation function that -\ncoupled with a full two-ply lookahead to pick the \nestimated best move - made it competitive with the best human players in the \nworld (Robertie, 1992; Tesauro, 1994). \nIn an early experiment we investigated a straightforward adaptation of Tesauro's \napproach to the Go domain. We trained a fully connected 82-40-1 backpropagation \nnetwork by randomized! self-play on a 9x9 Go board (a standard didactic size for \nhumans). The output learned to predict the margin of victory or defeat for black. \nThis undifferentiated network did learn to squeak past Wally, a weak public domain \nprogram (Newman, 1988), but it took 659,000 games of training to do so. We have \nfound that the efficiency of learning can be vastly improved through appropriately \nstructured network architectures and training strategies, and these are the focus of \nthe next two sections. \n\n1 Unlike backgammon, Go is a deterministic game, so we had to generate moves stochas(cid:173)\ntically to ensure sufficient exploration of the state space. This was done by Gibbs sam(cid:173)\npling (Geman and Geman, 1984) over values obtained from single-ply search, annealing the \ntemperature parameter from random towards best-predicted play. \n\n\fTemporal Difference Learning of Position Evaluation in the Game of Go \n\n819 \n\nE evaluation \n\n:;;;:::-:- :--s;; \n\n~ ___ ~~ ____ -'~ processed \n\n/ \n\nfeatures \n\n- - -\n\n',----:.> \n\nt \nC ____ c_o_n_s_t_r_a_in_t--::-s_a_t_is_f_a_c_t_io_n ____ ) .. \nt raw feature maps \n\nconnectivity map \n\n~~-----------------~--~ \n\nJ \n\n1 \n\nr \n\nr \n\n~/\"-... \n[::::J [::::) - -- ~~~~:try[::J [::J - --\n~ Go board ii?Q1 ~ \n\nFigure 1: A modular network architecture that takes advantage of board symme(cid:173)\ntries, translation invariance and localized reinforcement to evaluate Go positions. \nAlso shown is the planned connectivity prediction mechanism (see Discussion). \n\n2 NETWORK ARCHITECTURE \n\nOne of the particular advantages of Go for predictive learning is that there is much \nricher information available at the end of the game than just who won. Unlike \nchess, checkers or backgammon, in which pieces are taken away from the board \nuntil there are few or none left, Go stones generally remain where they are placed. \nThis makes the final state of the board richly informative with respect to the course \nof play; indeed the game is scored by summing contributions from each point on \nthe board. We make this spatial credit assignment accessible to the network by \nhaving it predict the fate of every point on the board rather than just the overall \nscore, and evaluate whole positions accordingly. This bears some similarity with \nthe Successor Representation (Dayan, 1993) which also integrates over vector rather \nthan scalar destinies. 2 \nGiven the knowledge-based approach of existing Go programs, there is an embar(cid:173)\nrassment of input features that one might adopt for Go: Wally already uses about \n30 of them, stronger programs disproportionately more. In order to demonstrate \nreinforcement learning as a viable alternative to the conventional approach, how(cid:173)\never, we require our networks to learn whatever set of features they might need. \nThe complexity of this task can be significantly reduced by exploiting a number \n\n2Sharing information within the network across multiple outputs restricts us to A = 0 \nfor efficient implementation of TD( A). Note that although (Tesauro, 1992) did not have this \nconstraint, he nevertheless found A = 0 to be optimal. \n\n\f820 \n\nSchraudolph, Dayan, and Sejnowski \n\nof constraints that hold a priori in this domain. Specifically, patterns of Go stones \nretain their properties under color reversal, reflection and rotation of the board, \nand - modulo the considerable influence of the board edges -\ntranslation. Each \nof these invariances is reflected in our network architecture: \nColor reversal invariance implies that changing the color of every stone in a Go \nposition, and the player whose tum it is to move, yields an equivalent position from \nthe other player's perspective. We build this constraint directly into our networks \nby using antisymmetric input values (+1 for black, -1 for white) and squashing \nfunctions throughout, and negating the bias input when it is white's tum to move. \nGo positions are also invariant with respect to the eightfold (reflection x rotation) \nsymmetry of the square. We provided mechanisms for constraining the network \nto obey this invariance by appropriate weight sharing and summing of derivatives \n(Le Cun et al., 1989). Although this is clearly beneficial during the evaluation of \nthe network against its opponents, it appears to impede the course of learning.3 \nTo account for translation invariance we use convolution with a weight kernel \nrather than multiplication by a weight matrix as the basic mapping operation in \nour network, whose layers are thus feature maps produced by scanning a fixed \nreceptive field across the input. One particular advantage of this technique is the \neasy transfer of learned weight kernels to different Go board sizes. \nIt must be noted, however, that Go is not translation-invariant: the edge of the board \nnot only affects local play but modulates other aspects of the game, and indeed \nforms the basis of opening strategy. We currently account for this by allowing each \nnode in our network to have its own bias weight, giving it one degree of freedom \nfrom its neighbors. This enables the network to encode absolute position at a \nmodest increse in the number of adjustable parameters. Furthermore, we provide \nadditional redundancy around the board edges by selective use of convolution \nkernels twice as wide as the input. \nFigure 1 illustrates the modular architecture suggested by these deliberations. In \nthe experiments described below we implement all the features shown except for \nthe connectivity map and lateral constraint satisfaction, which are the subject of \nfuture work. \n\n3 TRAINING STRATEGIES \n\nTern poral difference learning teaches the network to predict the consequences of \nfollowing particular strategies on the basis of the play they produce. The question \narises as to which strategies should be used to generate the large number of Go \ngames needed for training. We have identified three criteria by which we compare \nalternative training strategies: \n\n\u2022 the computational efficiency of move generation, \n\u2022 the quality of generated play, and \n\u2022 reasonable coverage of plausible Go positions. \n\n3We are investigating possible causes and cures for this phenomenon. \n\n\fTemporal Difference Learning of Position Evaluation in the Game of Go \n\n821 \n\nie. the network's own position eval(cid:173)\n\nTesauro trained TD-Gammon by self-play -\nuation was used in training to pick both players' moves. This technique does not \nrequire any external source of expertise beyond the rules of the game: the network \nis its own teacher. Since Go is a deterministic game, we cannot always pick the \nestimated best move when training by self-play without running the risk of trap(cid:173)\nping the network in some suboptimal fixed state. Theoretically, this should not \nhappen -\nthe network playing white would be able to predict the idiosyncrasies \nof the network playing black, take advantage of them thus changing the outcome, \nand forcing black's predictions to change commensurately- but in practice it is a \nconcern. We therefore pick moves stochastically by Gibbs sampling (Geman and \nGeman, 1984), in which the probability of a given move is exponentially related to \nthe predicted value of the position it leads to through a \"temperature\" parameter \nthat controls the degree of randomness. \nWe found self-play alone to be rather cumbersome for two reasons: firstly, the \nsingle-ply search used to evaluate all legal moves is com putationally intensive -\nand although we are investigating faster ways to accomplish it, we expect move \nevaluation to remain a computational burden. Secondly, learning from self-play is \nsluggish as the network must bootstrap itself out of ignorance without the benefit of \nexposure to skilled opponents. However, there is nothing to keep us from training \nthe network on moves that are not based on its own predictions -\nfor instance, it \ncan learn by playing against a conventional Go program, or even by just observing \ngames between human players. \nWe use three computer opponents to train our networks: a random move generator, \nthe public-domain program Wally (Newman, 1988), and the commercial program \nThe Many Faces of Go (Fotland, 1993). The random move generator naturally doesn't \nplay Go very we1l4 , but it does have the advantages of high speed and ergodicity \n-\na few thousand games of random Go proved an effective way to prime our \nnetworks at the start of training. The two conventional Go programs, by contrast, \nare rather slow and deterministic, and thus not suitable generators of training data \nwhen playing among themselves. However, they do make good opponents for the \nnetwork, which can provide the required variety of play through its Gibbs sam pIer. \nWhen training on games played between such dissimilar players, we must match \ntheir strength so as to prevent trivial predictions of the outcome. Against Many Faces \nwe use standard Go handicaps for this purpose; Wally we modified to intersperse its \nplay with random moves. The proportion of random moves is reduced adaptively \nas the network improves, providing us with an on-line performance measure. \nSince, in all cases, the strategies of both players are intimately intertwined in the \npredictions, one would never expect them to be correct overall when the network \nis playing a real opponent. This is a particular problem when the strategy for \nchoosing moves during learning is different from the policy adopted for 'optimal' \nnetwork play. \n(Samuel, 1959) found it inadvisable to let his checker program \nlearn from games which it won against an opponent, since its predictions might \notherwise reflect poor as well as good play. This is a particularly pernicious form \nof over-fitting -\nthe network can learn to predict one strategy in exquisite detail, \nwithout being able to play well in general. \n\n4In order to ensure a minimum of stability in the endgame, it does refuse to fill in its own \n\neyes - a particular, locally recognizable type of suicidal move. \n\n\f822 \n\nSchraudolph, Dayan, and Sejnowski \n\nhO-+reinf \n\nhl-+reinf \n\narchi tecture \n\n;0 q-q--