{"title": "Cobot: A Social Reinforcement Learning Agent", "book": "Advances in Neural Information Processing Systems", "page_first": 1393, "page_last": 1400, "abstract": "", "full_text": "Cobot: A Social Reinforcement Learning Agent\n\nCharles Lee Isbell, Jr.\nAT&T Labs-Research\n\nChristian R. Shelton\nStanford University\n\nMichael Kearns\n\nUniversity of Pennsylvania\n\nSatinder Singh\nSyntek Capital AT&T Labs-Research\n\nPeter Stone\n\nAbstract\n\nWe report on the use of reinforcement learning with Cobot, a software agent\nresiding in the well-known online community LambdaMOO. Our initial work on\nCobot (Isbell et al.2000) provided him with the ability to collect social statistics\nand report them to users. Here we describe an application of RL allowing Cobot\nto take proactive actions in this complex social environment, and adapt behavior\nfrom multiple sources of human reward. After 5 months of training, and 3171\nreward and punishment events from 254 different LambdaMOO users, Cobot\nlearned nontrivial preferences for a number of users, modi\ufb01ng his behavior based\non his current state. Here we describe LambdaMOO and the state and action\nspaces of Cobot, and report the statistical results of the learning experiment.\n\nIntroduction\n\n1\nWhile most applications of reinforcement learning (RL) to date have been to problems\nof control, game playing and optimization (Sutton and Barto1998), there has been a recent\nhandful of applications to human-computer interaction. Such applications present a number\nof interesting challenges to RL methodology (such as data sparsity and inevitable violations\nof the Markov property). These previous studies focus on systems that encounter human\nusers one at a time, such as spoken dialogue systems (Singh et al.2000).\nIn this paper, we report on an RL-based agent for LambdaMOO, a complex, open-ended,\nmulti-user chat environment, populated by a community of human users with rich and\noften enduring social relationships. Our long-term goal is to build an agent who can learn\nto perform useful, interesting and entertaining actions in LambdaMOO on the basis of user\nfeedback. While this is a deliberately ambitious and underspeci\ufb01ed goal, we describe here\nour implementation, the empirical experiences of our agent so far, and some of the lessons\nwe have learned about this challenging domain.\nIn previous work (Isbell et al.2000), we developed the software agent Cobot, who interacted\nin various ways with LambdaMOO users. Cobot had two primary functions. First, Cobot\ngathered \u201csocial statistics\u201d (e.g. how frequently and in what ways users interacted with\none another), and provided summaries of these statistics as a service. Second, Cobot had\nrudimentary chatting abilities based on the application of information retrieval methods\nto large documents. The original Cobot was entirely reactive  , in that he \u0001 never initiated\ninteraction with human users, but would only respond to their actions. As we documented\nin our earlier paper, Cobot proved tremendously popular with LambdaMOO users, setting\nthe stage for our current efforts.\nWe modi\ufb01ed Cobot to allow him to take certain actions (such as proposing conversation\ntopics, introducing users, or engaging in common word play routines) under his own ini-\ntiative. The hope is to build an agent that will eventually take unprompted actions that are\nmeaningful, useful or amusing to users. Rather than hand-code complex rules specifying\n\n\u0002 Here we mean \u201cresponding only to human-invoked interaction\u201d, rather than \u201cnon-deliberative\u201d.\n\u0003 Characters in LambdaMOO have gender. Cobot\u2019s description to users indicates that he is male.\n\n\fwhen each action is appropriate (rules that would be inaccurate and quickly become stale),\nwe wanted Cobot to learn the individual and communal preferences of users. Thus, we pro-\nvided a mechanism for users to reward or punish Cobot, and programmed Cobot to use RL\nalgorithms to alter his behavior on the basis of this feedback. The application of RL (or any\nmachine learning methodology) to such an environment presents a number of interesting\ndomain-speci\ufb01c challenges, including:\n\n Choice of an appropriate state space. To learn how to act in a social environment such\nas LambdaMOO, Cobot must represent the salient features. These should include social\ninformation such as which users are present, how experienced they are in LambdaMOO,\nhow frequently they interact with one another, and so on.\n\n Multiple reward sources. Cobot lives in an environment with multiple, often con\ufb02icting\nsources of reward from different human users. How to integrate these sources reasonably\nis a nontrivial empirical question.\n\nInconsistency and drift of user rewards and desires. Individual users may be inconsis-\ntent in the rewards they provide (even when they implicitly have a \ufb01xed set of preferences),\nand their preferences may change over time (for example, due to becoming bored or irri-\ntated with an action). Even when their rewards are consistent, there can be great temporal\nvariation in their reward pattern.\n\n Variability in user understanding. There is great variation in users\u2019 understanding of\n\nCobot\u2019s functionality, and the effects of their rewards and punishments.\n\n Data sparsity. Training data is scarce for many reasons, including user \ufb01ckleness, and the\n\nneed to prevent Cobot from generating too much spam in the environment.\n\nIrreproducibility of experiments. As LambdaMOO is a globally distributed community\nof human users, it is virtually impossible to replicate experiments taking place there.\n\nWe do not have any simple answers (nor do we believe that simple answers exist), but here\nwe provide a case study of our choices and \ufb01ndings. Our primary \ufb01ndings are:\n\nInappropriateness of average reward. We found that the average reward that Cobot re-\nceived over time, the standard measure of success for RL experiments, is an inadequate and\nperhaps even inappropriate metric of performance in the LambdaMOO domain. Reasons\ninclude that user preferences are not stationary, but drift as users become habituated or\nbored with Cobot\u2019s behavior; and the tendency for satis\ufb01ed users to stop providing Cobot\nwith any feedback, positive or negative. Despite the inadequacy of average reward, we are\nstill able to establish several measures by which Cobot\u2019s RL succeeds, discussed below.\n\n A small set of dedicated \u201cparents\u201d. While many users provided only a moderate or small\namount of RL training (rewards and punishments) to Cobot, a handful of users did invest\nsigni\ufb01cant time in training him.\n\n Some parents have strong opinions. While many of the users that trained Cobot did\nnot exhibit clear preferences for any of his actions over the others, some users clearly and\nconsistently rewarded and punished particular actions over the others.\n\n Cobot learns matching policies. For those users who exhibited clear preferences through\ntheir rewards and punishments, Cobot successfully learned corresponding policies of be-\nhavior.\n\n Cobot responds to his dedicated parents. For those users who invested the most train-\ning time in Cobot, the observed distribution of his actions is signi\ufb01cantly altered by their\npresence.\n\n Some preferences depend on state. Although some users for whom we have suf\ufb01cient\ndata seem to have preferences that do not depend upon the social state features we con-\nstructed for the RL, others do in fact appear to change their preferences depending upon\nprevailing social conditions.\n\nThe outline for the rest of the paper is as follows. In Section 2, we give brief background\non LambdaMOO. In Section 3, we describe our earlier (non-RL) work on Cobot. Section 4\nprovides some brief background on RL. In Sections 5, 6 and 7 we describe our implemen-\ntation of Cobot\u2019s RL action space, reward mechanisms and state features, respectively. Our\nprimary \ufb01ndings are presented in Section 8, and Section 9 offers conclusions.\n\n\n\n\n\f2 LambdaMOO\n\nLambdaMOO, founded in 1990 by Pavel Curtis at Xerox PARC, is the oldest continuously\noperating MUD, a class of online worlds with roots in text-based multiplayer role-playing\ngames. MUDs (multi-user dungeons) differ from most chat and gaming systems in their\nuse of a persistent representation of a virtual world, often created by the participants, who\nare represented as characters of their own choosing. LambdaMOO appears as a series of\ninterconnected rooms, populated by users and objects who may move between them. Each\nroom provides a shared chat channel, and typically has an elaborate text description that\nimbues it with its own \u201clook and feel.\u201d In addition to speech, users express themselves via\na large collection of verbs, allowing a rich set of simulated actions, and the expression of\nemotional states:\n\nBuster is overwhelmed by all these deadlines.\nBuster begins to slowly tear his hair out, one strand at a time.\n\n(1)\n(2)\n(3) HFh comforts Buster.\n(4) HFh [to Buster]: Remember, the mighty oak was once a nut like you.\n(5)\n(6)\n\nBuster [to HFh]: Right, but his personal growth was assured. Thanks anyway, though.\nBuster feels better now.\n\nLines (1) and (2) are initiated by verb commands by user Buster, expressing his emotional\nstate, while lines (3) and (4) are examples of verbs and speech acts, respectively, by HFh.\nLines (5) and (6) are speech and verb acts by Buster. Though there are many standard verbs,\nsuch as the use of the verb comfort in line (3) above, the variety is essentially unlimited,\nas players have the ability to create their own verbs.\nThe rooms and objects in LambdaMOO are created by users themselves, who devise de-\nscriptions, and control access by other users. Users can also create objects with verbs that\ncan be invoked by other players. As last count, the database contains 118,154 objects,\nincluding 4836 active user accounts. LambdaMOO\u2019s long existence and its user-created\nnature combine to give it one of the strongest senses of virtual community in the on-line\nworld. Many users have interacted extensively with each other over many years, and users\nare widely acknowledged for their contribution of interesting objects. LambdaMOO is an\nattractive environment for experiments in AI (Foner1997; Mauldin1994), including learn-\ning. The population is generally curious and technically savvy, and users are interested in\nautomated objects meant to display some form of intelligence.\n\n3 Cobot\n\nCobot is a software agent residing in LambdaMOO. Like a human user, he connects via tel-\nnet, and from the point of view of the LambdaMOO server, is a user with all the rights and\nresponsibilities implied. Once actually connected, Cobot wanders into the Living Room,\nwhere he spends most of his time. The Living Room is a central public place, frequented\nboth by many regulars, and by users new to LambdaMOO. There are several permanent\nobjects in the Living Room, including a couch with various features and a cuckoo clock.\nThe Living Room usually has between \ufb01ve and twenty users, and is perpetually busy. Over\na year, Cobot noted over 2.5 million separate events (about one event every eleven seconds)\nPreviously, we implemented a variety of functionality on Cobot centering around gather-\ning and reporting social statistics. Cobot notes who takes what actions, and on whom.\nCobot can answer queries about these statistics, and describe the similarities and differ-\nences between users. He also has a rudimentary chatting ability based on the application\nof information retrieval methods to large documents. He can also search the web to answer\nspeci\ufb01c questions posed to him. A more complete description of Cobot\u2019s abilities, and his\nearly experiences as a social agent in LambdaMOO, can be found in (Isbell et al.2000).\nOur focus here is to make Cobot proactive\u2014i.e., let him take actions under his own\ninitiative\u2014in a way that is useful, interesting, or pleasing to LambdaMOO users.\nIt is\nimpossible to program rules anticipating when any given action is appropriate in such a\ncomplex and dynamic environment, so we applied reinforcement learning to adapt directly\nfrom user feedback. We emphasize that Cobot\u2019s original reactive functionality remained\non during the RL experiment. Cobot\u2019s persona is largely due to this original functionality,\nand we felt it was most interesting, and even necessary, to add RL work in this context.\n\n\fNull Action\nTopic Change (4)\n\nRoll Call (2)\n\nSocial\nCommentary\n\nIntroductions\n\nChoose to remain silent for this time period.\nIntroduce a conversational topic. Cobot declares that he wants to discuss sports\nor politics, or he utters a sentence from either the sports section or political\nsection of the Boston Globe.\nInitiate a \u201croll call,\u201d a common word play routine in LambdaMOO. For ex-\nample, someone who is tired of Monica Lewinsky may emote \u201cTIRED OF\nLEWINSKY ROLL CALL.\u201d Sympathetic users agree with the roll call. Cobot\ntakes a recent utterance, and extracts either a single noun, or a verb phrase.\nMake a comment describing the current social state of the Living Room, such\nas \u201cIt sure is quiet\u201d or \u201cEveryone here is friendly.\u201d These statements are based\non Cobot\u2019s statistics from recent activity. Several different utterances possible,\nbut they are treated as a single action for RL purposes.\nIntroduce two users who have not yet interacted in front of Cobot.\n\nTable 1: The 9 RL actions available to Cobot.\n\n4 RL Background\nIn RL, problems of decision-making by agents interacting with uncertain environments are\nusually modeled as Markov decision processes (MDPs). In the MDP framework, at each\ntime step the agent senses the state of the environment, and chooses and executes an action\nfrom the set of actions available to it in that state. The agent\u2019s action (and perhaps other\nuncontrolled external events) cause a stochastic change in the state of the environment. The\nagent receives a (possibly zero) scalar reward from the environment. The agent\u2019s goal is to\nchoose actions so as to maximize the expected sum of rewards over some time horizon. An\noptimal policy is a mapping from states to actions that achieves the agent\u2019s goal.\nMany RL algorithms have been developed for learning good approximations to an optimal\npolicy from the agent\u2019s experience in its environment. At a high level, most algorithms\n-values) that map state-action pairs to\nuse this experience to learn value functions (or \nthe maximal expected sum of reward that can be achieved starting from that state-action\npair. The learned value function is used to choose actions stochastically, so that in each\nstate, actions with higher value are chosen with higher probability. In addition, many RL\nalgorithms use some form of function approximation (parametric representations of com-\nplex value functions) both to map state-action features to their values and to map states to\ndistributions over actions (i.e., the policy). See (Sutton and Barto1998) for an extensive\nintroduction to RL.\nIn the next sections, we describe the Cobot\u2019s actions, our choice of state features, and how\nwe dealt with multiple sources of reward. The particular RL algorithm we use is a variant\nof\nIts details are beyond the scope of\nthis paper; however, see (Shelton2000) for details. One aspect of our RL algorithm that\nis relevant to understanding our results is that we use a linear function approximator to\nstore our policy. In other words, for each state feature, we maintain a vector of real-valued\nweights indexed by the possible actions. A positive weight for some action means that the\nfeature increases the probability of taking that action, while a negative weight decreases\nthe probability. The weight\u2019s magnitude determines the strength of this contribution.\n5 Cobot\u2019s RL Actions\nTo have any hope of learning to behave in a way interesting to LambdaMOO users, Cobot\u2019s\nactions must \u201cmake sense\u201d to them, \ufb01t in with the social chat-based environment, and\nminimize the risk of causing irritation. Conversation, word play, and emoting routines are\namong the most common activity in LambdaMOO, so we designed a set of actions along\nthese lines, as detailed in Table 1. Many of these actions extract an utterance from the\nrecent conversations, or from a continually changing external source, such as the online\nBoston Globe. Thus a single action may cause an in\ufb01nite variety of behavior by Cobot.\nAt set time intervals (only every few minutes on average, to minimize spam), Cobot selects\nan action to perform from this set according to a distribution determined by the Q-values\nin his current state. Any rewards or punishments received before the next RL action are\nattributed to the current action, and used to update Cobot\u2019s value functions. It is worth\nremembering that Cobot has two different categories of action: those actions taken proac-\ntively as a result of the RL, and those actions taken in response to a user\u2019s action towards\nCobot. Some users are certainly aware of the distinction and can easily determine which\n\n(Sutton et al.1999)\u2019s policy gradient algorithm.\n\n\factions fall into which category, but other users may occasionally reward or punish Cobot\nin response to a reactive action. Such \u201cerroneous\u201d rewards and punishments act as a source\nof noise in the training process.\n6 The RL Reward Function\nCobot learns to behave directly from the feedback of LambdaMOO users, any of whom\ncan reward or punish him. There are both explicit and implicit feedback mechanisms. We\nimplemented explicit reward and punish verbs on Cobot that LambdaMOO users can\ninvoke at any time. These verbs give a numerical (positive and negative, respectively)\ntraining signal to Cobot that is the basis of the RL. The signal is attributed as immediate\nfeedback for the current state and RL action, and \u201cbacked up\u201d to previous states and actions\nin accordance with the standard RL algorithms.\nThere are several standard LambdaMOO verbs that are commonly used to express, some-\ntimes playfully, approval or disapproval. Examples of the former include the verb hug, and\nof the latter the verb spank. In the interest of allowing the RL process to integrate naturally\nwith the LambdaMOO environment, we chose to accept a number of such verbs as implicit\nreward and punishment signals for Cobot; however, such implicit feedback is numerically\nweaker than the feedback generated by the explicit mechanisms.\nOne fundamental design choice is whether to learn a single value function for the entire\ncommunity, or to learn separate value functions for each user based on individual feedback,\ncombining the value functions of those present to determine how to act at each moment.\nWe opted for the latter for three primary reasons.\nFirst, it was clear that for learning to have any hope of success, ths system must represent\nwho is present at any given moment\u2014different users simply have different personalities and\npreferences. We felt that representing which users are present as additional state features\nwould throw away valuable domain information, as the RL would have to discover on its\nown the primacy of user identity. Having separate reward functions for each user is thus a\nway of asserting the importance of identity to the learning process.\nSecond, despite the extremely limited number of training examples available in this domain\nper month), learning must be quick and signi\ufb01cant. Without a clear sense that their\n(\n\u0002\u0001\u0004\u0003\u0005\u0003\ntraining has some impact on Cobot\u2019s behavior, users will quickly lose interest in providing\nfeedback. A known challenge for RL is the \u201ccurse of dimensionality,\u201d (i.e. the size of the\nstate space increases exponentially with the number of state features). By avoiding the need\nto represent the presence or absence of roughly 250 users, we are able to maintain a fairly\nsmall state space and so speed up learning.\nThird, we (correctly) anticipated the fact that certain users would provide an inordinate\namount of training to Cobot, and we did not want the overall policy followed by Cobot\nto be dominated by the preferences of these individuals. By learning separate policies for\neach user, and then combining these policies among those users present, we can limit the\nimpact any single user can have on Cobot\u2019s actions.\n7 Cobot\u2019s RL State Features\nThe decision to maintain and learn separate value functions for each user means that we\ncan maintain separate state spaces as well, in the hopes of simplifying states and speeding\nlearning. Cobot can be viewed as running a large number of separate RL processes in\nparallel, with each process having a different state space. The state space for a user contains\na number of features containing statistics about that particular user.\nLambdaMOO is a social environment, and Cobot is learning to take social actions, so we\nfelt that his state features should contain information allowing him to gauge social activity\nand relationships. Table 2 provides a description of the state features used for RL by Cobot\nfor each user. Even though we have simpli\ufb01ed the state space by partitioning by user, the\nstate space for a single user remains suf\ufb01ciently complex to preclude standard table-based\nrepresentation of value functions (also, each user\u2019s state space is effectively in\ufb01nite, as\nthere are real-valued state features). Thus, linear function approximation is used for each\nuser\u2019s policy. Cobot\u2019s RL actions are then chosen according to a mixture of the policies of\nthe users present. We refer the reader to (Shelton2000) for more details on the method by\nwhich policies are learned and combined.\n\n\fSocial Summary\nVector\n\nMood Vector\n\nRates Vector\nCurrent Room\nRoll Call Vector\n\nBias\n\nA vector of four numbers: the rate at which the user is producing events; the\nrate at which events are being produced that are directed at the user; the per-\ncentage of the other users present who are among this user\u2019s ten most fre-\nquently interacted-with users (\u201cplaymates\u201d); and the percentage of the other\nusers present for whom this user is among their top ten playmates.\nA vector measuring the recent use of eight groups of common verbs (e.g., one\ngroup includes verbs grin and smile). Verbs were grouped according to how\nwell their usage was correlated.\nA vector measuring the rate at which events are produced by those present.\nThe room where Cobot currently resides.\nIndicates if Cobot\u2019s currently saved roll call text has been used before, if some-\none has done a roll call since the last time Cobot did, and if there has been a\nroll call since the last time Cobot grabbed new text.\nEach user has one feature that is always \u201con\u201d; that is, this bias is always set to\na value of 1. Intuitively, it is the feature indicating the user\u2019s \u201cpresence.\u201d\n\nTable 2: State space of Cobot. Each user has his own state space and value function; the table thus\ndescribes the state space maintained for a generic user.\n8 Experimental Procedure and Findings\nCobot has been present in LambdaMOO more or less continuously since September, 1999.\nThe RL version of Cobot debuted May 10, 2000. Again, Cobot\u2019s various reactive func-\ntionality was left intact for the duration of the RL experiment. Cobot is a working system\nwith real human users, and we wanted to perform the RL experiment in this context. Upon\nlaunching the RL functionality publicly in the Living Room, Cobot logged all RL-related\ndata (states visited, actions taken, rewards received from each user, parameters of the value\nfunctions, etc.) from May 10 until October 10, 2000. During this time, 63123 RL actions\nwere taken (in addition, of course, to many more reactive non-RL actions), and 3171 re-\nward and punishment events were received from 254 different users. The \ufb01ndings we now\nsummarize are based on these extensive logs:\nInappropriateness of average reward. The most standard and obvious sign of successful RL would\nbe an increase in the average reward over time. Instead, as shown in Figure 1a, the average cumu-\nlative reward received by Cobot actually goes down. However, rather than indicating that users are\nbecoming more dissatis\ufb01ed as Cobot learns, the decay in reward reveals some peculiarities of human\nfeedback in such an open-ended environment. There are at least two dif\ufb01culties with average cumu-\nlative reward in an environment of human users. The \ufb01rst is that humans are \ufb01ckle, and their tastes\nand preferences may drift over time. Indeed, our experiences as users, and with the original reactive\nfunctionality of Cobot, suggest that novelty is highly valued in LambdaMOO. Thus a feature that is\npopular and exciting to users when it is introduced may eventually become an irritant (there are many\nexamples of this phenomenon). In RL terminology, we do not have a \ufb01xed, consistent reward func-\ntion, and thus we are always learning a moving target. While dif\ufb01cult to quantify in such a complex\nenvironment, this phenomenon is suf\ufb01ciently prevalent in LambdaMOO to cast serious doubts on the\nuse of average cumulative reward as the primary measure of performance.\n\nThe second and related dif\ufb01culty is that even when users do maintain relatively \ufb01xed preferences,\nthey tend to give Cobot less feedback of either type (reward or punishment) as he manages to learn\ntheir preferences accurately. Simply put, once Cobot seems to be behaving as they wish, users feel\nno need to continually provide reward for his \u201ccorrect\u201d actions or to punish him for the occasional\n\u201cmistake.\u201d This reward pattern is in contrast to typical RL applications, where there is an automated\nand indefatigable reward source. Strong empirical evidence for this second phenomenon is provided\nby User M and User S. These two users were among Cobot\u2019s most dedicated trainers, each had strong\npreferences for certain actions, and Cobot learned to strongly modify his behavior in their presence to\nmatch their preferences. Nevertheless, both users tended to provide less frequent feedback to Cobot\nas the experiment progressed, as shown in Figure 1a. We conclude that there are serious conceptual\ndif\ufb01culties with the use of average cumulative reward in such a human-centric application of RL, and\nthat alternative measures must be investigated, which we do below.\nA small set of dedicated \u201cparents.\u201d Among the 254 users who gave at least one reward or punish-\nment event to Cobot, 218 gave 20 or fewer, while 15 gave 50 or more. Thus, we found that while\nmany users exhibited a passing interest in training Cobot, there was a small group that was willing to\ninvest nontrivial time and effort in teaching Cobot their preferences. In particular, User M and User\nS, generated 594 and 69 rewards and punishments events, respectively.\n\nBy \u201cevent\u201d, we simply mean an RL action that received some feedback. The actual absolute\n\n\n\n\fUser O\n\nUser B\n\nUser C\n\nUser P\n\n\u0002\u0001\u0004\u0003\n\u0005\u0007\u0006\t\b\n\n\u0007\n\n\u0002\n\u0005\u0002\u0005\n\nRoll Call. User O appears to especially dislike roll call actions when there have\nbeen repeated roll calls and/or Cobot is repeating the same roll calls.\nRates. The overall rate of events being generating has slightly more relevance\nthan that of the rate of events being generated just by User O.\nSocial Summary. User B is effected by the presence of his friends. Not shown\nhere are other Social Summary features (deviating about 6 degrees). It appears\nthat User B is more likely to ignore Cobot when he is with many friends.\nRoll Call. User C appears to have strong preferences about Cobot\u2019s behavior\nwhen a \u201croll call party\u201d is in progress (i.e., everyone is generating roll calls).\nRoom. User P would follow Cobot to his home, where he is generally alone, and\nhas trained him there. He appears to have different preferences for Cobot under\nthose circumstances.\n\nTable 3: Relevant features for users with non-uniform policies. Several of our top users had some\nfeatures that deviated from their bias feature. The second column indicates the number of degrees\nbetween the weight vectors for those features and the weight vectors for the bias feature. We have\nonly included features that deviated by more than 10 degrees. For the users above the double line,\nwe have included only features whose weights had a length greater than 0.2. Each of these users had\nbias weights of length greater than 1. For those below the line, we have included only features with\na length greater than 0.1 (these all had bias weights of length much less than 1).\n\nSome parents have strong opinions. For the vast majority of users who participated in the RL\ntraining of Cobot, the policy learned was quite close to the uniform distribution. Quanti\ufb01cation of\nthis statement is somewhat complex, since policies are dependent on state. However, we observed\nthat for most users the learned policy\u2019s dependence on state was weak, and the resulting distribution\nnear uniform (though there are interesting and notable exceptions, as we shall see below). This\nresult is perhaps to be expected: most users provided too little feedback for Cobot to detect strong\npreferences, and may not have been exhibiting strong and consistent preferences in the feedback\nthey did provide. However, there was again a small group of users for whom a highly non-uniform\npolicy was learned. In particular, for Users M and S mentioned above, the resulting policies were\nrelatively independent of state and their entropies were 0.03 and 1.93, respectively. (The entropy of\nthe uniform distribution over the actions is 2.2.) Several other users also exhibited less dramatic but\nstill non-uniform distributions. User M seemed to have a strong preference for roll call actions, with\nthe learned policy selecting these with probability 0.99, while User S preferred social commentary\nactions, with the learned policy giving them probability 0.38. (Each action in the uniform distribution\nis given weight 1/9 = 0.11.)\n\nCobot learns matching policies.\nIn Figure 1b, we demonstrate that the policy learned by Cobot\nfor User M does in fact re\ufb02ect the empirical pattern of rewards received over time. Similar results\nobtain for User S, not shown here. Thus, repeated feedback given to Cobot for a non-uniform set of\npreferences clearly pays off in a corresponding policy.\n\nCobot responds to his dedicated parents. The policies learned by Cobot for users can have strong\nimpact on the empirical distribution of actions he actually ends up taking in LambdaMOO. For User\nM, we \ufb01nd that his presence causes a signi\ufb01cant shift towards his preferences. In other words, Cobot\ndoes his best to \u201cplease\u201d these dedicated trainers whenever they arrive in the Living Room, and\nreturns to a more uniform policy upon their departure.\n\nSome preferences depend on state. Finally, we show that the policies learned by Cobot sometimes\ndepend upon the features Cobot maintains in his state. We use two facts about the RL weights\n(described in Section 4) maintained by Cobot to determine which features are relevant for a given\nuser. First, we note that by construction, the RL weights learned for the bias feature described in\nTable 2 represent the user\u2019s preferences independent of state (since this feature is always on whenever\nthe user is present). Second, we note that because we initialized all weights to 0, only features with\nnon-zero weights will contribute to the policy that Cobot uses. Thus, we can determine that a feature\nis relevant for a user if that feature\u2019s weight vector is far from that user\u2019s bias feature weight vector,\nand from the all-zero vector. For our purposes, we have used (1) the normalized inner product (the\ncosine of the angle between two vectors) as a measure of a feature\u2019s distance from the bias feature,\nand (2) a feature\u2019s weight vector length to determine if it is away from zero. These measures show that\nfor most users, Cobot learned a state-independent policy (e.g., User M prefers roll calls); however, as\nwe can see in Table 3, Cobot has learned a policy for some users that depends upon state.\nnumerical reward received may be larger or smaller than 1 at any time, as implicit rewards provide\nfractional reward, and the user may repeatedly reward or punish an action, with the feedback being\nsummed. For example, the total absolute value of rewards and punishments provided by User M was\n607.63 over 594 feedback events.\n\n\u0003\n\u000b\n\u0001\n\u0003\n\u0003\n\u0003\n\fAverage Cumulative Reward per Timestep\n\nRewards / Policy / Empirical Distribution Comparision for User M\n\nreward all users\nabs reward all users\nreward user M\nabs reward user M\nreward user S\nabs reward user S\n\ne\ng\nn\na\nh\nc\n\n1\n\n0.8\n\n0.6\n\n0.4\n\n0.2\n\n0\n\n\u22120.2\n\n\u22120.4\n\nrewards\npolicy\nempirical\n\n1\n\n2\n\n3\n\n4\n\n5\n\naction\n\n6\n\n7\n\n8\n\n9\n\n0.2\n\n0.18\n\n0.16\n\n0.14\n\n0.12\n\n0.1\n\n0.08\n\n0.06\n\n0.04\n\n0.02\n\nd\nr\na\nw\ne\nR\n\n0\n\n0\n\n1\n\n2\n\n3\nTime\n\n4\n\n5\n\n6\nx 104\n\nFigure 1: a) Average Cumulative Reward Over Time. b) Rewards received, policy learned, and\neffect on actions for User M. Figure a) shows that average cumulative reward decreases over time,\nfor both total and absolute reward; however, Figure b shows that proper learning is taking place. For\neach of the RL actions, three quantities are shown. The blue bars (left) show the average reward\ngiven by User M for each action (the average reward given by User M across all actions has been\nsubtracted off to indicate relative preferences). The yellow bars (middle) show the policy learned by\nCobot for User M (the probability assigned to each action in the uniform distribution (1/9) has been\nsubtracted off). The red bars (right) show the empirical frequency with which each action was taken\nin the presence of User M (minus the empirical frequency with which that action was taken by Cobot\nover all time steps). These bars indicate the extent to which the presence of User M biases Cobot\u2019s\nbehavior towards M\u2019s preferences. We see that the policy learned by Cobot for User M aligns nicely\nwith the preferences expressed by M and that Cobot\u2019s behavior shifts strongly towards the learned\npolicy for User M whenever M is present. To go beyond a qualitative visual analysis, we have de\ufb01ned\na metric that measures the extent to which two rankings of actions agree, while taking into account\nthat some actions are extremely close in the each ranking. The details are beyond the scope of the\npaper, but the agreement between the action rankings shown here are in near-perfect agreement by\nthis measure. Similar results obtain for User S.\n\n9 Conclusions\nWe have reported on our efforts to apply reinforcement learning in a complex human online\nsocial environment where many of the standard assumptions (stationary rewards, Marko-\nvian behavior, appropriateness of average reward) are clearly violated. We feel that the\nresults obtained with Cobot so far are compelling, and offer promise for the application\nof RL in such open-ended social settings. Cobot continues to take RL actions and receive\nrewards and punishments from LambdaMOO users, and we plan to continue and embellish\nthis work as part of our overall efforts on Cobot.\nReferences\nFoner, L. (1997). Entertaining Agents: a Sociological Case Study.\n\nIn Proceedings of the First\n\nInternational Conference on Autonomous Agents.\n\nIsbell, C. L., Kearns, M., Kormann, D., Singh, S., and Stone, P. (2000). Cobot in LambdaMOO: A\n\nSocial Statistics Agent. To appear in Proceedings of AAAI-2000.\n\nMauldin, M. (1994). Chatterbots, TinyMUDs, and the Turing Test: Entering the Loebner Prize\n\nCompetition. In Proceedings of the Twelfth National Conference on Arti\ufb01cial Intelligence.\n\nShelton, C. R. (2000). Balancing Multiple Sources of Reward in Reinforcement Learning. Submitted\n\nfor publication in Neural Information Processing Systems-2000.\n\nSingh, S., Kearns, M., Littman, D., and Walker, M. (2000). Empirical Evaluation of a Reinforcement\n\nLearning Dialogue System. To appear in Proceedings of AAAI-2000.\n\nSutton, R. S. and Barto, A. G. (1998). Reinforcement Learning: An Introduction. MIT Press, Cam-\n\nbridge, MA.\n\nSutton, R. S., McAllester, D., Singh, S., and Mansour, Y. (1999). Policy gradient methods for rein-\nforcement learning with function approximation. In Neural Information Processing Systems-\n1999.\n\n\f", "award": [], "sourceid": 2118, "authors": [{"given_name": "Charles", "family_name": "Isbell", "institution": null}, {"given_name": "Christian", "family_name": "Shelton", "institution": null}]}