{"title": "Direct memory access using two cues: Finding the intersection of sets in a connectionist model", "book": "Advances in Neural Information Processing Systems", "page_first": 635, "page_last": 641, "abstract": null, "full_text": "Direct memory access using two cues: Finding \nthe intersection of sets in a connectionist model \n\nJanet Wiles, Michael S. Humphreys, John D. Bain and Simon Dennis \n\nDepartments of Psychology and Computer Science \nUniversity of Queensland QLD 4072 Australia \n\nemail: janet@psych.psy.uq.oz.au \n\nAbstract \n\nFor lack of alternative models, search and decision processes have provided the \ndominant paradigm for human memory access using two or more cues, despite \nevidence against search as an access process (Humphreys, Wiles & Bain, 1990). \nWe present an alternative process to search, based on calculating the intersection \nof sets of targets activated by two or more cues. Two methods of computing \nthe intersection are presented, one using information about the possible targets, \nthe other constraining the cue-target strengths in the memory matrix. Analysis \nusing orthogonal vectors to represent the cues and targets demonstrates the \ncompetence of both processes, and simulations using sparse distributed \nrepresentations demonstrate the performance of the latter process for tasks \ninvolving 2 and 3 cues. \n\n1 INTRODUCTION \nConsider a task in which a subject is asked to name a word that rhymes with oast. The \nsubject answers \"most\", (or post, host, toast, boast, ... ). Now the subject is asked to find \na word that means a mythical being that rhymes with oast. She or he pauses slighUy and \nreplies \"ghost\". \n\nThe difference between the first and second questions is that the first requires the use of \none cue to access memory. The second question requires the use of two cues - either \ncombining them before the access process, or combining the targets they access. There \nare many experimental paradigms in psychology in which a subject uses two or more \ncues to perform a task (Rubin & Wallace, 1989). One default assumption underlying \n\n635 \n\n\f636 Wiles, Humphreys, Bain, and Dennis \n\nmany explanations for the effective use of two cues relies on a search process through \nmemory. \n\nModels of human memory based on associative access (using connectionist models) have \nprovided an alternative paradigm to search processes for memory access using a single cue \n(Anderson, Silverstein, Ritz & Jones, 1977; McClelland & Rumelhart, 1986), and for \ntwo cues which have been studied together (Humphreys, Bain & Pike 1989). In some \nrespects, properties of these models correspond very closely to the characteristics of \nhuman memory (Rumelhart, 1989). In addition to the evidence against search processes \nfor memory access using a single cue, there is also experimental evidence against \nsequential search in some tasks requiring the combination of two cues, such as cued recall \nwith an extra-list cue, cued recall with a part-word cue, lexical access and semantic access \n(Humphreys, Wiles & Bain, 1990). Furthermore, in some of these tasks it appears that \nthe two cues have never jointly occurred with the target. In such a situation, the tensor \nproduct employed by Humphreys et. a1. to bind the two cues to the target cannot be \nemployed, nor can the co-occurrences of the two cues be encoded into the hidden layer of a \nthree-layer network. In this paper we present the computational foundation for an \nalternative process to search and decision, based on parallel (or direct) access for the \nintersection of sets of targets that are retrieved in response to cues that have not been \nstudied together. \nDefinition of an intersection in the cue-target paradigm: Given a set of cue-target pairs, \nand two (or more) access cues, then the intersection specified by the access cues is defined \nto be the set of targets which are associated with both cues. If the cue-target strengths are \nnot binary, then they are constrained to lie between 0 and 1, and targets in the intersection \nare weighted by the product of the cue-target strengths. A complementary definition for a \nunion process could be the set of targets associated with anyone or more of the access \ncues, weighted by the sum of the target strengths. \n\nIn the models that are described below, we assume that the access cues and targets are \nrepresented as vectors, the cue-target associations are represented in a memory matrix and \nthe set of targets retrieved in response to one or more cues is represented as a linear \ncombination, or blend, of target vectors associated with that cue or cues. Note that under \nthis definition, if there is more than one target in the intersection, then a second stage is \nrequired to select a unique target to output from the retrieved linear combination. We do \nnot address this second stage in this paper. \n\nA task requiring intersection: In the rhyming task described above, the rhyme and \nsemantic cues have extremely low separate probabilities of accessing the target, ghost, \nbut a very high joint probability. In this study we do not distinguish between the \nrepresentation of the semantic and part-word cues, although it would be required for a \nmore detailed model. Instead, we focus on the task of retrieving a target weakly associated \nwith two cues. We simulate this condition in a simple task using two cues, C1 and C2, \nand three targets, T1, T2 and T3. Each cue is strongly associated with one target, and \nweakly associated with a second target, as follows (strengths of association are shown \nabove the arrows): \n\n\fDirect Memory Access Using Two Cues \n\n637 \n\nThe intersection of the targets retrieved to the two cues, Cl and C2, is the target, T2, \nwith a strength of 0.01. Note that in this example, a model based on vector addition \nwould be insufficient to select target, T2, which is weakly associated with both cues, in \npreference to either target, Tl or TJ, which are strongly associated with one cue each. \n\n2 IMPLEMENTATIONS OF INTERSECTION PROCESSES \n\n2.1 LOCAL REPRESENTATIONS \n\nGiven a local representation for two sets of targets, their intersection can be computed by \nmultiplying the activations elicited by each cue. This method extends to sparse \nrepresentations with some noise from cross product terms, and has been used by Dolan \nand Dyer (1989) in their tensor model, and Touretzky and Hinton (1989) in the \nDistributed Connectionist Production System (for further discussion see Wiles, \nHumphreys, Bain & Dennis, 1990). However, multiplying activation strengths does not \nextend to fully distributed representations, since multiplication depends on the basis \nrepresentation (Le., the target patterns themselves) and the cross-product terms do not \nnecessarily cancel. One strong implication of this for implementing an intersection \nprocess, is that the choice of patterns is not critical in a linear process (such as vector \naddition) but can be critical in a non-linear process (which is necessary for computing \nintersections). An intersection process requires more information about the target patterns \nthemselves. \n\nIt is interesting to note that the inner product of the target sets (equivalent to the match \nprocess in Humphreys et. al.1s (1989) Matrix model) can be used to determine whether or \nnot the intersection of targets is empty, if the target vectors are orthogonal, although it \ncannot be used to find the particular vectors which are in the intersection. \n\n2.2 USING INFORMATION ABOUT TARGET VECfORS \n\nA local representation enables multiplication of activation strengths because there is \nimplicit knowledge about the allowable target vectors in the local representation itself. \nThe first method we describe for computing the intersection of fully distributed vectors \nuses information about the targets, explicitly represented in an auto-associative memory, \nto filter out cross-product terms: In separate operatiOns, each cue is used to access the \nmemory matrix and retrieve a composite target vector (the linear combination of \nassociated targets). A temporary matrix is formed from the outer product of these two \ncomposite vectors. This matrix will contain product terms between all the targets in the \nintersection set as well as noise in the form of cross-product terms. The cross-product \nterms can be filtered from the temporary matrix by using it as a retrieval cue for accessing \na three-dimensional auto-associator (a tensor of rank 3) over all the targets in the original \nmemory. \nIf the target vectors are orthonormal, then this process will produce a vector \nwhich contains no noise from cross-product terms, and is the linear combination of all \ntargets associated with both cues (see Box 1). \n\n\f638 Wiles, Humphreys, Bain, and Dennis \n\nBox 1. Creating a temporary matrix from the product of the target vectors, then filtering \nout the noise terms: Let the cues and targets be represented by vectors which are mutually \northonormal (Le., Ci.Ci = Ti.Ti = 1, Ci,Cj = Ti.Tj = 0, i, j = 1,2,3). The memory \nmatrix, M, is formed from cue-target pairs, weighted by their respective strengths, as \nfollows: \n\nwhere T' represents the transpose of T, and Cj T;' is the outer product of Cj and Tj \u2022 \nIn addition, let Z be a three-dimensional auto-associative memory (or tensor of rank 3) \ncreated over three orthogonal representations of each target (i.e., Tj is a column vector, T;' \nis a row vector which is the transpose of Tj , and Tj \" is the vector in a third direction \northogonal to both, where i=I,2,3), as follows: \n\nz = I\u00b7 T- T-' T\u00b7\" \n\nI \n\nI \n\nI \n\nI \n\nLet a two-dimensional temporary matrix, X, be formed by taking the outer product of \ntarget vectors retrieved to the access cues, as follows: \n\nX = \n= \n\n(Cl M) (C2 M)' \n(0.9Tl + 0.lT2) (0.1T2 + 0.9Tj )' \n\n= \n\n0.09Tl T2' + 0.81Tl Tj' + 0.01T2T2' + 0.09T2Tj' \n\nUsing the matrix X to access the auto-associator Z, will produce a vector from which all \nthe cross-product terms have been flltered, as follows: \n\nX Z = \n= \n\n(0.09Tl T2' + 0.81 TlT3' + 0.01T2T2' + 0.09T2Tj' ) (Ij Tj T;' T;\") \n\n(0.09TlT2') (Ii Ti T;' T;',) + (0.81Tl Tl) (Ii Tj T;' Ti\") \n+ (0.01 T2T2') ( Ii Tj T;' Ti \") + (0.09T2Tj')