{"title": "Neural Networks: The Early Days", "book": "Advances in Neural Information Processing Systems", "page_first": 828, "page_last": 842, "abstract": null, "full_text": "828 \n\nCowan \n\nNeural networks: the early days \n\nJ.D. Cowan \n\nDepartment of Mathematics, Committee on \nNeurobiology, and Brain Research Institute, \n\nThe University of Chicago, 5734 S. Univ. Ave., \n\nChicago, Illinois 60637 \n\nABSTRACT \n\nA short account is given of various investigations of neural network \nproperties, beginning with the classic work of McCulloch & Pitts. \nEarly work on neurodynamics and statistical mechanics, analogies with \nmagnetic materials, fault tolerance via parallel distributed processing, \nmemory, learning, and pattern recognition, is described. \n\n1 INTRODUCTION \n\nIn this brief account of the early days in neural network research, it is not possible to be \ncomprehensive. This article then is a somewhat subjective survey of some, but not all, of \nthe developments in the theory of neural networks in the twent-five year period, from \n1943 to 1968, when many of the ideas and concepts were formulated, which define the \nfield of neural network research. This comprises work on connections with automata \ntheory and computability; neurodynamics, both deterministic and statistical; analogies \nwith magnetic materials and spin systems; reliability via parallel and parallel distributed \nprocessing; modifiable synapses and conditioning; associative memory; and supervised \nand unsupervised learning. \n\n2 McCULLOCH-PITTS NETWORKS \n\nThe modem era may be said to have begun with the work of McCulloch and Pitts (1943). \nThis is too well-known to need commenting on. Let me just make some historical re(cid:173)\nmarks. McCulloch, who was by training a psychiatrist and neuroanatomist, spent some \ntwenty years thinking about the representation of event in the nervous system. From 1941 \nto 1951 he worked in Chicago. Chicago at that time was one of the centers of neural of \n\n\fNeural Networks: The Early Days \n\n829 \n\nFigure1: Warren McCulloch circa 1962 \n\nnetwork research, mainly through the work of the Rashevsky group in the Committee on \nMathematical Biology at the University of Chicago. Rashevsky, Landahl, Rapaport and \nShim bel, among others, carried out many early investigations of the dynamics of neural \nnetworks, using a mixture of calculus and algebra. In 1942 McCulloch was introduced to \nWalter Pitts, then a 17 year old student of Rashevsky's. Pitts was a mathematical prodigy \nwho had joined the Committee sometime in 1941. There is an (apocryphal) story that \nPitts was led to the Rashevsky group after a chance meeting with the philosopher \nBertrand Russell, at that time a visitor to the University of Chicago. In any event Pitts \nwas already working on algebraic aspects of neural networks, and it did not take him long \nto see the point behind McCulloch's quest for the embodiment of mind. \nIn one of \nMcCulloch later essays (McCulloch 1961) he describes the history of his efforts thus: \n\nMy object, as a psychologist, was to invent a least psychic event, or \n\"psychon\", that would have the following properties: First, it was to be \nso simple an event that it either happened or else it did not happen. \nSecond, it was to happen only if its bound cause had happened-shades \nof Duns Scotus!-that is, it was to imply its temporal antecedent. \nThird it was to propose this to subsequent psychons. Fourth, these \nwere to be compounded to produce the equivalents of more \ncomplicated propositions concerning their antecedents .. .In 1921 it \ndawned on me that these events might be regarded as the all-or(cid:173)\nnothing impulses of neurons, combined by convergence upon the next \nneuron to yield complexes of propositional events. \n\nTheir subsequent 1943 paper was remarkable in many respects. It is best appreciated \nwithin the zeitgeist of the era when it was written. As Papert has documented in his \nintroduction to a collection of McCulloch's papers (McCulloch 1967), 1943 was a semi-\n\n\f830 \n\nCowan \n\nnal year for the development of the science of the mind. Craik's monograph The Nature \nof Explanation and the paper \"Behavior, Purpose and Teleology, by Rosenbleuth, \nWiener and Bigelow, were also published in 1943. As Papert noted, \"The common \nfeature [of these publications] is their recognition that the laws governing the \nembodiment of mind should be sought among the laws governing information rather than \nenergy or matter\". The paper by McCulloch and Pitts certainly lies within this \nframework. \n\nFigure 2: Walter Pitts circa 1952 \n\nMcCulloch-Pitts networks (hence-forth referred to as MP networks), are finite state \nautomata embodying the logic of propositions, with quantifiers, as McCulloch wished; \nand permit the framing of sharp hypotheses about the nature of brain mechanisms, in a \nform equivalent to computer programs. This was a remarkable achievement. It \nestablished once and for all, the validity of making formal models of brain mechanisms, \nif not their veridicality. It also established the possibility of a rigorous theory of mind, in \nthat neural networks with feedback loops can exhibit purposive behavior, or as \nMcCulloch and Pitts put it: \n\nboth the formal and the final aspects of that activity which we are \nwont to call mental are rigorously deducible from present \nneurophysiology ... [and] that in [imaginable networks] ... \"Mind\" no \nlonger \"goes more ghostly than a ghost\". \n\n2.1 FAULT TOLERANCE \n\n:MP networks were the first designed to perform specific logical tasks; and of course logic \ncan be mapped into arithmetic. Landahl, McCulloch and Pitts (1943), for example, \nnoted that the arithmetical operations +, 1-, and x can be obtained in MP networks via the \nlogical operations OR. NOT, and AND. Thus the arithmetical expression a-a.b = a.(l-b) \n\n\f", "award": [], "sourceid": 198, "authors": [{"given_name": "Jack", "family_name": "Cowan", "institution": null}]}