{"title": "Software for ANN training on a Ring Array Processor", "book": "Advances in Neural Information Processing Systems", "page_first": 781, "page_last": 788, "abstract": null, "full_text": "Software for ANN training on a Ring Array Processor \n\nPhil Kohn, Jeff Bilmes, Nelson Morgan, James Beck \nInternational Computer Science Institute, \n1947 Center St., Berkeley CA 94704, USA \n\nAbstract \n\nExperimental research on Artificial Neural Network (ANN) algorithms requires \neither writing variations on the same program or making one monolithic program \nwith many parameters and options. By using an object-oriented library, the size \nof these experimental programs is reduced while making them easier to read, \nwrite and modify. An efficient and flexible realization of this idea is Connection(cid:173)\nist Layered Object-oriented Network Simulator (CLONES). CLONES runs on \nUNIX1 workstations and on the 100-1000 MFLOP Ring Array Processor (RAP) \nthat we built with ANN algorithms in mind. In this report we describe CLONES \nand show how it is implemented on the RAP. \n\n1 Overview \n\nAs we continue to experiment with Artificial Neural Networks (ANNs) to generate phoneme \nprobabilities for speech recognition (Bourlard & Morgan, 1991), two things have become \nincreasingly clear: \n\n1. Because of the diversity and continuing evolution of ANN algorithms, the program(cid:173)\n\nming environment must be both powerful and flexible. \n\n2. These algorithms are very computationally intensive when applied to large databases \n\nof training patterns. \n\nIdeally we would like to implement and test ideas at about the same rate that we come up \nwith them. We have approached this goal both by developing application specific parallel \n\nlUNIX is a trademark of AT&T \n\n781 \n\n\f782 \n\nKahn, Bilrnes, Morgan, and Beck \n\nSystem \n\nPerfonnance \n\nLanguages Supported \n\nAssem \n\nC \n\nc++ Sather pSather \n\nSparcStation 2 \n\n~ \n~ RAP9jstam \n~~mID \n\nDesktop RAP + \nSun 4/330 Host \n\nNctwmcd RAP \n(1-10 Boards) \n\n2MFLOP \n\n-/ -/ -/ -/ -/ \n\nlOOMFLOP \n\n1 GFLOP \n\n-I -I -I -I \n\nSPERTBowd \n\n~~~L7 \n\nSparcSlation + \nSPERT Board \n\nlOOP \n\nif if if ~ \n\nS \no \nu \nr \nc \nc \n\nC \no \nm \n\np \u2022 t \n\ni \nb \nI \nc \n\nCNS-l System \n\n200 GOP ~ ~ ~ ~ ~ \n\n- / Completed ~ In Design <,...-__ L_in_kC_r_c_om-::pa_lI_o b_IC ___ > \n\nFigure 1: Hardware and software configurations \n\nhardware, the Ring Array Processor (RAP) (Morgan et al., 1990; Beck, 1990; Morgan et al., \n1992), and by building an object-oriented software environment, the Connectionist Layered \nObject-oriented Network Simulator (CLONES) (Kohn, 1991). By using an object-oriented \nlibrary, the size of experimental ANN programs can be greatly reduced while making them \neasier to read, write and modify. CLONES is written in C++ and utilizes libraries previously \nwritten in C and assem bIer. \n\nOur ANN research currently encompasses two hardware platforms and several languages. \nshown in Figure 1. Two new hardware platforms, the SPERT board (Asanovic et al., 1991) \nand the CNS-l system are in design (unfilled check marks), and will support source code \ncompatibility with the existing machines. The SPERT design is a custom VLSI parallel \nprocessor installed on an SBUS card plugged into a SPARC workstation. Using variable \nprecision fixed point arithmetic, a single SPERT board will have performance comparable \nto a 10 board RAP system with 40 processors. The CNS-l system is based on multiple \nVLSI parallel processors interconnected by high speed communication rings. \n\nBecause the investment in software is generally large, we insiston source level compatibility \nacross hardware platforms at the level of the system libraries. These libraries include matrix \nand vector classes that free the user from concern about the hardware configuration. It is \nalso considered important to allow routines in different languages to be linked together. \nThis includes support for Sather, an object-oriented language that has been developed at \nICSI for workstations. The parallel version of Sather, called pSather, will be supported on \n\n\fSottware for ANN training on a Ring Array Processor \n\n783 \n\ntheCNS-l. \n\nCLONES is seen as the ANN researcher's interface to this multiplatform, multi language \nenvironment. Although CLONES is an application written specifically for ANN algorithms, \nit's object-orientation gives it the ability to easily include previously developed libraries. \nCLONES currently runs on UNIX workstations and the RAP; this paper focuses on the \nRAP implementation. \n\n2 RAP hardware \n\nThe RAP consists of cards that are added to a UNIX host machine (currently a VME based \nSun SPARe). A RAP card has four 32 MFlop Digital Signal Processor (DSP) chips (TI \nTMS32OC30), each with its own local 256KB or 1MB of fast static RAM and 16MB of \nDRAM. \n\nInstead of sharing memory, the processors communicate on a high speed ring that shifts \ndata in a single machine cycle. For each board, the peak transfer rate between 4 nodes \nis 64 million words/sec (256 Mbytes/second). This is a good balance to the 64 million \nmultiply-accumulates per second (128 MFLOPS) peak performance of the computational \nelements. \n\nUp to 16 of these boards can be interconnected and used as one Single Program operating \non Multiple Data stream (SPMD) machine. In this style of parallel computation, all the \nprocessors run the same program and are doing the same operations to different pieces of the \nsame matrix or vector 2. The RAP can run other styles of parallel computation, including \npipelines where each processor is doing a different operation on different data streams. \nHowever, for fully connected back-propagation networks, SPMD parallelism works well \nand is also much easier to program since there is only one flow of control to worry about. \n\nA reasonable design for networks in which all processors need all unit outputs is a single \nbroadcast bus. However, this design is not appropriate for other related algorithms such \nas the backward phase of the back-propagation learning algorithm. By using a ring, back(cid:173)\npropagation can be efficiently parallelized without the need to have the complete weight \nmatrix on all processors. The number of ring operations required for each complete matrix \nupdate cycle is of the same order as the number of units, not the square of the number of \nunits. It should also be noted that we are using a stochastic or on-line learning algorithm. \nThe training examples are not di viding among the processors then the weights batch updated \nafter a complete pass. All weights are updated for each training example. This procedure \ngreatly decreases the training time for large redundant training sets since more steps are \nbeing taken in the weight-space per training example. \n\nWe have empirically derived formulae that predict the performance improvement on back(cid:173)\npropagation training as a function of the number of boards. Theoretical peak performance is \n128 MFlops/board, with sustained performance of 30-90% for back-propagation problems \nof interest to us. Systems with up to 40 nodes have been tested, for which throughputs \n\n1'he hardware does not automatically keep the processors in lock step; for example, they may \nbecome out of sync because of branches conditioned on the processor's node number or on the \ndata. However, when the processors must communicate with each other through the ring, hardware \nsynchronization automatically occurs. A node that attempts to read before data is ready. or to write \nwhen there is already data waiting. will stop executing until the data can be moved. \n\n\f784 \n\nKahn, Bilrnes, Morgan, and Beck \n\nof up to 574 Million Connections Per Second (MCPS) have been measured, as well as \nlearning rates of up to 106 Million Connection Updates Per Second (MCUPS) for training. \nPractical considerations such as workstation address space and clock skew restrict current \nimplementations to 64 nodes, but in principle the architecture scales to about 16,000 nodes \nfor back-propagation. \n\nWe now have considerable experience with the RAP as a day-to-day computational tool for \nour research. With the aid of the RAP hardware and software, we have done network training \nstudies that would have over a century on a UNIX workstation such as the SPARCstation-2. \nWe have also used the RAP to simulate variable precision arithmetic to guide us in the \ndesign of higher performance hardware such as SPERT. \n\nThe RAP hardware remains very flexible because of the extensive use of programmable \nlogic arrays. These parts are automatically downloaded when the host machine boots up. \nBy changing the download files, the functionality of the communications ring and the host \ninterface can be modified or extended without any physical changes to the board. \n\n3 RAP software \n\nThe RAP DSP software is built in three levels (Kohn & Bilmes, 1990; Bilmes & Kohn, \n(990). At the lowest level are hand coded assembler routines for matrix, vector and ring \noperations. Many standard matrix and vector operations are currently supported as well \nas some operations specialized for efficient back-propagation. These matrix and vector \nroutines do not use the communications ring or split up data among processing nodes. \nThere is also a UNIX compatible library including most standard C functions for file, math \nand string operations. All UNIX kernel calls (such as file input or output) cause requests \nto be made to the host SPARC over the VMEbus. A RAP dremon process running under \nUNIX has all of the RAP memory mapped into its virtual address space. It responds to the \nRAP system call interrupts (from the RAP device driver) and can access RAP memory with \na direct memory copy function or assignment statement. \n\nAn intermediate level consists of matrix and vector object classes coded in C++. A \nprogrammer writing at this level or above can program the RAP as if it were a conventional \nserial machine. These object classes divide the data and processing among the available \nprocessing nodes, using the communication ring to redistribute data as needed. For example, \nto multiply a matrix by a vector, each processor would have its own subset of the matrix \nrows that must be multiplied. This is equivalent to partitioning the output vector elements \namong the processors. If the complete output vector is needed by all processors, a ring \nbroadcast routine is called to redistribute the part of the output vector from each processor \nto all the other processors. \n\nThe top level of RAP software is the CLONES environment. CLONES is an object-oriented \nlibrary for constructing, training and utilizing connectionist networks. It is designed to \nrun efficiently on data parallel computers as well as uniprocessor workstations. While \nefficiency and portability to parallel computers are the primary goals, there are several \nsecondary design goals: \n\n1. minimize the learning curve for using CLONES; \n2. minimize the additional code required for new experiments; \n3. maximize the variety of artificial neural network algorithms supported; \n\n\fSoftware for ANN training on a Ring Array Processor \n\n785 \n\n4. allow heterogeneous algorithms and training procedures to be interconnected and \n\ntrained together; \n\n5. allow the trained network to be easily embedded into other programs. \n\nThe size of experimental ANN programs is greatly reduced by using an object-oriented \nlibrary; at the same time these programs are easier to read, write and evolve. \n\nResearchers often generate either a proliferation of versions of the same basic program, \nor one giant program with a large number of options and many potential interactions and \nside-effects. Some simulator programs include (or worse, evolve) their own language \nfor describing networks. We feel that a modem object-oriented language (such as C++) \nhas all the functionality needed to build and train ANNs. By using an object-oriented \ndesign, we attempt to make the most frequently changed parts of the program very small \nand well localized. The parts that rarely change are in a centralized library. One of the \nmany advantages of an object-oriented library for experimental work is that any part can \nbe specialized by making a new class of object that inherits the desired operations from a \nlibrary class. \n\n4 CLONES overview \n\nTo make CLONES easier to learn, we restrict ourselves to a subset of the many features \nof C++. Excluded features include multiple inheritance, operator overloading (however, \nfunction overloading is used) and references. Since the multiple inheritance feature of C++ \nis not used, CLONES classes can be viewed as a collection of simple inheritance trees. \nThis means that all classes of objects in CLONES either have no parent class (top of a class \ntree) or inherit the functions and variables of a single parent class. \n\nCLONES consists of a library of C++ classes that represent networks (Net), their com(cid:173)\nponents (Net-part) and training procedures. There are also utility classes used during \ntraining such as: databases of training data (Database), tables of parameters and arguments \n(Param), and perfonnance statistics (Stats). Database and Param do not inherit from any \nother class. Their class trees are independent of the rest of CLONES and each other. The \nStats class inherits from Net-behavior. \n\nThe top level of the CLONES class tree is a class called NeLbehavior. It defines function \ninterfaces for many general functions including file save or restore and debugging. It also \ncontains behavior functions that are called during different phases of running or training a \nnetwork. For example, there are functions that are called before or after a complete training \nrun (pre_training, posLtraining), before or after a pass over the database (pre_epoch, \npost-epoch) and before or after a forward or backward run of the network (pre_forw-pass, \npost1orw_pass, pre_back_pass, posLback_pass). The Net, NeLpart and Stats classes \ninherit from this class. \n\nAll network components used to construct ANNs are derived from the two classes Layer \nand Connect. Both of these inherit from class NeLpart. A CLONES network can be \nviewed as a graph where the nodes are Layer objects and the arcs are Connect objects. \nEach Connect connects a single input Layer with a single output Layer. A Layer holds \nthe data for a set of units (such as an activation vector), while a Connect transforms the \ndata as it passes between Layers. Data flows along Connects between the pair of Layers \nby calling forw_propagate (input to output) or back_propagate (output to input) behavior \n\n\f786 \n\nKahn, Bilrnes, Morgan, and Beck \n\nfunctions in the Connect object. \nCLONES does not have objects that represent single units (or artificial neurons). Insteadt \nLayer objects are used to represent a set of units. Because arrays of units are passed \ndown to the lowest level routinest most of the computation time is focused into a few small \nassembly coded loops that easily fit into the processor instruction cache. Time spent in all \nof the levels of control code that call these loops becomes less significant as the size of the \nLayer is increased. \n\nThe Layer class does not place any restrictions on the representation of its internal infor(cid:173)\nmation. For examplet the representation for activations may be a floating point number \nfor each unit (AnalogJayer)t or it may be a set of unit indicest indicating which units \nare active (BinaryJayer). AnalogJayer and BinaryJayer are built into the CLONES \nlibrary as subclasses of the class Layer. The AnalogJayer class specifies the repre(cid:173)\nsentation of activationst but it still leaves open the procedures that use and update the \nactivation array. BP ...analogJayer is a subclass of AnalogJayer that specify these pro(cid:173)\ncedures for the back-propagation algorithm. Subclasses of AnalogJayer may also add \nnew data structures to hold extra internal state such as the error vector in the case of \nBP ...analogJayer. The BP -AnalogJaycr class has subclasses for various transfer func(cid:173)\ntions such as BP ...sigmoidJayer and BP Jinear Jayer. \n\nLayer classes also have behavior functions that are called in the course of running the \nnetwork. For examplet one of these functions (pre_forw-propagate) initializes the Layer \nfor a forward passt perhaps by clearing its activation vector. After all of the connections \ncoming into it are runt another Layer behavior function (postJorw_propagate) is called \nthat computes the activation vector from the partial results left by these connections. For \nexamplet this function may apply a transfer function such as the sigmoid to the accumulated \nsum of all the input activations. \n\nThese behavior functions can be changed by making a subclass. BP ...analogJayer leaves \nopen the activation transfer function (or squashing function) and its derivative. Subclasses \ndefine new transfer functions to be applied to the activations. A new class of back(cid:173)\npropagation layer with a customized transfer function (instead of the default sigmoid) can \nbe created with the following C++ code: \n\nMy_new_BP_layer_class(int number_of_units) \n\n: BP_analog_layer(number_of_units)i \n\nII constructor \n\nvoid transfer (Fvec *activation) { \n\n1* apply forward transfer function to my activation vector *1 \n\nvoid d_transfer(Fvec *activation, Fvec *err) \n\n1* apply backward error transfer to err (given activation) *1 \n\n} i \n\nA Connect class includes two behavior functions: one that transforms activations from the \nincoming Layer into partial results in the outgoing Layer (forw-propagate) and one that \ntakes outgoing errors and generates partial results in the incoming Layer (back-propagate). \n\n\fSoftware for ANN training on a Ring Array Processor \n\n787 \n\nThe structure of a partial result is part of the Layer class. The subclasses of Connect include: \nBus_connect (one to one), Full_connect (all to all) and Sparse_connect (some to some). \n\nEach subclass of Connect may contain a set of internal parameters such as the weight \nmatrix in a BP JulLconnect. Subclasses of Connect also specify which pairs of Layer \nsubclasses can be connected. When a pair of Layer objects are connected, type checking \nby the C++ compiler insures that the input and output Layer subclasses are supported by \nthe Connect object. \n\nIn order to do its job efficiently, a Connect must know something about the internal \nrepresentation of the layers that are connected. By using C++ overloading, the Connect \nfunction selected depends not only on the class of Connect, but also on the classes of \nthe two layers that are connected. Not all Connect classes are defined for all pairs of \nLayer classes. However, Connects that convert between Layer classes can be utilized to \ncompensate for missing functions. \n\nCLONES allows the user to view layers and connections much like tinker-toy wheels and \nrods. ANNs are built up by creating Layer objects and passing them to the create functions \nof the desired Connect classes. Changing the interconnection pattern does not require any \nchanges to the Layer classes or objects and vice-versa. \n\nAt the highest level, a Net object delineates a subset of a network and controls its training. \nOperations can be performed on these subsets by calling functions on their Net objects. The \nLayers of a Net are specified by calling one of new_inputJayer, new_hidden.Jayer, or \nnew_outputJayer on the Net object for each Layer. Given the Layers, the Connects that \nbelong to the Net are deduced by the Net-order objects (see below). Layer and Connect \nobjects can belong to any number of Nets. \n\nThe Net labels all of its Layers as one of input, output or hidden. These labels are \nused by the NeLorder objects to determine the order in which the behavior functions of \nthe NeLparts are called. For example, a Net object contains NeLorder objects called \nforward_pass_order and backward_pass_order that control the execution sequence for a \nforward or backward pass. The Net object also has functions that call a function by the \nsame name on all of its component parts (for example set.Jearning-.rate). \n\nWhen a Net-order object is built it scans the connectivity of the Net. The rules that relate \ntopology to order of execution are centralized and encapsulated in subclasses of NeLorder. \nChanges to the structure of the Net are localized to just the code that creates the Layers \nand Connects; one does not need to update separate code that contains explict knowledge \nabout the order of evaluation for running a forward or backward pass. \nThe training procedure is divided into a series of steps, each of which is a call to a function \nin the Net object. At the top level, calling run_training on a Net performs a complete \ntraining run. In addition to calling pre_training, posLtraining behavior functions, it calls \nrun_epoch in a loop until the the nextJearning-.rate function returns zero. The run_epoch \nfunction calls run_forward and run_backward. \n\nAt a lower level there are functions that interface the database(s) of the Net object to the \nLayers of the Net. For example, seLinput sets the activations of the input Layers for a \ngiven pattern number of the database. Another of these sets the error vector of the output \nlayer (seLerror). Some of these functions, such as is_correct evaluate the performance of \nthe Net on the current pattern. \n\n\f788 \n\nKahn, Bilmes, Morgan, and Beck \n\nIn addition to database related functions, the Net object also contains useful global variables \nfor all of its components. A pointer to the Net object is always passed to all behavior \nfunctions of its Layers and Connects when they are called. One of these variables is a \nParam object that contains a table of parameter names, each with a list of values. These \nparameters usually come from the command line and/or parameter files. Other variables \ninclude: the current pattern, the correct target output, the epoch number, etc. \n\n5 Conclusions \n\nCLONES is a useful tool for training ANNs especially when working with large training \ndatabases and networks. It runs efficiently on a variety of parallel hardware as well as on \nUNIX workstations. \n\nAcknowledgements \n\nSpecial thanks to Steve Renals for daring to be the first CLONES user and making significant \ncontributions to the design and implementation. Others who provided valuable input to \nthis work were: Krste Asanovi~, Steve Omohundro, Jerry Feldman, Heinz Schmidt and \nChuck Wooters. Support from the International Computer Science Institute is gratefully \nacknowledged. \n\nReferences \n\nAsanovi~, K., Beck, J., Kingsbury, B., Kohn, P., Morgan, N., & Wawrzynek, J. (1991). \nSPERT: A VLIW ISIMD Microprocessor for Artificial Neural Network Computations. \nTech. rep. TR-91-072, International Computer Science Institute. \n\nBeck, J. (1990). The Ring Array Processor (RAP): Hardware. Tech. rep. TR-90-048, \n\nInternational Computer Science Institute. \n\nBilmes, J. & Kohn, P. (1990). The Ring Array Processor (RAP): Software Architecture. \n\nTech. rep. TR-90-050, International Computer Science Institute. \n\nBourlard, H. & Morgan, N. (1991). Connectionist approaches to the use of Markov models \nfor continuous speech recognition. In Touretzky, D. S. (Ed.), Advances in Neural \nInformation Processing Systems, Vol. 3. Morgan Kaufmann, San Mateo CA. \n\nKohn, P. & Bilmes, J. (1990). The Ring Array Processor (RAP): Software Users Manual \n\nVersion 1.0. Tech. rep. TR-90-049, International Computer Science Institute. \n\nKohn, P. (1991). CLONES: Connectionist Layered Object-oriented NEtwork Simulator. \n\nTech. rep. TR-91-073, International Computer Science Institute. \n\nMorgan, N., Beck, J., Kohn, P., Bilmes, J., Allman, E., & Beer, J. (1990). The RAP: a ring \narray processor for layered network calculations. In Proceedings IEEE International \nConference on Application Specific Array Processors, pp. 296-308 Princeton NI. \n\nMorgan, N., Beck, J., Kohn, P., & Bilmes, J. (1992). Neurocomputing on the RAP. In \nPrzytula, K. W. & Prasanna, V. K. (Eds.), Digital Parallellmplemencations of Neural \nNetworks. Prentice-Hall, Englewood Cliffs NJ. \n\n\f", "award": [], "sourceid": 584, "authors": [{"given_name": "Phil", "family_name": "Kohn", "institution": null}, {"given_name": "Jeff", "family_name": "Bilmes", "institution": null}, {"given_name": "Nelson", "family_name": "Morgan", "institution": null}, {"given_name": "James", "family_name": "Beck", "institution": null}]}