{"title": "Comparing the Performance of Connectionist and Statistical Classifiers on an Image Segmentation Problem", "book": "Advances in Neural Information Processing Systems", "page_first": 614, "page_last": 621, "abstract": null, "full_text": "614 \n\nGish and Blanz \n\nComparing the Performance of Connectionist \n\nand Statistical Classifiers on an Image \n\nSegmentation Problem \n\nSheri L. Gish w. E. Blanz \nIBM Almaden Research Center \n\n650 Harry Road \n\nSan Jose, CA 95120 \n\nABSTRACT \n\nIn this study, we test the suitability of a connection(cid:173)\n\nIn the development of an image segmentation system for real time \nimage processing applications, we apply the classical decision anal(cid:173)\nysis paradigm by viewing image segmentation as a pixel classifica.(cid:173)\ntion task. We use supervised training to derive a classifier for our \nsystem from a set of examples of a particular pixel classification \nproblem. \nist method against two statistical methods, Gaussian maximum \nlikelihood classifier and first, second, and third degree polynomial \nclassifiers, for the solution of a \"real world\" image segmentation \nproblem taken from combustion research. Classifiers are derived \nusing all three methods, and the performance of all of the classi(cid:173)\nfiers on the training data set as well as on 3 separate entire test \nimages is measured. \n\nIntroduction \n\n1 \nWe are applying the trainable machine paradigm in our development of an image \nsegmentation system to be used in real time image processing applications. We \nview image segmentation as a classical decision analysis task; each pixel in a scene \nis described by a set of measurements, and we use that set of measurements with \na classifier of our choice to determine the region or object within a scene to which \nthat pixel belongs. Performing image segmentation as a decision analysis task pro(cid:173)\nvides several advantages. We can exploit the inherent trainability found in decision \n\n\fComparing the Performance of Connectionist and Statistical Classifiers \n\n615 \n\nanalysis systems [1 J and use supervised training to derive a classifier from a set of \nexamples of a particular pixel classification problem. Classifiers derived using the \ntrainable machine paradigm will exhibit the property of generalization, and thus can \nbe applied to data representing a set of problems similar to the example problem. In \nour pixel classification scheme, the classifier can be derived solely from the qU8J1ti(cid:173)\ntative characteristics of the problem data. Our approach eliminates the dependency \non qualitative characteristics of the problem data which often is characteristic of \nexplicitly derived classification algorithms [2,3J. \n\nClassical decision 8J1alysis methods employ statistical techniques. We have com(cid:173)\npared a connectionist system to a set of alternative statistical methods on classifi(cid:173)\ncation problems in which the classifier is derived using supervised training, 8J1d have \nfound that the connectionist alternative is comparable, and in some cases prefer(cid:173)\nable, to the statistical alternatives in terms of performance on problems of varying \ncomplexity [4J. That comparison study also 8J.lruyzed the alternative methods in \nterms of cost of implementation of the solution architecture in digital LSI. hl terms \nof our cost analysis, the connectionist architectures were much simpler to implement \nthan the statistical architectures for the more complex classification problems; this \nproperty of the connectionist methods makes them very attractive implementation \nchoices for systems requiring hardware implementations for difficult applications. \n\nIn this study, we evaluate the perform8J.lce of a connectionist method and several \nstatisticrumethods as the classifier component of our real time image segmentation \nsystem. The classification problem we use is a \"real world\" pixel classification task \nusing images of the size (200 pixels by 200 pixels) and variable data quality typical \nof the problems a production system would be used to solve. We thus test the \nsuitability of the connectionist method for incorporation in a system with the per(cid:173)\nformance requirements of our system, as well as the feasibility of our exploiting the \nadv8J.ttages the simple connectionist architectures provide for systems implemented \nin hardware. \n\n2 Methods \n2.1 The Image Segmentation System \n\nThe image segmentation system we use is described in [5J, and summarized in \nFigure 1. The system is designed to perform low level image segmentation in real \ntime; for production, the feature extraction and classifier system components are \nimplemented in hardware. The classifier par8J.neters are derived during the Training \nPhase. A user at a workstation outlines the regions or objects of interest in a \ntraining image. The system performs low level feature extraction on the training \nimage, and the results of the feature extraction plus the input from the user are \ncombined automatically by the system to form a training data set. The system then \napplies a supervised training method making use of the training data set in order \nto derive the coefficients for the classifier which can perform the pixel classification \ntask. The feature extraction process is capable of computing 14 classes of features \nfor each pixel; up to 10 features with the highest discriminatory power are used to \n\n\f616 \n\nGish and Blanz \n\ndescribe all of the pixels in the image. TIns selection of features is based only on an \nanalysis of the results of the feattue extraction process and is independent of the \nsupervised learning paradigm being used to derive the classifier [6]. The identical \nfeature extraction process is applied in both the Training and Running Phases for \na particular image segmentation problem. \n\nTraining Images \n\nTest Image \n\nCoefficients \n\nfor \n\nClassifier \n\nTRAINING \n\nPHASE \n\nSegmented \n\nImage \n\nRUNNING \n\nPHASE \n\nFigure 1: Diagram of the real time image segmentation system. \n\n2.2 The Image Segmentation Problem \n\nThe image segmentation problem used in this study is from combustion research and \nis described in [3]. The images are from a series of images of a combustion chamber \ntaken by a high speed camera during the inflammation process of a gas/air mix(cid:173)\nhue. The segmentation task is to determine the area of inflamed gas in the image; \ntherefore, the pixels in the image are classified into 3 different classes: cylinder, \nuninflamed gas, and flamed gas (See Figure 2). Exact determination of the area \nof flamed gas is not possible using pixel classification alone, but the greater the \nsuccess of the pixel classification step, the greater the likelihood that a real time \nimage segmentation system could be used successfully on this problem. \n\n2.3 The Classifiers \n\nThe set of classifiers used in tIns study is composed of a connectionist classifier \nbased on the Parallel Distributed Processing (PDP) model described in [7] and two \nstatistical methods: a Gaussian maximum likelihood classifier (a Bayes classifier), \nand a polynomial classifier based on first, second, and third degree polynomials. \nTlus set of classifiers was used in a general study comparing the performance of \n\n\fComparing the Performance of Connectionist and Statistical Classifiers \n\n617 \n\nFigure 2: The imnge segmentntion problem is to classify each imllge pixel into 1 \nof 3 regions. \n\nthe alternatives on a set of classification problems; all of the classifiers as well as \nadaptation procedures are described in detnil in that study [4]. Implementation \nand adaptation of nll classifiers in this study was performed as software simulation. \nThe connectionist classifier was implemented in eMU Common Lisp rmming on an \nIBM RT workstation. \n\nThe connectionist classifier nrchitecture is a multi-Inyer feedforwnrd network with \none hidden layer. The network is fully connected, but there nre only connections \nbetween ndjacent layers. The number of units in the input and output layers are \ndetermined by the number of features in the fenture vector describing ench pixel \nand a binary encoding scheme for the class to which the pixel belongs, respectively. \nThe number of units in the hidden layer is an architectural \"free parnmeter.\" The \nnetwork used in this study has 10 units in the input layer, 12 units in the hidden \nlayer, and 3 units in the outPllt layer. \n\nNetwork activation is achieved by using the continuous, nonlinear logistic function \ndefined in [8]. The connectionist adaptation procedure is the applicntion of the \nbackpropagation learning rule also defined in [8]. For this problem, the learning rnte \nTJ = 0.01 and the momentum a = 0.9; both terms were held conshmt throughout \nadaptntion. The presentation of all of the patterns ill the training data set is termed \na trial; network weights nnd unit binses were updated after the presentation of each \npattern during a trial. \n\nThe training data set for this problem was generated automatically by the image \nsegmentation system. This training data set consists of approximately 4,000 ten \nelement (feature) vectors (each vector describes one pixel); each vector is labeled as \nbelonging to one of the 3 regions of interest in the imnge. The training data set was \nconstructed from one entire training image, and is composed of vectors stntistically \nrepresentative of the pixels in each of the 3 regions of interest in that image. \n\n\f618 \n\nGish and Dlanz \n\nAll of the classifiers tested in this study were adapted from the same training data \nset. The connectionist classifier was defined to be converged for tlus problem before \nit was tested. Network convergence is determined from the results of two separate \ntests. III the first test, the difference between the network output and the target \noutput averaged over the entire training data set has to reach a minimum. In the \nsecond test, the performance of the network in classifying the training data set is \nmeasured, and the number of misclassifications made by the network has to reach \na minimum. Actual network performance in classifying a pattern is measured after \npost-processing of the output vector. The real outputs of each unit in the output \nlayer are assigned the values of 0 or 1 by application of a 0.5 decision threshold. \nIn our binary encoding scheme, the output vector should have only one element \nwith the value 1; that element corresponds to one of the 3 classes. H the network \nproduces an output vector with either more than one element with the value 1 or all \nelements with the value 0, the pattern generating that output is considered rejected. \nFor the test problem in this study, all of the classifiers were set to reject patterns \nin the test data samples. All of the statistical classifiers had a rejection threshold \nset to 0.03. \n\n3 Results \n\nThe performance of each of the classifiers (connectionist, Gaussian maximum like(cid:173)\nlihood, and linear, quadratic, and cubic polynomial) was measured on the training \ndata set and test data representing 3 entire images taken from the series of com(cid:173)\nbustion chamber images. One of those images, labeled Inlage 1, is the image from \nwhich the training data set was constructed. The performance of all of the classifiers \nis summarized in Table 1. \n\nAlthollgh all of the classifiers were able to classify the training data set with com(cid:173)\nparably few misclassifications, the Gaussian maximum likelihood classifier and the \nquadratic polynomial classifier were unable to perform on any of the 3 entire test \nimages. The connectionist classifier was the only alternative tested in this study to \ndeliver acceptable performance on all 3 test images; the connectionist classifier had \nlower error rates on the test images than it delivered on the training data sample. \nBoth the linear polynomial and cubic polynomial classifiers performed acceptably \non the test Image 2, but then both exhibited high error rates on the other two \ntest images. For this image segmentation problem, only the connectionist method \ngeneralized from the training data set to a solution with acceptable performance. \n\nIn Figure 3, the results from pixel classification performed by the connectionist and \npolynonual classifiers on all 3 test images are portrayed as segmented images. The \nactual test images are included at the left of the figure. \n\n4 Conclusions \n\nOur results demonstrate the feasibility of the application of a connectionist decision \nanalysis method to the solution of a ureal world\" image segmentation problem. The \n\n\fComparing the Performance of Connectionist and Statistical Classifiers \n\n619 \n\n-~auss;an --] \nClassifier \nError I Reject \n12.84% ---0.12%-\n\n94.27% \n\n0.00% \n\n----~~----~~----~ \n\n69.09% \n\n0.01% \n\n88.35% \n\n0.00% \n\n~ata Sel \n\nII \n\nClassifier \n\nConne;;l;on;sl \nError fl I Rejectb \n1O.40%-~-.64% \n\n,----T \n\n. . \n\n8.84% \n\nImage 1 C \n\nraInIng \nData \n\n1 \n2 \n3 \n1 \n2 \n3 \nr-~------~~--~-r~~~~~r----\n1 \n2 \n3 \n\nImage 2 \n\n1.53% \n\n1.72% \n\n5.82% \n\nImage 3 \n\n6.31 % \n\n- -::-1.-=-6-=3 %=o- tf---- 1=----\n\n- Polynom;al \nClassifier \n\n--\nDegree I Error I Reject \n1.62% \n1.41% \n1.05% \n4.63% \n3.66% \n0.28% \n2.00% \n0.58% \n0.26% \n5.43 % \n1.41% \n0.28% \n\n'1l.25% \n9.61% \n8.13% \n41.70% \n57.55% \n25 .86% \n12.01% \n68.01 % \n4.68% \n19.68 % \n45.89% \n25.75% \n\n2 \n3 \n\n, ______ _ ~ _ __ _ __ L_ _\n\n_ ____ ~ ______ L_ _____ _ ~ __ ____ ~ ___ ___ ~ ______ __ \n\nflPercent misclauificatioDi for all patterns. \n\nbpercent of all patterns rejected. \n\nClmage from which training data let was taken. \n\nTable 1: A sununary of the performance of the c16Ssifiers. \n\ninclusion of a connectionist classifier in our supervised segmentation system will al(cid:173)\nlow us to meet our performance requirements under real world problem constraints. \n\nAlthough the application of connectionism to the solution of real time machine \nvision problems represents a new processing method, our solution strategy h6S re(cid:173)\nmained consistent with the decision analysis paradigm. Our connectionist cl6Ssifiers \nare derived solely from the quantitative characteristics of the problem data; our con(cid:173)\nnectionist architecture thus remains simple and need not be re-designed according \nto qualitative characteristics of each specific problem to which it will be applied. \nOur connectionist architecture is independent of the image size; we have applied the \nidentical architecture successfully to images which range in size from 200 pixels by \n200 pixels to 512 pixels by 512 pixels [9). In most research to date in which neural \nnetworks are applied to machine vision, entire images explicitly are mapped to net(cid:173)\nworks by making each pixel in an image correspond to a different unit in a network \nlayer (see [10,11) for examples). This \"pixel map\" representation makes scaling up \nto larger image sizes from the idealized \"toy\" research images a significant problem. \n\nMost statistical pattern classification methods require that problem data satisfy \ntIle assumptions of statistical models; unfortunately, real world problem data are \ncomplex and of variable quality and thus rarely can be used to guide the choice of an \nappropriate method for the solution of a particular problem a priori. For the image \nsegmentation problem reported in this study, our cI6Ssifier performance results show \nthat the problem data actually did not satisfy the assumptions behind the statistical \nmodels underlying the Gaussian maximum likelihood classifier or the polynomial \n\n\f620 \n\nGish and Blanz \n\nFigure 3: The grey levels assigned to each region nre: Black -\nGrey -\nfigure. \n\ncylinder, Light \nfhnned gas. Original images nre at the left of the \n\nuninflamed gas, Grey -\n\nclassifiers. It appenrs that the Gaussian model least fits our problem data, the \npolynomial classifiers provide a slightly better fi t, and the connect.ionist method \nprovides the fit required for the solution of the problem. It is also notable that all \nthe alternative m.ethods in this study could be aflapted to perform acceptably on \nthe training data set, but extensive testing on several different entire images was \nrequired in order to demonstrate the true performance of the n1t.ernntive lllethods \non the actual problem., rather than just on the trnining data set. \n\nThese results show that a connectionist method is a viable choice for n system. such \nas ours which requires a simple nrchitecture readily implemented in hardware, the \nflexibility to handle cOl1lpi('x problems described by large amounts of data, and the \nrobustness to not require problem data to meet, many model assnmptions 11 priori. \n\n\fComparing the Performance of Connectionist and Statistical Classifiers \n\n621 \n\nReferences \n[lJ R. O. Duda a.nd P. E. H6I't. Pattern Cla$$ification and Scene Analy,i$. Wiley, \n\nNew York, 1973. \n\n[2J W. E. Blanz, J. L. C. Sanz, and D. Petkovic. Control-free low-level image \nsegmentation: Theory, architecture,a.nd experimentation. In J. L. C. Sanz, \neditor, Advance$ of Machine Vi$ion, Application$ and Architect-ure\" Springer(cid:173)\nVerlag, 1988. \n\n[3] B. Straub and W. E. Blanz. Combined decision theoretic and syntactic ap(cid:173)\n\nproach to image segmentation. Machine Vi,ion and Application$, 2(1 ):17-30, \n1989. \n\n[4J Sheri L. Gish and W. E. Blallz. Comparing a Connectioni$t Trainable Clauifier \nwith Clauical Stati$tical Deci,ion AnalY$i$ Method$. Research Report RJ 6891 \n(65717), IBM, Jtme 1989. \n\n[5] W. E. Bla.nz, B. Slmng, C. Cox, W. Greiner, B. Dom, a.nd D. Petkovic. De$ign \nand implementation of a low level image ,egmentation architecture - LISA. \nResearch Report RJ 7194 (67673), IBM, December 1989. \n\n[6] W. E. BI611z. Non-p6I'ametric feature selection for multiple class processes. In \n\nProc. 9th Int. Con/. Pattern Recognition, Rome, Italy, Nov. 14-17 1988. \n\n[7J David E. Rumelhart, J61ues L. McClelland, et a1. Parallel Di$tributed Proceu(cid:173)\n\ning. MIT Press, C61ubridge, Massachusetts, 1986. \n\n[8] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Willia.ms. Le6I'nillg in(cid:173)\nternal representations by error propagation. In David E. Rumelllart, James L. \nMcClell611d, et aI., editors, Parallel Di,tributed Proce66ing, chapter 8, MIT \nPress, Cambridge, Massachusetts, 1986. \n\n[9] W. E. Bla.nz 6l1d Sheri L. Gish. A Connectioni,t Clauifier Architecture Applied \nTo Image Segmentation. Rese6I'ch Report RJ 7193 (67672), IBM, December \n1989. \n\n[10} K. Fukushima, S. Miyake, and T. Ito. Neocognitron: a neura.lnetwork model \nfor a mechanism of visual pattern recognition. IEEE Tran$actio1t$ on Sy,tem\" \nMan, and Cybernetic$, SMC-13(5):826-834, 1983. \n\n[U} Y. Hirai. A model of humau associative processor. \n\nIEEE Tran$action$ on \n\nSy,tem$, Man, and Cybernetic$, SMC-13(5):851-857, 1983. \n\n\f", "award": [], "sourceid": 236, "authors": [{"given_name": "Sheri", "family_name": "Gish", "institution": null}, {"given_name": "W.", "family_name": "Blanz", "institution": null}]}