{"title": "Address Block Location with a Neural Net System", "book": "Advances in Neural Information Processing Systems", "page_first": 785, "page_last": 792, "abstract": null, "full_text": "Address Block Location with a Neural Net System \n\nHans Peter Graf \n\nEric Cosatto \n\nAT&T Bell Laboratories \nCrawfords Corner Road \nHolmdel, NJ 07733, USA \n\nAbstract \n\nWe developed a system for finding address blocks on mail pieces that can \nprocess four  images per second. Besides  locating the  address block,  our \nsystem also determines the writing style, handwritten or machine printed, and \nmoreover,  it measures  the  skew  angle  of the  text  lines  and  cleans noisy \nimages.  A  layout  analysis  of all  the  elements  present  in  the  image  is \nperformed in order to distinguish drawings and dirt from text and to separate \ntext of advertisement from that of the destination address. \nA  speed  of more  than  four  images per second  is obtained on a  modular \nhardware platform, containing a board with two of the NET32K neural net \nchips,  a  SP ARC2  processor  board,  and  a  board  with  2  digital  signal \nprocessors. The system has been tested with more than 100,000 images. Its \nperformance depends on the quality of the images, and lies between 85% \ncorrect location in very noisy images to over 98% in cleaner images. \n\n1 \n\nINTRODUCTION \n\nThe system described here has been integrated into an address reading machine developed for \nthe 'Remote Computer Reader' project of the United States Postal Service.  While the  actual \nreading of the text is done by other modules, this system solves one of the major problems, \nnamely, finding reliably the location of the destination address. There are only a few constraints \non how and where an  address has to be written, hence they may appear in a wide variety of \nstyles and layouts. Often an envelope contains advertising that includes  images as well as text. \n\n785 \n\n\f786 \n\nGraf and Cosatto \n\nSometimes. dirt covers part of the envelope image. including the destination address. Moreover. \nthe image captured by the camera is thresholded and  the reader is given a binary image.  This \nbinarization process introduces additional distortions; in particular. often the destination address \nis surrounded by a heavy texture. The high complexity of the images and their poor quality make \nit difficult to find the location of the destination address.  requiring an analysis of all the elements \npresent in the image.  Such an analysis is  compute-intensive and in our system it turned out to \nbe the major bottleneck for a fast throughput.  In fact. finding the address requires much more \ncomputation than reading it.  Special-purpose hardware in the form of the NET32K neural net \nchips (Graf. Henderson. 90) is used to solve the address location problem. \n\nFinding address blocks has been the focus of intensive research recently. as several companies \nare developing address reading machines (United States Postal Service 92).  The wide variety \nof images that have to be handled has led other researchers to apply several different analysis \ntechniques to each image and then try to combine the results at the end. see e.g. (palumbo et a1. \n92). In order to achieve the throughput required in an industrial application. special purpose \nprocessors for finding connected components and/or for executing Hough transforms have been \napplied. \n\nIn our system we use the NET32K processor to extract geometrical features from an image.  The \nhigh  compute  power  of this  chip  allows  the  extraction  of a  large  number  of  features \nsimultaneously. From this feature representation.  an interpretation of the image's content can \nthen be achieved with  a standard processor. Compared to an analysis of the original image. the \nanalysis of the feature maps requires  several orders of magnitude less computation. Moreover. \nthe feature representation introduces a high level of robustness against noise.  This paper gives \na brief overview of the hardware platfOlm  in section 2 and then describes the algorithms to find \nthe address blocks in section 3. \n\n2 THE HARDWARE \n\nThe NET32K system has been designed to serve as a high-speed image processing platform. \nwhere neural nets as well as conventional algorithms can be executed.  Three boards form  the \nwhole  system.  Two  NET32K  neural  net  chips  are  integrated  with  a  sequencer  and  data \nformatting  circuits on one board. The  second board contains two  digital  signal  processors \n(DSPs). together with 6 Mbytes of memOly.  Control of the whole system is provided by a board \ncontaining a SP ARC2  processor plus 64 Mbytes of memory.  A schematic of this system  is \nshown in Figure 1. \n\nImage buffering and communication with other modules in the address reader are handled by \nthe board with the SP ARC2 processor. When an image is received. it is sent to the DSP board \nand from there over to the NET32K processor.  The feature maps produced by the NET32K \nprocessor are stored on the DSP board.  while the SP ARC2 starts with the analysis of the feature \nmaps. The DSP's main task is formatting of the data. while the NET32K processor extracts all \nthe features. Its speed of computation is more than 100 billion multiply-accumulates per second \nwith operands that have one or two bits of resolution.  Images with a size of Sl2xS 12 pixels are \nprocessed at a rate of more than 10 frames per second. and 64 convolution kernels. each with \na size of 16x 16 pixels. can be scanned simultaneously over the  image. Each such kernel  IS \ntuned to detect the presence of  a feature.  such as a line, an edge or a comer. \n\n\fAddress Block Location with a Neural Net System \n\n787 \n\nNET32K MODULE \n\n.................................................................................... \n! \n\nI N~K I N~  r1~\u00b7~\u00b7~\u00b7:\u00b7A\u00b7T .. \nv. \n\nI \n\nI \n\n: \n\nDSP32C \n\nr ........ -.... -.-....... -..... ::::.fr=:::::::.~~~::=::::.~~=.=:::.=n:::=:: ..... _._ ...... _ ...... ! \n. \ni \n\n\" ). \n~~ \n~r \n~\"  ~U 'I \n\n~ \"> \n~ It \n~ \n~ \"  ~ \nL. .... ~~~ .. ~.~.~.~~., .... -.-.... ---.-.. -.. l-........ --... _ .... + .... _ ...... :;:~ ........... ..l \n\nSRAM \n1 MEG \n\nSRAM \n1 MEG \n\nDRAM \n4 MEG \n\nAfr \n~  '1 \n\nDSP32C \n\n\u2022 \n\n~lt \n~ .... '1 \n\n~ ________________________ ~.  SPARC \n\nVME BUS \n\nFigure 1: Schematic of the whole NET32K system. Each of the dashed \nboxes represents one 6U VME board.  The aITOWS show the \nconununication paths. \n\n3. SEQUENCE OF ALGORITHMS \nThe final result of the address block location system is a box describing a tight bmmd around \nthe destination address, if the address is machine printed. Of handwritten addresses, only the \nzip  code  is  read,  and  hence,  one  has  to  find  a  tight  boundary  around  the  zip  code.  This \ninformation is then passed along to reader modules of the address reading machine.  There is no \na  priori  knowledge  about  the  writing  style.  Therefore  the  system  first  has  to  discriminate \nbetween handwritten and machine Plinted text. At the end of the address block location process, \nadditional algorithms are executed to improve the accuracy of the reader. An overview of the \nsequence of algorithms used to solve these tasks is shown in Figure 2. The whole process is \ndivided into three major steps: Preprocessing, feature extraction. and high-level analysis based \non the feature information. \n\n3.1. Preprocessing \nTo quickly get an idea about the complexity of the image, a coarse evaluation of its layout is \ndone.  By sampling the density of the black pixels in various places of the image, one can see \nalready whether the image is clean or noisy and whether the text is lightly printed or is  dark. \n\n\f788 \n\nOraf and Cosatto \n\nThe images are divided into four categories, depending on their darkness and the level of noise. \n'This infonnation is used in the subsequent processing to guide the choice of the features. Only \nabout one percent of the pixels are taken into account for this analysis,  therefore,  it  can be \nexecuted quickly on the SP ARC2 processor. \n\nclean. light \n\nPreprocessing \n\nclean. dark \n\n=-.P IF. ~ \n--\n.... -.... ~ \n---. \n= \n~.= \n\n-\n\nExtract features \nNET32K \n\n16 Feature \n\nmaps \n\n8 Feature \n\nmaps \n\n: :~, \n\n' ,,' \n-I'\" ..... \n\nExtract text lines \nCluster lines into groups \n- - - Classify groups of lines \n\nMACHINE PRINT \n\nHANDWRITIEN \n\nAnalyse group of lines \nDetermine level of noise \nClean with NET32K; \n\nCluster text segments into lines \nAnalyse group of lines \nSegment lines to  find  ZIP \nDetermine slanVskew angle; \n\nFigure 2:  Schematic of the sequence of algorithms for finding the \nposition of the address blocks. \n\n3.2. Feature Extraction \n\nAfter the preprocessing, the  image  is  sent to the  NET32K board where simple  geometrical \nfeatures,  such as edges, corners and lines are extracted. Up to  16  different feature maps are \ngenerated, where a pixel in one of the maps indicates  the presence of a feature in this location. \nSome of these feature maps are used by the host processor, for example, to decide whether text \nis  handwritten or machine  printed.  Other feature  maps  are  combined  and  sent  once  more \nthrough the NET32K processor in order to  search for combinations of features  representing \nmore complex features.  Typically, the feature maps are thresholded,  so that only one bit per \npixel is kept. More resolution of the computation results is available from the neural net chips. \nbut in this way the amount of data that has to be analyzed is minimal. and one bit of resolution \nturned out to be sufficient. \n\nExamples of kernels used for the detection of strokes and text lines are shown in Figure 3.  In \nthe chip, usually four line detectors of increasing height plus eight stroke detectors of different \norientations are stored. Other detectors are tuned to edges and strokes of machine printed text. \nThe line detectors respond to any black line of the proper height.  Due to the large width of 16 \n\n\fAddress Block Location with a Neural Net System \n\n789 \n\npixels. a kernel stretches over one or even several characters. Hence a text line gives a  response \nsimilar to that produced by a continuous black line.  When the threshold is set properly. a text \nline in the original image produces a continuous  line in the feature map. even across the gaps \nbetween characters and across small empty spaces between words.  For an interpretation of a \nline feature map only the left and right end points of each connected component are stored.  In \nthis way one obtains a compact representation of the lines' positions that are well suited for the \nhigh-level analysis of the layout. \n\nKernel: Line detector \n\nImage \n\n\u2022 \nIC::GUla \n\nt  the NET32K syste \n\nFeature \n\nKernel: Stroke detector \n\nFeature map \n\nFigure 3:Examples of convolution kernels and their results. The kernels' sizes \nare  16x 16 pixels, and their pixels' values are + 1, O.  -1 . The upper part illustrates \nthe response of a line detector on a machine printed text line. The lower kernel \nextracts strokes of a celtain orientation from handwritten text. \n\nHandwritten lines are detected by a second technique, because they are more irregular in height \nand the characters may be spaced apm1 widely.  Detectors for strokes, of the type shown in the \nlower half ofFigw-e 3. are well suited for sensing the presence of handwritten text.  The feature \nmaps resulting form handwritten text tend to exhibit blobs of pixels along the text line.  By \nsmearing such feature  maps in horizontal  direction  the responses of individual  strokes are \nmerged into lines that can then be used in the same way as described for the machine printed \nlines. \n\nHorizontal smearing of text lines. combined with connected component analysis is a well-known \n\n\f790 \n\nGraf and Cosatto \n\ntechnique, often applied in layout analysis, to find words and whole lines of text.  But when \napplied to the pixels of an image, such an  approach works well only in clean images.  As soon \nas there is noise present, this technique produces ilTegular responses.  The key to success in a \nreal  world  environment  is  robustness  against  noise.  By  extracting  features  first  and  then \nanalyzing the feature maps, we drastically reduce  the influence of noise. Each of the convolution \nkernels  covers a range of 256 pixels and its response depends on several dozens of pixels inside \nthis area. If pixels in the image are corrupted by noise, this has only a minor effect on the result \nof the convolution and, hence, the appearance of the feature map. \n\nWhen the analysis is started, it is unknown,  whether the address is machine printed or hand \nwritten.  In order to distinguish between the two writing styles,  a simple one-layer classifier \nlooks at the results of four stroke detectors and of four line detectors. It can determine reliably \nwhether text  is handwritten or machine  printed.  Additional  useful  information  that  can  be \nextracted easily from the feature maps, is the skew angle of handwritten text.  People tend to \nwrite with a skew anywhere from -45 degrees to almost +90 degrees.  In order to improve the \naccuracy of a reader, the text is first deskewed.  The most time consuming part of this operation \nis to determine the skew angle of the writing.  The stroke detector with the maximum response \nover a line is a good indicator of the skew angle of the text.  We compared this simple technique \nwith several alternatives and found  it to be as reliable as the best other algorithm and much \nfaster to compute. \n\n3.3. High-level Analysis \nThe results of the feature extraction process are line segments, each one marked as handwritten \nor machine printed. Only the left and right end points of such lines are stored.  At this point, \nthere may still be line segments in this group that do not correspond to text, but rather to solid \nblack lines or to line drawings. Therefore each line segment is checked, to determine whether \nthe ratio of black and white pixels is that found typically in text. \n\nBlocks of lines are identified by clustering the line segments into groups.  Then each block is \nanalyzed, to see whether it can represent the destination address.  For this purpose such features \nas the number of lines in the block, its size, position, etc.  are used.  These features are entered \ninto a classifier that ranks each of the blocks.  Certain conditions, such as a size that is too large, \nor if there are  too many text lines in the block, will lead to an attempt to split blocks.  If  no good \nresult is obtained, clustering is tried again with a changed distance metric, where the horizontal \nand the vertical distances between lines are weighted differently. \n\nIf an address is machine printed, the whole address block is passed on to the reader, since  not \nonly the zip code, but the whole address, including the city name, the street name and the name \nof the recipient have to be read.  A big problem for the reader present images of poor quality, \nparticularly those with background noise and texture.  State-of-the-art readers handle machine \nprinted text reliably if the image quality is good, but they may fail totally if the text is buried in \nnoise.  For that  reason,  an  address block  is cleaned before sending it to the  reader.  Feature \nextraction with the NET32K board is used once more for this task, this time with detectors tuned \nto find  all the strokes of the machine printed text.  Applying stroke detectors with the proper \nwidth allows a good discrimination between the text and any noise. Even texture that consists \nof lines can be rejected reliably, if the line thickness of the texture is not the same as that of the \ntext. \n\n\fAddress Block Location with a Neural Net System \n\n791 \n\n.:  \" \n\n.  ', '  \n\n\"3\"\" /\"ksiQ \\i~.\\.  Cal! [~ \n~\"S'~e\".I \u2022 \n. ~ .. ~ ~ ===t  ,o;;;r;;;a.e;2  . \n\n.  :  . \n\n..... \n\nt1r \n\u00b7\u00b7\u00b7\u00b7ee-5AT'fO\u00b7\u00b7t;~a.\u00b7\u00b7\u00b7\u00b7\u00b7 \n\"'~;Au'j'':f;,:.)i'''\\i\u00b7bl,..~~\u00b7\u00b7\u00b7~t ....... \n\"\u00b7~\u00b7S\\;.\u00b7\\\u00b7.cs.\",~\u00b7A'\u00b7\u00b7\u00b7\"\" \n-~.W\" .. -,\\e\"'..4*!~ .. _Q33.~2..:-\n\nFigure 4: Example of an envelope image at various stages of the processing. Top: The \nresult of the clustering process to find the bounding box of the address. Bottom right:  The \ntext lines within the address block are marked.  Bottom left:  Cuts in the text line with the \nzip code and below that the result of the reader.  (The zip code is actually the second \nsegment sent to the reader; the first one is the string 'USA'). \n\nIf the address is handwritten, only the zip code is sent to the reader. In order to find the zip code, \nan analysis of the internal stmcture of the address block has to be done, which starts with finding \nthe true text lines.  Handwritten lines are often not straight, may be heavily skewed, and may \ncontain large gaps. Hence simple techniques, such as connected component analysis, do not \nprovide proper results. ClusteJing of the line segments obtained from the feature maps, provides \na reliable solution of this problem. Once the lines are found, each one is segmented into words \nand some of them are selected as candidates for the zip code and are sent to the reader. Figure \n4 shows an example of an envelope image as it progresses through the various processing steps. \n\nThe system has been tested extensively on overall more than  100,000 images.  Most of these \ntests were done in the assembled address reader, but during development of the system, large \n\n\f792 \n\nGraf and Cosatto \n\ntests were also done with the address location module alone.  One of the problems for evaluating \nthe peIformance is the lack of an objective quality measure.  When has an address been located \ncorrectly?  Cutting  off  a  small  part  of the  address  may  not  be  detrimental  to  the  final \ninterpretation, while a bounding box that includes some additional text may slow the reader \ndown too much. or it may throw off the interpretation.  Therefore, it is not always clear when a \nbounding  box,  describing  the  address'  location,  is  tight  enough.  Another  important  factor \naffecting the accw-acy numbers is, how many candidate blocks one actually considers. For all \nthese reasons, accw-acy numbers given for address block location have to be taken with some \ncaution. The results mentioned here were obtained by  judging the images by eye. If images are \nclean and  the address is surrounded by a white space larger than two line heights, the location \nis found correctly in more than 98% of the cases. Often more than one text block is found  and \nof these the destination address is the first choice in 90% of the images, for  a typical layout. If \nthe image is very noisy,  which actually happens surprisingly often,  a tight bound around the \naddress is found  in 85% of the cases. These results were obtained with 5,000 images, chosen \nfrom more than 100,000 images to represent as much variety as possible.  Of these 5,000 images \nmore than  1,200 have a  texture around the  address,  and often this texture  is so dark that a \nhuman has difficulties to make out each character. \n\n4. CONCLUSION \nMost of our algorithms described here  consist of two parts: feature extraction implemented with \na convolution and interpretation, typically implemented with a small classifier.  Surprisingly \nmany algorithms can be cast into such a fOimat.  This common framework for algorithms  has \nthe advantage of facilitating the implementation, in particular when algorithms are mapped into \nhardware. Moreover, the feature extraction with large convolution kernels makes the system \nrobust against noise. This robustness is probably the biggest advantage of our approach.  Most \nexisting automatic reading systems are  very  good  as  long as the  images are clean, but they \ndeteriorate rapidly with decreasing image quality. \n\n'The biggest drawback of convolutions is that they require a lot of computation. In fact, without \nspecial purpose hardware, convolutions are often too slow.  Our system relies on the NET32K \nnew-al net chips to obtain the necessary throughput.  The NET32K system is, we believe, at the \nmoment  the  fastest  board  system  for  this  type  of computation.  This  speed  is  obtained  by \nsystematically exploiting the fact that only a low resolution of the computation is required.  This \nallows to use analog computation inside the chip and hence much smaller circuits than would \nbe the case in an all-digital circuit. \n\nReferences \nUnited States Postal Service, (1992), Proc. Advanced Technology Conf.,  Vol.  3, Section on \naddress block location: pp.  1221  - 1310. \n\nP.W. Palumbo, S.N. Srihari, J.  Soh, R. Sridhar,  V.  Demjanenko, (1992), !'Postal Address Block \nLocation in Real Time\", IEEE COMPUTER, Vol.  25n, pp. 34 - 42. \n\nH.P.  Oraf and D.  Henderson,  (1990),  \"A Reconfigurable  CMOS Neural  Network\",  Digest \nIEEE Int.  Solid State Circuits Conf.  p.  144. \n\n\f", "award": [], "sourceid": 779, "authors": [{"given_name": "Hans", "family_name": "Graf", "institution": null}, {"given_name": "Eric", "family_name": "Cosatto", "institution": null}]}