{"title": "Using Unlabeled Data for Supervised Learning", "book": "Advances in Neural Information Processing Systems", "page_first": 647, "page_last": 653, "abstract": null, "full_text": "Using Unlabeled  Data for  Supervised \n\nLearning \n\nGeoffrey Towell \n\nSiemens  Corporate Research \n\n755  College Road East \nPrinceton, N J  08540 \n\nAbstract \n\nMany classification problems have the property that the only costly \npart of obtaining examples  is  the class  label.  This  paper suggests \na  simple  method  for  using  distribution  information  contained  in \nunlabeled  examples  to augment  labeled  examples  in  a  supervised \ntraining framework.  Empirical  tests  show  that  the  technique  de(cid:173)\nscribed  in  this  paper  can  significantly  improve  the  accuracy  of  a \nsupervised  learner  when  the  learner  is  well  below  its  asymptotic \naccuracy level. \n\n1 \n\nINTRODUCTION \n\nSupervised learning problems often have the following property:  unlabeled examples \nhave little or no cost while class labels have a high cost.  For example, it is  trivial to \nrecord  hours  of heartbeats from  hundreds  of patients.  However,  it  is  expensive to \nhire cardiologists to label each of the recorded beats.  One response to the expense of \nclass labels is to squeeze the most information possible out of each labeled example. \nRegularization  and  cross-validation  both have  this  goal.  A  second  response  is  to \nstart with a small set of labeled examples and request labels of only those currently \nunlabeled  examples that  are expected  to provide  a significant  improvement  in  the \nbehavior of the classifier  (Lewis  & Catlett,  1994; Freund  et  al.,  1993). \n\nA  third  response  is  to  tap  into  a  largely  ignored  potential  source  of information; \nnamely,  unlabeled  examples.  This  response  is  supported  by  the  theoretical  work \nof Castelli and Cover  (1995)  which suggests that unlabeled examples have value  in \nlearning classification problems.  The algorithm described in this paper, referred to \nas SULU (Supervised learning  Using Labeled and Unlabeled examples), takes this third \n\n\f648 \n\nG.  TOWELL \n\npath by using distribution information from  unlabeled examples during supervised \nlearning.  Roughly,  SULU uses the centroid of labeled and unlabeled examples in the \nneighborhood  of a  labeled  example  as  a  new  training  example.  In  this  way,  SULU \nextracts information  about  the local  variability of the  input  from  unlabeled  data. \nSULU  is  described in Section 2. \n\nIn  its  use  of unlabeled  examples to alter labeled  examples,  SULU  is  reminiscent  of \ntechniques for  adding noise  to networks during training  (Hanson,  1990;  Matsuoka, \n1992).  SULU is  also reminiscent of instantiations of the EM algorithm that attempt \nto fill  in  missing parts of examples  (Ghahramani &  Jordan,  1994).  The similarity \nof SULU  to these,  and other, works is  explored in Section 3. \n\nSULU is intended to work on classification problems for which there is insufficient la(cid:173)\nbeled training data to allow a learner to approach its asymptotic accuracy level.  To \nexplore this problem, the experiments described in Section 4 focus on the early parts \nof the learning curves  of six  datasets  (described  in  Section  4.1).  The results  show \nthat  SULU  consistently,  and  statistically  significantly,  improves  classification  accu(cid:173)\nracy over systems trained with only the labeled data.  Moreover, SULU is consistently \nmore  accurate than  an  implementation  of the  EM-algorithm  that  was  specialized \nfor  the task of filling  in missing class labels.  From these results, it is  reasonable to \nconclude that SULU is able to use the distribution information in unlabeled examples \nto improve classification accuracy. \n\n2  THE ALGORITHM \n\nSULU  uses  standard  neural-network  supervised  training  techniques  except  that  it \noccasionally replaces a  labeled example with  a  synthetic example.  in  addition,  the \ncriterion to stop training is  slightly modified  to require that the network  correctly \nclassify almost every labeled example and a majority of the synthetic examples.  For \ninstance, the experiments reported in Section 4 generate synthetic examples 50% of \nthe time;  the stopping criterion requires that 80%  of the examples seen in a  single \nepoch  are  classified  correctly.  The  main  function  in  Table  1  provides  psuedocode \nfor  this process. \n\nThe synthesize function in Table 1 describes the process through which an example is \nsynthesized.  Given a labeled example to use as a seed, synthesize collects neighboring \nexamples  and  returns  an  example  that  is  the  centroid  of the  collected  examples \nwith the label  of the starting point.  synthesize  collects  neighboring  examples  until \nreaching one of the following three stopping points.  First, the maximum number of \npoints  is  reached;  the goal  of SULU  is  to get  information  about  the  local  variance \naround  known  points,  this  criterion  guarantees  locality.  Second,  the  next  closest \nexample to the seed is a labeled example with a different label; this criterion prevents \nthe inclusion of obviously incorrect  information  in  synthetic examples.  Third,  the \nnext  closest  example  to the  seed  is  an  unlabeled  example  and  the  closest  labeled \nexample to that unlabeled example has a different label from the seed; this criterion \nis  intended to detect  borders between  classification areas in example space. \n\nThe call to synthesize from  main effectively samples with replacement  from  a  space \ndefined  by  a  labeled  example  and its neighbors.  As  such,  there  are many  ways  in \nwhich  main  and  synthesize  could  be  written.  The  principle  consideration  in  this \nimplementation is memory;  the space around the labeled examples can be huge. \n\n\fUsing Unlabeled Data for Supervised Learning \n\n649 \n\nTable  1:  Pseudocode for  SULU \n\nRANDOH(min,max): \n\nreturn  a  uniformly  distributed  random  integer  between  min  and  max,  inclusive \n\nHAIN(B,H): \n\nin  [0 .. 100],  controls  the  rate  of  example  synthesis  */ \n*/ \n\n/*  B  -\n/*  H - controls  neighborhood  size  during  synthesis \nLet:  E \nU \nN \n\n*/ \n/*  a  set  of  labeled  examples \n/*  a  set  of  unlabeled  examples \n*/ \n/*  an  appropriate  neural  network  */ \n\nRepeat \n\nPermute  E \nForeach  e  in  E \n\nif  random(0,100)  >  B  then \n\ne  (- SYNTHESIZE(e,E,U,random(2,M\u00bb \n\nTRAIN  N using  e \n\nUntil  a  stopping  criterion  is  reached \n\nSYNTHESIZE(e,E,U,m): \n\nLet:  C \nFor  i  from  1  to  m \n\n/*  will  hold  a  collection  of  examples  */ \n\nc  (- ith  nearest  neighbor  of  e  in  E  union  U \nif  \u00abc  is  labeled)  and  (label  of  c  not  equal  to  label  of  e\u00bb \nif  c  is  not  labeled \n\ncc  (- nearest  neighbor  of  c  in  E \nif  label  of  cc  not  equal  to  label  of  e  then  STOP \n\nadd  c  to  C \n\nreturn  an  example  whose  input  is  the  centroid  of  the \n\ninputs  of  the  examples  in  C  and  has  the  class  label  of  e. \n\nthen  STOP \n\n3  RELATED WORK \n\nSULU is similar to two methods of exploring the input space beyond the boundaries of \nthe labeled examples;  example generation and noise  addition.  Example generation \ncommonly  uses  a  model  of how  a  space  deforms  and  an  example of the  space  to \ngenerate  new  examples.  For  instance,  in  training  a  vehicle  to  turn,  Pomerleau \n(1993)  used information about how the scene shifts when a car is turned to gener\u00b7ate \nexamples of turns.  The major problem with example generation is that deformation \nmodels  are uncommon. \n\nBy  contrast  to  example  generation,  noise  addition  is  a  model-free  procedure.  In \ngeneral,  the  idea  is  to  add  a  small  amount  of noise  to  either  inputs  (Matsuoka, \n1992),  link  weights  (Hanson,  1990),  or  hidden  units  (Judd  &  Munro,  1993).  For \nexample,  Hanson  (1990)  replaces  link weights  with a  Gaussian.  During a  forward \npass,  the  Gaussian is  sampled to determine the link weight.  Training affects  both \nthe  mean  and  the  variance  of the  Gaussian.  In  so  doing,  Hanson's  method  uses \ndistribution information in the labeled examples to estimate the global variance of \neach input dimension.  By contrast, SULU uses both labeled and unlabeled examples \nto make local  variance estimates.  (Experiments,  results not  shown,  with  Hanson's \nmethod indicate that it cannot improve classification results as much  as SULU.) \n\nFinally,  there  has  been  some  other  work  on  using  unclassified  examples  during \ntraining.  de  Sa (1994)  uses  the  co-occurrence of inputs in  multiple sensor modali-\n\n\f650 \n\nG.  TOWELL \n\nties to substitute for missing class information.  However, sensor data from  multiple \nmodalities  is  often  not  available.  Another  approach  is  to  use  the  EM  algorithm \n(Ghahramani  &  Jordan,  1994)  which  iteratively  guesses  the  value  of missing  in(cid:173)\nformation  (both  input  and  output)  and  builds  structures  to  predict  the  missing \ninformation.  Unlike  SULU,  EM  uses  global  information  in  this  process  so  it  may \nnot perform well on highly disjunctive problems.  Also SULU may have an advantage \nover EM in domains in which only the class label is missing as that is SULU'S specific \nfocus. \n\n4  EXPERIMENTS \n\nThe  experiments  reported  in  this  section  explore  the  behavior  of  SULU  on  six \ndatasets.  Each  of  the  datasets  has  been  used  previously  so  they  are  only  briefly \ndescribed  in  the first  subsection.  The  results  of  the  experiments  reported  in  the \nlast  part  of  this  section  show  that  SULU  significantly  and  consistently  improves \nclassification results. \n\n4.1  DATASETS \n\nThe first  two datasets are from  molecular biology.  Each take  a DNA  sequence  and \nencode  it  using  four  bits  per  nucleotide.  The first  problem,  promoter recognition \n(Opitz &  Shavlik,  1994),  is:  given  a  sequence of 57 DNA  nucleotides,  determine  if \na  promoter begins  at  a  particular  position  in  the  sequence.  Following  Opitz and \nShavlik,  the  experiments  in  this  paper  use  234  promoters  and  702  non promoters. \nThe  second  molecular  biology  problem,  splice-junction  determination  (Towell  & \nShavlik,  1994), is:  given a  sequence of 60  DNA  nucleotides,  determine if there is  a \nsplice-junction  (and  the  type of the junction)  at  the middle  of the sequence.  The \ndata consist  of 243  examples of one junction type  (acceptors),  228 examples of the \nother junction type  (donors)  and 536 examples of non-junctions.  For both of these \nproblems,  the  best  randomly  initialized  neural  networks  have  a  small  number  of \nhidden units in a  single layer  (Towell & Shavlik,  1994). \n\nThe  remaining four  datasets  are  word  sense  disambiguation  problems  (Le.  deter(cid:173)\nmine  the  intended  meaning  of the  word  \"pen\"  in  the  sentence  \"the  box is  in  the \npen\").  The problems are to learn to distinguish  between six  noun senses  of  \"line\" \nor four  verb senses of \"serve\"  using either topical or local encodings  (Leacock  et al., \n1993)  of a  context around the target word.  The line dataset contains 349 examples \nof each  sense.  Topical  encoding,  retaining  all  words  that  occur  more  than  twice, \nrequires  5700  position  vectors.  Local  encoding,  using  three  words  on  either  side \nof line,  requires 4500  position vectors.  The serve dataset contains 350 examples of \neach sense.  Under  the same conditions  as  line,  topical  encoding requires  4400  po(cid:173)\nsition vectors while  local encoding requires 4500  position vectors.  The best neural \nnetworks for  these  problems have no hidden units  (Leacock  et  al.,  1993). \n\n4.2  METHODOLOGY \n\nThe  following  methodology  was  used  to  test  SULU  on  each  dataset.  First,  the \ndata was  split  into  three  sets,  25  percent  was  set  aside  to  be  used  for  assessing \ngeneralization,  50  percent  had  the  class  labels  stripped  off,  and  the  remaining  25 \npercent was  to be used for  training.  To create learning curves,  the training set  was \n\n\fUsing Unlabeled Data for Supervised Learning \n\n651 \n\nTable  2:  Endpoints  of the  learnings  curves  for  standard  neural  networks  and  the \nbest result for  each of the six datasets. \nSplice \n\nServe \n\nLine \n\nTraining \nSet size \nsmallest \nlargest \nasymptotic \n\nPromoter \n\n74.7 \n90.3 \n95.8 \n\nJunction  Local  Topical  Local  Topical \n\n66.4 \n85.4 \n94.4 \n\n53.9 \n71.7 \n83.1 \n\n41.8 \n63.0 \n75.5 \n\n38.7 \n58.8 \n70.1 \n\n40.6 \n63.3 \n79.2 \n\nfurther subdivided into sets containing 5, 10, 15, 20 and 25  percent of the data such \nthat smaller sets were  always subsets of larger sets.  Then,  a  single  neural network \nwas created and copied 25 times.  At each training set size, a new copy of the network \nwas  trained  under  each  of the following  conditions:  1)  using  SULU,  2)  using  SULU \nbut supplying only the labeled training examples to synthesize, 3) standard network \ntraining, 4)  using a variant of the EM algorithm that has been specialized to the task \nof filling  in  missing  class  labels,  and  5)  using  standard network training but with \nthe 50% unlabeled prior to stripping the labels.  This procedure was repeated eleven \ntimes to average out the effects of example selection and network initialization. \n\nWhen  SULU  was  used,  synthetic examples replaced labeled  examples  50  percent  of \nthe  time.  Networks  using  the  full  SULU  (case  1)  were  trained  until  80  percent  of \nthe examples  in  a  single  epoch  were  correctly  classified.  All  other  networks  were \ntrained  until  at  least  99.5%  of  the  examples  were  correctly  classified.  Stopping \ncriteria  intended  to  prevent  overfitting  were  investigated,  but  not  used  because \nthey never improved generalization. \n\n4.3  RESULTS  &  DISCUSSION \n\nFigure  1  and  Table  2  summarize  the  results  of  these  experiments.  The  graphs \nin  Figure  1  show  the  efficacy  of  each  algorithm.  Except  for  the  largest  training \nset on the splice junction problem,  SULU  always results in a  statistically significant \nimprovement over the standard neural network with at least 97.5 percent confidence \n(according to a one-tailed paired-sample t-test).  Interestingly,  SULU'S improvement \nis consistently between :t  and ~ of that achieved by labeling the unlabeled examples. \nThis result contrasts Castelli and Cover's (1995) analysis which suggests that labeled \nexamples are exponentially more valuable than unlabeled examples. \n\nIn addition, SUL U is consistently and significantly superior to the instantiation of the \nEM-algorithm when there are very few  labeled  samples.  As  the number of labeled \nsamples increases the advantage of SULU  decreases.  At  the largest training set sizes \ntested,  the two systems are roughly equally effective. \n\nA  possible criticism of SULU  is  that it does  not actually need  the unlabeled exam(cid:173)\nples;  the procedure may be  as  effective  using only the labeled  training data.  This \nhypothesis is incorrect, As  shown in Figure 1, SULU when given no unlabeled exam(cid:173)\nples is  consistently and significantly inferior ti SULU  when  given  a  large number of \nunlabeled examples.  In addition,  SULU  with no unlabeled examples is  consistently, \nalthough not always significantly, inferior to a standard neural network. \n\nThe  failure  of  SULU  with  only  labeled  examples  points  to  a  significant  weakness \n\n\f652 \n\nG.  TOWELL \n\n~r-----------~~_~_~.~~7_~~~~~--' \n\n~r---~------~~_~_~~---+~~-~----~ \n\n--SI..l.U Wl1h _un~ \n- - - EM\"\"'_~ \n\nSUlU .... O~ \n\n+  SIIl&IIIcII't'-..penorto5U.U \no  SIIIblIlcllly _ new .. SULU \n\n\" \n\n'8~ \n\n\"\\. \n\n................ \n\n--8ll.u .... 50'2un~ \n~ g \n~~~ :.~== \n_ \n....... ~~..:lIIIaaly .. nor \u00bb SULU \n..fi~ \n+  Strolllk:lllJ~tDSLl.U \nj:r---4~~\n\n\"-\"-\"~\"'~\"~\"~\"-\"-\"~\"-\"--~-\"-\"~~~':~--~ \n\n----G----_---~ \n\n\"\"\"-+-_ \n\n0----\n\n~r---~c---~,~---,~--~~--~ \n\n~r---~--~l ~O --~l ~~~--~~~{\u00b7 \n\nSize oIlraining sel \n\nSize oIlralnlng sel \n\n~~~ ____ ~~~~~_Ud_~~U~~_Dh~~~~~~-' \n\n'-'-,-\n\n'\n\n........ \n\n......... ......... \n\n- - - --- SlJ..UWllhOurNbe'-d \n\n== :-~-:oo~= \n---..... _---. \n\n+  Stabs\\ll::aly' M.lpenor to SlLU \n......... ~o $tabstlcaly' ...... m SlJ..U \n\n'\n\n.......... \n\n........... \n\n......... '+-- -- __ \n-\n\n-\n\n-\n\n- - SLl..U wrIh  1046 un1 . . . .  \nEM\"lrI l04&~ \n-\n~----- su. Uwrttl ()lA'l~ \n+  SlatlSt.c.Ity alpena, m SUlU \n__ ....... -.!! _stat\\s\\lealyn1erD'tDSlJ..U \n\n- --+-----. \n\n~ \n\n__ -..e-- -\n\n~o \n0.. ....  -\nC'\" \n~ \n~o~----~~----~-----&------~ \n<>. \n\n__ __ -Ouu - - _u __ -e-. __ ~ \n\nFigure 1:  The effect of five  training procedures on each of six learning problems.  In \neach of the above graphs, the effect of standard neural learning has been subtracted \nfrom  all  results  to  suppress  the  increase  in  accuracy  that  results  simply  from  an \nincrease in the number of labeled training examples.  Observations marked by a  '0' \nor a  '+' respectively indicate that the point  is  statistically significantly  inferior or \nsuperior to a  network trained using  SULU. \n\nin  its  current  implementation.  Specifically,  SULU  finds  the  nearest  neighbors  of \nan  example  using  a  simple  mismatch  counting procedure.  Tests  of this  procedure \nas  an  independent  classification  technique  (results  not  shown)  indicate  that  it  is \nconsistently much worse than any of the methods plotted in in  Figure 1.  Hence, its \nuse  imparts a  downward bias to the generalizatio~ results. \n\nA  second  indication  of room  for  improvement  in  SULU  is  the  difference  in  gener(cid:173)\nalization  between  SULU  and  a  network  trained  using  data in  which  the  unlabeled \nexamples provided to SULU  have labels  (case 5  above).  On every dataset, the gain \nfrom  labeling  the examples  is  statistically significant.  The  accuracy of a  network \ntrained with all labeled examples is  an upper bound for  SULU, and one that is likely \nnot reachable.  However, the distance between the upper bound and SULU'S current \nperformance indicate that there is  room for  improvement. \n\n\fUsing Unlabeled Data for Supervised Learning \n\n653 \n\n5  CONCLUSIONS \n\nThis paper has presented the SULU algorithm that combines aspects of nearest neigh(cid:173)\nbor classification  with  neural  networks  to learn  using  both labeled  and  unlabeled \nexamples.  The  algorithm  uses  the  labeled  and  unlabeled  examples  to  construct \nsynthetic examples  that capture information  about  the  local  characteristics of the \nexample  space.  In  so  doing,  the  range  of examples  seen  by  the  neural  network \nduring its supervised learning is  greatly expanded which results in improved gener(cid:173)\nalization.  Results of experiments on  six real-work datasets indicate that  SULU  can \nsignificantly improve generalization when  when  there  is  little labeled  data.  More(cid:173)\nover, the results indicate that SULU  is consistently more effective at using unlabeled \nexamples than the EM-algorithm when there is very little labeled data.  The results \nsuggest  that  SULU  will  be effective  given  the following  conditions:  1)  there is  little \nlabeled training data, 2)  unlabeled training data is  essentially free,  3)  the accuracy \nof the classifier when trained with all of the available data is  below the level which \nis  expected  to  be  achievable.  On  problems with  all of these  properties  SULU  may \nsignificantly improve the generalization accuracy of inductive classifiers. \n\nReferences \n\nCastelli, V.  &  Cover, T. (1995).  The  relative  value  of labeled and unlabeled samples in pat(cid:173)\ntern  recognition  with  an  unknown mixing parameter.  (Technical Report 86),  Department \nof Statistics:  Stanford University. \n\nde  Sa,  V.  (1994).  Learning classification with unlabeled data.  Advances  in  Neural  Infor(cid:173)\nmation Processing  Systems,  6. \n\nFreund,  Y.,  Seung,  H.  S.,  Shamit,  E.,  &  Tishby,  N.  (1993).  Information,  prediction and \nquery by committee.  Advances  in  Neural  Information  Processing  Systems,  5. \n\nGhahramani,  Z.  &  Jordan,  M.  I.  (1994).  Supervised learning from  incomplete  data via \nan EM  approach.  Advances  in Neural  Information  Processing  Systems,  6. \n\nHanson,  S.  J.  (1990).  A stochastic version  of the delta rule.  Physica  D,  42,  265-272. \n\nJudd,  J.  S.  &  Munro,  P.  W.  (1993).  Nets  with  unreliable  hidden  units  learn  errOr(cid:173)\ncorrecting codes.  Advances  in Neural  Information  Processing  Systems,  5. \n\nLeacock,  C.,  Towell,  G.,  &  Voorhees,  E.  M.  (1993).  Towards  building  contextual  rep(cid:173)\nresentations of word  senses using statistical  models.  Proceedings  of SIGLEX  Workshop: \nAcquisition  of Lexical  Knowledge  from  Text.  Association for  Computational Linguistics. \n\nLewis,  D.  D.  &  Catlett,  J.  (1994).  Heterogeneous  uncertainty sampling  for  supervised \nlearning.  Eleventh  International  Machine  Learning  Conference. \n\nMatsuoka,  K.  (1992).  Noise  injection  into  inputs  in  back-propagation  learning.  IEEE \nTransactions  on  Systems,  Man  and  Cybernetics,  22,  436-440. \n\nOpitz,  D.  W.  &  Shavlik,  J.  W.  (1994).  Using  genetic search  to refine  knowledge-based \nneural networks.  Eleventh  International  Machine  Learning  Conference. \n\nPomerleau, D.  A.  (1993).  Neural  Network Perception for Mobile  Robot Guidance.  Boston: \nKluwer. \n\nTowell,  G.  G.  &  Shavlik,  J.  W.  (1994).  Knowledge-based  artificial  neural  networks. \nArtificial Intelligence,  70,  119-165. \n\n\f", "award": [], "sourceid": 1097, "authors": [{"given_name": "Geoffrey", "family_name": "Towell", "institution": null}]}