{"title": "Recognizing Evoked Potentials in a Virtual Environment", "book": "Advances in Neural Information Processing Systems", "page_first": 3, "page_last": 9, "abstract": null, "full_text": "Recognizing Evoked Potentials in a Virtual \n\nEnvironment * \n\nJessica D. Bayliss and Dana H. Ballard \n\nDepartment of Computer Science \n\nUniversity of Rochester \nRochester, NY  14627 \n\n{bayliss,dana}@cs.rochester.edu \n\nAbstract \n\nVirtual  reality  (VR)  provides  immersive  and  controllable  experimen(cid:173)\ntal  environments.  It expands  the  bounds  of possible  evoked  potential \n(EP)  experiments by  providing complex,  dynamic  environments in  or(cid:173)\nder  to  study  cognition  without  sacrificing  environmental  control.  VR \nalso serves as a safe dynamic testbed for brain-computer .interface (BCl) \nresearch.  However, there has been some concern about detecting EP sig(cid:173)\nnals in a complex VR environment.  This paper shows that EPs exist at \nred,  green, and yellow stop lights in  a virtual driving environment.  Ex(cid:173)\nperimental results show  the existence of the  P3  EP at  \"go\" and  \"stop\" \nlights and  the contingent negative variation (CNY)  EP  at \"slow down\" \nlights.  In order to  test the  feasibility  of on-line recognition in  VR,  we \nlooked at recognizing the P3 EP at red stop tights and the absence of this \nsignal at yellow slow down lights.  Recognition results show that the P3 \nmay successfully be used to control the brakes of a VR car at stop lights. \n\n1  Introduction \n\nThe controllability of VR makes it an excellent candidate for use in studying cognition.  It \nexpands the bounds of possible evoked potential (EP) experiments by providing complex, \ndynamic environments in order to  study decision making in  cognition without sacrificing \nenvironmental control. We have created a flexible system for real-time EEG collection and \nanalysis from within virtual environments. \n\nThe ability of our system to give quick feedback enables it to be used in brain-computer in(cid:173)\nterface  (BCl)  research,  which  is  aimed  at helping individuals with  severe  motor deficits \nto  become  more  independent.  Recent  BCl  work  has  shown  the  feasibility  of on-line \naveraging  and  biofeedback  methods  in  order  to  choose  characters  or  move  a  cursor  on \na  computer  screen  with  up  to  95%  accuracy  while  sitting  still  and  concentrating  on \nthe  screen  [McFarland  et  aI.,  1993;  Pfurtscheller  et  al.,  1996;  Vaughn  et  al.,  1996; \nFarwell  and  Donchin,  1988].  Our focus  is  to  dramatically  extend  the  BCl  by  allowing \nevoked potentials to propel the user through alternate virtual environments. For example, a \n*This  research was supported by  NIHIPHS grantl-P41-RR09283.  It was  also facilitated in  part \nby a National Physical Science Consortium Fellowship and by stipend support from NASA Goddard \nSpace Flight Center. \n\n\f4 \n\nJ.  D.  Bayliss and D. H.  Ballard \n\nFigure  1:  (Left)  An  individual  demonstrates  driving  in  the  modified  go  cart.  (Right) A typical \nstoplight scene in the virtual environment. \n\nuser could choose a virtual living room from a menu of rooms, navigate to the living room \nautomatically in the head-mounted display, and then choose to turn on the stereo. \nAs  shown  in  [Farwell and  Donchin,  1988], the P3  EP may  be  used  for  a brain-computer \ninterface that picks characters on a computer monitor.  Discovered by [Chapman and Brag(cid:173)\ndon, 1964; Sutton et aI.,  1965] and extensively studied (see [Polich,  1998] for a literature \nreview),  the P3  is  a positive waveform occurring approximately 300-500 ms  after an  in(cid:173)\nfrequent task-relevant stimulus.  We  show that requiring subjects to  stop  or go at  virtual \ntraffic lights elicits this EP.  The contingent negative variation (CNV), an EP that happens \npreceding an expected stimulus, occurs at slow down lights. \n\nIn  order  to  test  the  feasibility  of on-line  recognition  in  the  noisy  VR  environment,  we \nrecognized the P3  EP  at red  stop  lights  and  the  lack of this  signal  at yellow  slow  down \nlights.  Results  using a  robust Kalman filter  for  off-line recognition indicate that the  car \nmay be stopped reliably with an average accuracy of 84.5% while the on-line average for \ncar halting is 83%. \n\n2  The Stoplight Experiments \n\nThe first  experiment we  performed in the virtual driving environment shows that a P3  EP \nis obtained when subjects stop or go at a virtual light and that a CNV occurs when subjects \nsee a slow down light.  Since all subjects received the same light colors for the slow down, \ngo, and stop conditions we then performed a second experiment with different light colors \nin order to disambiguate light color from the occurrence of the P3 and CNV. \n\nPrevious P3 research has concentrated primarily on static environments such as the contin(cid:173)\nuous performance task  [Rosvold et aI.,  1956].  In the visual continuous performance task \n(VCPT), static images are flashed on a screen and the subject is told to press a button when \na rare stimulus occurs or to count the number of occurrences of a rare stimulus. This makes \nthe stimulus both rare and task relevant in order to evoke a P3.  As an example, given red \nand yellow stoplight pictures, a P3 should occur if the red picture is less frequent than the \nyellow and subjects are told to press a mouse button only during the red light. We assumed \na similar response would occur in a VR driving world if certain lights were infrequent and \nsubjects  were told  to  stop  or go  at them.  This  differs  from  the  VCPT in  two  important \nways: \n\n1.  In the  VCPT subjects  sit passively  and  respond  to  stimuli.  In  the  driving  task, \n\n\fRecognizing Evoked Potentials in a Virtual Environment \n\n5 \n\nsubjects control when the stimuli appear by where they drive in the virtual world. \n\n2.  Since subjects are actively involved and fully immersed in the virtual world, they \nmake more eye and head movements.  The movement amount can be reduced by \na particular experimental paradigm, but it can not be eliminated. \n\nThe first difference makes the VR environment a more natural experimental environment. \nThe second difference means that subjects create more data artifacts with extra movement. \nWe  handled these artifacts  by first  manipulating the experimental environment to reduce \nmovements where important stimulus events occurred. This meant that all stoplights were \nplaced at the end of straight stretches of road in order to avoid the artifacts caused by turning \na corner. For our on-line recognition, we then used the eye movement reduction technique \ndescribed in  [Semlitsch et al.,  1986] in order to subtract a combination of the remaining \neye and head movement artifact. \n\n2.1  Experimental Setup \n\nAll  subjects used  a modified go cart in order to control the virtual car (see Figure 1).  The \nvirtual reality  interface is rendered on a  Silicon Graphics Onyx  machine with 4  proces(cid:173)\nsors and an Infinite Reality Graphics Engine.  The environment is presented to the subject \nthrough a head-mounted display (HMD). Since scalp EEG recordings are measured in mi(cid:173)\ncrovolts, electrical signals may easily interfere during an experiment. We tested the effects \nof wearing a VR4 HMD containing an ISCAN eye tracker and discovered that the noise \nlevels  inside of the VR helmet were comparable to  noise levels  while  watching  a laptop \nscreen [Bayliss and Ballard,  1998]. \n\nA  trigger pulse containing information about the color of the light was  sent to  the  EEG \nacquisition system whenever a light changed. While an epoch size from -100 ms to  1 sec \nwas specified, the data was recorded continuously. Information about head position as well \nas gas, braking, and steering position were saved to an external file.  Eight electrodes sites \n(FZ, CZ, CPZ, PZ, P3, P4, as well as 2 vertical EOG channels) were arranged on the heads \nof seven  subjects  with  a  linked  mastoid reference.  Electrode  impedances  were between \n2  and  5  kohms  for  all  subjects.  Subjects ranged  in  age  from  19  to  52 and  most had  no \nprevious experiences in a virtual environment.  The EEG signal was amplified using Grass \namplifiers with an analog bandwidth from 0.1  to 100 Hz.  Signals were then digitized at a \nrate of 500 Hz and stored to a computer. \n\n2.2  Ordinary Traffic Light Color Experiment \n\nFive subjects were instructed to slow down on yellow lights, stop for red lights, and go for \ngreen lights.  These are normal traffic  light colors.  Subjects were allowed  to  drive in  the \nenvironment before the experiment to get used to driving in  VR. \n\nIn  order to  make slow down lights  more frequent, all  stoplights turned  to  the slow down \ncolor  when  subjects  were  further  than  30 meters  aways  from  them.  When  the  subject \ndrove closer than 30 meters the light then turned to either the go or stop color with equal \nprobability.  The rest of the light sequence followed normal stoplights with  the stop light \nturning to the go light after 3 seconds and the go light not changing. \n\nWe calculated the grand averages over red,  green,  and yellow light trials (see Figure 2a). \nEpochs affected  by  artifact were ignored  in  the averages  in  order  to  make sure  that any \nexisting movements were not causing a P3-like signal.  Results show that a  P3  EP occurs \nfor both red and green lights.  Back averaging from the green/red lights to the yellow light \nshows the existence of a CNV starting at approximately 2 seconds before the light changes \nto red or green. \n\n\f6 \n\nJ.  D.  Bayliss and D. H  Ballard \n\nStop Light \n\nGo Light \n\nSlow Down Light \n\n-5 uv \n\n~ 1\\ \n\ni \n\" \n\n.~'\"\\ \n\n\\, \n\n.v'l,,/AI \n\\ \n\\ \nI \nI \n! \n\\ \nI \n\\ \nI \nI \n\\ \ni :  \nI) \n\n+lOuv \n-8 uv \n\n\\\"~'~ :\"' \n\n; \n\n~ \n\n,: .~ \n\n\" \n\n\".; \n\n, \n\n,~t r/'\" \n\n'\\  f \nI\".j\" \n\n\"\\ ..... \\ \n\n\u2022 \n\nA \n, \n\nf \nI \nf...j Vv \n\n.E bO \n;J \nu \n~ \nb)  ~ \n> \n\nt) \n\n'::1 j \n\n'-lOOms \n\nt<iooms' \n\n'-lOOms \n\nlOOOms I \n\nh'-3~000ii1S;;:::::::::==::;;2;;OO~ms~1  + 12 uv \n\nFigure 2:  a)  Grand averages  for  the red  stop,  green  go,  and  yellow  slow down  lights.  b)  Grand \naverages for  the yellow stop,  red  go,  and green  slow down  lights.  All  slow down  lights  have been \nback-averaged from the occurrence of the go/stop light in order to show the existence of a CNY. \n\n2.3  Alternative Traffic Light Colors \n\nThe P3  is related to  task relevance and should not be related to color, but color needed to \nbe disambiguated  as  the  source of the P3  in  the experiment.  We  had  two  subjects slow \ndown at green lights, stop at yellow lights, and go at red lights.  In order to get used to this \ncombination of colors, subjects were allowed to drive in the town before the experiment. \n\nThe grand averages for each light color were calculated in the same manner as the averages \nabove and are shown in Figure 2b.  As expected, a P3 signal existed for the stop condition \nand a CNV for the slow down condition.  The go condition P3  was much noisier for these \ntwo subjects, although a slight P3-like signal is still visible. \n\n3  Single Trial Recognition Results \n\nWhile averages show the existence of the P3 EP at red stop lights and the absence of such \nat yellow slow down lights, we needed to discover if the signal was clean enough for single \ntrial  recognition  as  the quick feedback  needed  by  a  BCI  depends  on  quick recognition. \nWhile there were three light conditions to recognize, there were only two distinct kinds of \nevoked potentials. We chose to recognize the difference between the P3 and the CNV since \ntheir averages are  very  different.  Recognizing the  difference between two  kinds  of EPs \ngives us the ability to use a BCI in any task that can be performed using a series of binary \ndecisions.  We  tried three methods for classification of the P3 EP: correlation, independent \ncomponent analysis (ICA), and a robust Kalman filter. \n\nApproximately, 90 slow down yellow light and 45  stop red light trials from  each  subject \nwere classified.  The reason we allowed a yellow light bias to enter recognition is  because \nthe yellow light currently represents an  unimportant event in  the environment.  In  a real \nBCI unimportant events are  likely  to  occur more than  user-directed actions,  making  this \nbias justifiable. \n\n\fRecognizing Evoked Potentials in a Virtual Environment \n\n7 \n\nTable 1: Recognition Results (p < 0.01) \n\nCorrelation %Correct \n\nSubjects  Red  Yel \n51 \nS1 \nS2 \n63 \n56 \nS3 \nS4 \n60 \nS5 \n66 \n\n81 \n95 \n89 \n81 \n63 \n\nTotal \n64 \n73 \n66 \n67 \n65 \n\nICA %Correct \nRed  Yel  Total  Red  Yel \n86 \n76 \n94 \n86 \n72 \n85 \n91 \n73 \n92 \n65 \n\n55 \n82 \n74 \n65 \n78 \n\nRobust Kalman Filter %Correct \n\n77 \n88 \n87 \n69 \n79 \n\n77 \n87 \n82 \n71 \n74 \n\nTotal \n77 \n90 \n81 \n82 \n87 \n\nTable 2:  Recognition Results for Return Subjects \n\nRobust K-Filter % Correct \n\nSubjects  Red  Yel \n90 \nS4 \nS5 \n87 \n\n73 \n67 \n\nTotal \n85 \n80 \n\nAs expected,  the  data obtained  while  driving  contained  artifacts,  but  in  an  on-line  BCI \nthese artifacts must be reduced in order to make sure that what the recognition algorithm is \nrecognizing is  not an artifact such as eye movement.  In order to reduce these artifacts, we \nperformed the on-line linear regression technique described in  [Semlitsch et aI. , 1986]  in \norder to subtract a combination of eye and head movement artifact. \n\nIn order to create a baseline from which to compare the performance of other algorithms, \nwe  calculated the correlation of all  sample  trials  with  the  red  and yellow  light averages \nfrom each subject's maximal P3 electrode site using the following formula: \n\ncorrelation = \n\n(sample * aveT)/(11  sample II  * II  ave II) \n\n(1) \n\nwhere sample and ave are both  1  x  500 vectors representing the trial epochs and light \naverages (respectively).  We used the whole trial  epoch for recognition because it yielded \nbetter recognition than just the time area around the P3. If the highest correlation of a trial \nepoch with the red and yellow averages was greater than 0.0, then the signal was classified \nas  that type of signal.  If both averages correlated negatively with the single trial, then  the \ntrial  was counted as  a  yellow  light signal.  As can be seen in  Table  1,  the correct signal \nidentification of red lights was extremely high while the yellow light identification pulled \nthe results down.  This may be explained by the greater variance of the yellow light epochs. \nCorrelations in general were poor with typical correlations around 0.25. \nICA has successfully  been  used  in order to  minimize artifacts  in  EEG data [Jung et at. , \n1997; Vigario,  1997] and has also proven useful in separating P3 component data from an \naveraged waveform [Makeig et aI.,  1997].  The next experiment used ICA in  order to  try \nto separate the background EEG signal from the P3 signal.  Independent component anal(cid:173)\nysis (lCA) assumes that n EEG data channels x  are a linear combination of n  statistically \nindependent signals s : \n\nwhere x  and s are n x  1 vectors. We used the matlab package mentioned in [Makeig et aI. , \n1997] with default learning values, which finds a matrix W  by stochastic gradient descent. \n\nx= As \n\n(2) \n\n\f8 \n\nJ  D.  Bayliss and D.  H.  Ballard \n\nThis  matrix W  performs component separation.  All  data was  sphered in order to  speed \nconvergence time. \nAfter training the W  matrix, the source channel showing the closest P3-like signal (using \ncorrelation with the average) for the red light average data was chosen as  the signal with \nwhich  to  correlate  individual  epochs.  The  trained  W  matrix  was  also  used  to  find  the \nsources of the yellow light average. The red and yellow light responses were then correlated \nwith individual epoch sources in the manner of the first experiment. \n\nThe third  experiment used  the robust Kalman filter  framework formulated  by  Rao  [Rao, \n1998].  The Kalman filter assumes a linear model similar to the one ofICA in equation 2, \nbut assumes  the EEG output x  is  the  observable output of a  generative or  measurement \nmatrix A  and an internal state vector s of Gaussian sources.  The output may also have an \nadditional noise component n, a Gaussian stochastic noise process with  mean zero and a \ncovariance matrix given by ~ = E[nn Tj, leading to the model expression: x = As + n. In \norder to find the most optimal value of s, a weighted least-squares criterion is formulated: \n\nwhere s follows a Gaussian distribution with mean s and covariance M.  Minimizing this \ncriterion by setting  ~; =  0 and using the substitution N  =  (AT~-lU + M-1)-1 yields \nthe  Kalman filter  equation,  which is  basically equal to  the  old estimate plus the  Kalman \ngain times the residual error. \n\n(3) \n\n(4) \n\nIn  an  analogous  manner,  the  measurement  matrix  A  may  be estimated  (learned)  if one \nassumes the physical relationships encoded by the measurement matrix are relatively stable. \nThe learning rule for  the  measurement matrix may  be derived in  a manner similar to  the \nrule for the internal state vector.  In addition, a decay term is often needed in order to avoid \noverfitting the data set.  See [Rao,  1998] for details. \nIn  our experiments both  the  internal  state  matrix s  and  the  measurement matrix  A  were \nlearned by training them on the average red light signal and the average yellow light signal. \nThe signal is measured from the start of the trial which is known since it is triggered by the \nlight change.  We  used a Kalman gain of 0.6 and a decay of 0.3.  After training, the signal \nestimate for each epoch is  correlated with the red and yellow light signal estimates in  the \nmanner of experiment 1.  We  made the Kalman filter statistically robust by ignoring parts \nof the EEG signal that fell outside a standard deviation of 1.0 from the training signals. \n\nThe overall recognition results in Table 1 suggest that both the robust Kalman filter and ICA \nhave a statistically significant advantage over correlation (p  < 0.01).  The robust Kalman \nfilter has a very small advantage over ICA (not statistically significant). \n\nIn order to  look at  the reliability  of the  best algorithm  and  its  ability  to  be used  on-line \ntwo of the Subjects (S4 and SS) returned for another VR driving session. In these sessions \nthe brakes of the driving simulator were controlled by the robust Kalman filter recognition \nalgorithm for red stop and yellow slow down lights. Green lights were ignored. The results \nof this  session  using  the  Robust  Kalman  Filter  trained  on  the  first  session  are shown  in \nTable 2.  The recognition numbers for red and yellow lights between the two sessions were \ncompared using correlation.  Red light scores between the sessions correlated fairly  highly \n- 0.82 for S4  and 0.69 for SS.  The yellow light scores between sessions correlated poorly \nwith both S4 and SS at approximately -0.1. This indicates that the yellow light epochs tend \nto correlate poorly with each other due to the lack of a large component such as the P3 to \ntie them together. \n\n\fRecognizing Evoked Potentials in a  Virtual Environment \n\n9 \n\n4  Future Work \n\nThis paper showed the viability of recognizing the P3 EP in a VR environment.  We  plan \nto  allow  the P3  EP to  propel the  user through alternate virtual rooms through the  use of \nvarious binary decisions.  In order to  improve recognition for the BCI we need to  experi(cid:173)\nment with  a wider and  more complex variety of recognition algorithms.  Our most recent \nwork has shown a dependence between the human computer interface used in the BCI and \nrecognition.  We  would like to explore this dependence in  order to improve recognition as \nmuch as possible. \n\nReferences \n\n[Bayliss and Ballard, 1998)  lD. Bayliss and  D.H. Ballard,  ''The Effects of Eye Tracking in a VR \nHelmet on EEG Recording,\"  TR 685,  University of Rochester National Resource Laboratory for \nthe Study of Brain and Behavior, May 1998. \n\n[Chapman and Bragdon, 1964)  R.M.  Chapman and H.R.  Bragdon,  \"Evoked responses to numerical \n\nand non-numerical visual stimuli while problem solving.,\"  Nature, 203: 1155-1157, 1964. \n\n[Farwell and Donchin,  1988)  L. A. Farwell and E. Donchin, \"Talking off the top of your head:  toward \na mental prosthesis utilizing event-related brain potentials,\"  Electroenceph.  Clin. Neurophysiol., \npages 510-523,  1988. \n\n[Jung et al., 1997)  1'.P. Jung, C. Humphries,1'.  Lee, S. Makeig, M.J. McKeown,  Y.  lragui, and 1'.l \nto \n\nSejnowski,  \"Extended ICA  Removes  Artifacts  from  Electroencephalographic  Recordings,\" \nAppear in Advances in Neural Information Processing Systems, 10, 1997. \n\n[Makeig et al.,  1997)  S.  Makeig,  1'.  Jung,  A.J.  Bell,  D. Ghahremani,  and 1'.J.  Sejnowski,  \"Blind \nSeparation  of Auditory  Event-related  Brain  Responses  into  Independent  Components,\"  Proc. \nNat'l Acad. Sci. USA , 94:10979-10984,  1997. \n\n[McFarland et al., 1993)  D.l McFarland, G.w. Neat, R.F.  Read, and J.R. Wolpaw,  \"An EEG-based \n\nmethod for graded cursor control,\"  Psychobiology, 21(1):77-81,  1993. \n\n[Pfurtscheller et al. , 1996)  G.  Pfurtscheller,  D.  Flotzinger,  M.  Pregenzer,  J.  Wolpaw,  and  D.  Mc(cid:173)\n\nFarland,  \"EEG-based Brain Computer Interface (BCI),\"  Medical Progress through Technology, \n21:111-121 , 1996. \n\n[Polich,  1998]  J. Polich,  \"P300 Clinical Utility  and Control of Variability,\"  J.  of Clinical Neuro(cid:173)\n\nphYSiology,  15(1): 14-33, 1998. \n\n[Rao,  1998]  R.  P.N. Rao,  \"Visual Attention during Recognition,\"  Advances in Neural Information \n\nProcessing Systems,  10, 1998. \n\n[Rosvold et al.,  1956]  H.E. Rosvold, A.F.  Mirsky, I. Sarason, E.D. Bransome Jr., and L.H. Beck,  \"A \n\nContinuous Performance Test of Brain Damage,\" 1.  Consult.  Psychol., 20,  1956. \n\n[SemIitschetal., 1986)  H.Y.  SemIitsch,  P. Anderer,  P  Schuster,  and O.  Presslich,  \"A  solution  for \nreliable and valid reduction of ocular artifacts applied to the P300 ERP;'  Psychophys., 23:695-\n703,1986. \n\n[Sutton et al.,  1965)  S. Sutton,  M. Braren, J. Zublin,  and  E.  John,  \"Evoked potential correlates of \n\nstimulus uncertainty,\"  Science,  150: 1187-1188,  1965. \n\n[Vaughn et al., 1996)  1'.M. Vaughn,  J.R.  Wolpaw,  and  E.  Donchin,  \"EEG-Based Communication: \n\nProspects and Problems,\"  IEEE Trans. on Rehabilitation Engineering, 4(4):425-430,  1996. \n\n[Vigario,  1997)  R.  Vigario,  \"Extraction of ocular artifacts from  eeg using independent component \n\nanalysis,\"  Electroenceph.  Clin.  Neurophysiol.,  103:395-404, 1997. \n\n\f", "award": [], "sourceid": 1683, "authors": [{"given_name": "Jessica", "family_name": "Bayliss", "institution": null}, {"given_name": "Dana", "family_name": "Ballard", "institution": null}]}