{"title": "Analyzing and Visualizing Single-Trial Event-Related Potentials", "book": "Advances in Neural Information Processing Systems", "page_first": 118, "page_last": 124, "abstract": null, "full_text": "Analyzing and  Visualizing  Single-Trial \n\nEvent-Related Potentials \n\nTzyy-Ping Jung1,2,  Scott  Makeig2,3,  Marissa Westerfield2 \n\nJeanne Townsend2,  Eric  Courchesne2,  Terrence J.  Sejnowskp,2 \n\n1 Howard  Hughes  Medical Institute and Computational Neurobiology  Laboratory \n\nThe Salk Institute,  P.O.  Box 85800,  San  Diego,  CA  92186-5800 \n\n{jung,scott,terry}~salk.edu \n\n2University  of California, San  Diego,  La Jolla, CA  92093 \n\n3Naval  Health  Research  Center,  P.O.  Box 85122,  San  Diego,  CA  92186-5122 \n\nAbstract \n\nEvent-related  potentials (ERPs),  are portions  of electroencephalo(cid:173)\ngraphic  (EEG)  recordings  that  are  both  time- and  phase-locked \nto  experimental  events.  ERPs  are  usually  averaged  to  increase \ntheir  signal/noise  ratio  relative  to  non-phase  locked  EEG  activ(cid:173)\nity,  regardless  of the  fact  that  response  activity  in  single  epochs \nmay vary  widely  in  time course  and scalp distribution.  This study \napplies a linear decomposition tool, Independent Component Anal(cid:173)\nysis  (ICA)  [1],  to  multichannel single-trial  EEG  records  to  derive \nspatial filters  that  decompose  single-trial  EEG epochs  into  a  sum \nof temporally  independent  and spatially fixed  components  arising \nfrom  distinct  or  overlapping  brain  or  extra-brain  networks.  Our \nresults  on  normal  and  autistic  subjects  show  that  ICA  can  sep(cid:173)\narate  artifactual,  stimulus-locked,  response-locked,  and.  non-event \nrelated  background  EEG  activities  into separate  components,  al(cid:173)\nlowing ( 1) removal of pervasive artifacts of all types from single-trial \nEEG records,  and (2) identification of both stimulus- and response(cid:173)\nlocked EEG components.  Second, this study proposes a new visual(cid:173)\nization tool, the 'ERP image', for investigating variability in laten(cid:173)\ncies  and amplitudes of event-evoked responses  in spontaneous EEG \nor  MEG  records.  We show  that sorting single-trial ERP epochs  in \norder  of reaction  time  and  plotting  the  potentials  in  2-D  clearly \nreveals  underlying  patterns  of response  variability  linked  to  per(cid:173)\nformance.  These  analysis  and  visualization  tools  appear  broadly \napplicable to electrophyiological research  on both normal and clin(cid:173)\nical  populations. \n\n\fAnalyzing and Visualizing Single-Trial Event-Related Potentials \n\n119 \n\n1 \n\nIntroduction \n\nScalp-recorded  event-related  potentials  (ERPs)  are voltage  changes  in the ongoing \nelectroencephalogram  (EEG)  that  are  both  time- and  phase-locked  to some exper(cid:173)\nimental  events.  These  field  potentials  are  usually  averaged  to  increase  their  sig(cid:173)\nnal/noise ratio relative  to  artifacts and  other non-phase  locked  EEG  activity.  The \naveraging  method  disregards  the  fact  that  in  single epochs  response  activity  may \nvary  widely  in  both  time  course  and  scalp  distribution.  These  differences  are  in \npart attributed  to different  strategies employed by  subjects for  processing  different \nstimuli, to  differences  in  expectation,  attention , and  arousal occurring  in  different \ntrials,  and/or  to  variations  in  alertness  and  fatigue  [2 ,  3].  Single-trial  analysis, \non  the  other  hand,  can  avoid  problems  due  to  time  and/or  phase  shifts  and  can \npotentially  reveal  much  richer  information  about  event-related  brain  dynamics in \nendogenous  ERPs,  but suffers  from  pervasive  artifacts  associated  with  blinks,  eye(cid:173)\nmovements,  and  muscle noise,  and  poor  signal-to-noise  ratio  arising from  the fact \nthat non-phase locked background EEG activities often are larger than phase-locked \nresponse  components. \n\nWe  present  here  new  methods for  analyzing  and  visualizing  multichannel unaver(cid:173)\naged  single-trial  ERP  records  that  alleviate  these  problems.  First,  multi-channel \nEEG epochs were analyzed using Independent  Component Analysis (ICA), a signal \nprocessing  technique  that can  decompose multichannel complex data into spatially \nfixed  and temporally independent  components.  Next,  a  new  visualization tool, the \n' ERP image', is introduced for visualizing relations between single-trial ERP records \nand their contributions to the  ERP average.  To form  an  ERP image, the recorded \npotentials at one  channel  are  plotted  as  parallel lines  and  single-trial ERP  epochs \nare sorted  in  order  of reaction  time.  ICA ,  applied  to  the single-trial  EEG  records \nfrom normal and autistic subjects in a visual selective attention experiment , derived \ncomponents whose dynamics were  affected by stimulus presentations and/or subject \nresponses  in  distinct  ways.  We  demonstrate,  through  analysis of two sample data \nsets,  the  power  of the  proposed  analysis  and  visualization  tools  for  increasing  the \namount and quality of information about event-related  brain  dynamics that can be \nderived from single-trial EEG  data. \n\n2 \n\nIndependent  Component  Analysis of EEG data \n\nBell and Sejnowski [5]  have proposed a simple neural network algorithm that blindly \nseparates  mixtures, x, of independent sources, s , using infomax. They showed that \nmaximizing the joint entropy,  H(y),  of the output of a  neural processor  minimizes \nthe  mutual information among the output  components,  Yi  =  g( Ui),  where  g( ud  is \nan invertible  bounded  nonlinearity and u  = Wx,  a  version  of the original sources, \ns , identical save for  scaling and permutation .  Lee  et al.  [1]  generalized  the infomax \nalgorithm  to  perform  blind  source  separation  on  linear  mixtures  of sources  with \neither  sub- or super-Gaussian  distributions.  Please see  [5,  1]  for  details  regarding \nthe algorithms. \n\nICA  is  suitable for  performing  blind source separation  on  EEG  data because:  (1) \nit  is  plausible that  EEG  data recorded  at  multiple scalp sensors  are linear sums of \ntemporally independent  components arising  from  spatially fixed,  distinct  or  over(cid:173)\nlapping  brain  or  extra-brain  networks,  and,  (2)  spatial  smearing  of EEG  data by \nvolume  conduction  does  not  involve significant  time  delays!.  In  single-trial  EEG \nanalysis,  the rows  of the input  matrix x  are the  EEG  signals recorded  at  different \nelectrodes,  while  the  columns are  measurements recorded  at  different  time points. \n\nlSee [4]  for  details  regarding  lCA assumptions  underlying  EEG  analysis. \n\n\f120 \n\nT.-P Jung et al. \n\nSingle-trial  EAPs at Cz \n\nOrdered by AT \n\nWith 20-trlal smoothing \n\n25 \n\n20 \n\n15 \n\n10 \n\n5 \n\no \n\n-5 \n\n-10 \n\n-15 \n\n- 20 \n\n- 25 \n\n/.LV \n\n-100  100  300  500  700  900 \n\nTime (msec) \n\nFigure  1:  ERP  images. \n(left  panel)  Single-trial  ERPs  recorded  at  a  central  electrode \n(Cz)  and  time-locked  to  onsets  of  visual  target  stimuli  (vertical  left  line),  plotted  with \nsubject  reaction  times  (thick  black  line).  (middle  panel) The  390  single  trials  were  then \nsorted  (bottom  to  top)  in  order  of  increasing  reaction  time. \n(right  panel)  To  increase \nsignal-to-noise  ratio  and  minimize  EEG  signals  not  both  time- and  phase-locked  to  the \nexperimental  events,  the  trials  were  averaged  vertically  using  a  30-trial  moving  window \nadvanced in  one-trial  increments. \n\nThe  rows  of the  independent  output  data  matrix  u  =  Wx  are  time  courses  of \nactivation  of the  lCA  components,  and  the  columns  of the  inverse  matrix,  W-l , \ngive the  projection  strengths  of the  respective  components onto the scalp  sensors. \nThe scalp topographies of the components provide evidence as to their physiological \norigin  (e.g., eye activity should  project  mainly to frontal sites) .  EEG  signals of in(cid:173)\nterest  (e.g. , event-related  brain signals) can then be obtained by  projecting selected \nlCA  components  back  onto  the  scalp  as  x' = (W)-lu',  where  u'  is  the  matrix of \nactivation waveforms, u ,  with  rows  representing  activations of \"irrelevant\"  sources \nset  to zero. \n\n3  Methods and  Materials \n\nEEG data were recorded at 29 scalp electrodes and 2 EOG placements from 2 normal \nand  1  autistic subjects  who participated in a  2-hr visual selected  attention  task in \nwhich  they  were  instructed  to  attend  to  circles  flashed  in  random  order  at  one  of \nfive  locations laterally arrayed 0.8 cm above a  central fixation point.  Locations were \noutlined by five evenly spaced  1.6-cm blue squares displayed on a  black background \nat  visual  angles  of \u00b12.7 deg  and  \u00b15.5 deg  from  fixation.  Attended  locations  were \nhighlighted through entire  90-sec  experimental blocks.  Subjects  were  instructed  to \nmaintain fixation on the central cross and press a button each time they saw a  circle \nin  the attended  location  (see  [6]  for  details). \n\n4  Results \n\nThe  lCA  algorithm was  applied separately  to concatenated  31-channel single-trial \nEEG  records  from two  normal and one autistic subjects.  The  derived  independent \ncomponents  had  a  variety  of distinct  relations  to  task  events.  Some  were  clearly \ntime-locked  to  stimuli  presentations ,  while  others  were  time-locked  to  subject  re-\n\n\fAnalyzing and Visualizing Single-Trial Event-Related Potentials \n\n121 \n\nsponses.  Still others captured spontaneous  EEG  activity together  with blinks, eye(cid:173)\nmovements,  and  muscle  artifacts,  while others  accounted  for  oscillatory and  other \nbackground  EEG  phenomena. \n\n4.1  ERP image \n\nTo investigate variability in the latencies and amplitudes of event-evoked  responses \nin spontaneous EEG, we here introduce a new visualization tool, the ERP image.  An \nexample shown in Figure 1 (left paneQ  plots 390 single-trial ERP epochs time-locked \nto onsets oftarget stimuli ( vertical left line) and recorded at a central electrode (Cz) \nfrom  a  normal subject.  Each  horizontal  trace  represents  a  I-sec  single-trial  ERP \nrecord  whose  potential  variations  are  plotted  in  different  colors.  The  thick  line \nplots  the  subject  reaction  times  (RT)  in  successive  trials.  Note  the  trial-to-trial \nfluctuations  in  ERP  latency  and  reaction  time.  The  ERP  average  of these  trials \nis  plotted  in  the  bottom of the  panel.  Next,  the  single  trials  were  sorted  in  order \nof increasing  reaction  time  (Fig.  1  middle  paneQ,  and  were  then  smoothed with  a \n30-trial  moving average  (right  paneQ.  Note  that,  in  all  but  the  longest-RT  trials, \nthe  early  positive  feature  (P2)  is  time-locked  to  stimulus onset  (i.e.  is  stimulus(cid:173)\nlocked),  and  that  the  P3  feature  follows  RT  in  nearly  all  trials  (i.e.  is  response(cid:173)\nlocked).  ERP  image  plots  allow  visualization  of relations  between  event-related \nEEG  trials  and  single-trial  contributions  to  their  ERP  averaged.  They  disclose  a \ntight link between the amplitudes and latencies of individual event-related responses \nand subject  behavior. \n\n4.2  Removing blink and eye-movement artifacts from  EEG  records \n\nAutistic  subjects  tend  to  blink  more  frequently  than  normal  subjects  [8]. \nICA, \napplied  to  this  data  set  in  which  about  50%  of the  trials  were  contaminated  by \nblinks,  successfully  isolated  blink  artifacts  to  a  single  component  (Fig.  2A,  left) \nwhose  contributions  could  be  removed  from  the  EEG  records  by  subtracting  out \nthe  component  projection  [7].  Though  the subjects  were  instructed  to  fixate  dur(cid:173)\ning  each  90-sec  blocks,  it  has  been  suspected,  though  poorly  documented,  that \ntheir eyes  tended  to  drift  towards  target stimuli presented  at peripheral  locations. \nHere,  a second  ICA component accounted for these small horizontal eye-movements \n(Fig. 2B,  right).  Fig.  2B  (5 traces)  also shows separate ERP averages  (at periocular \nsite EOG2) of responses to targets presented at the five  different attended locations. \nThe size  of the  prominent eye  movement-related component  is  proportional to the \nangle  between  the  stimulus  location  and  the  fixation  point.  Figure  2C  shows  the \naveraged ERPs at the same site in response  to stimuli presented at the five  different \nattended locations, before (faint traces) and after (solid traces) artifact removal.  Af(cid:173)\nter  artifact correction,  the averaged  ERPs to stimuli presented  at the five  different \nlocations were  independent  of stimulus location. \n\n4.3  Extracting event-related brain activity from  EEG  Records \n\nIn  these  data, ICA  also separated stimulus-locked, response-locked,  and non-phase \nlocked background EEG activities into different  independent components.  Numbers \nof components in each class varied across subjects.  Figure 3A shows the projections \nof the subgroups of ICA components accounting primarily for (left) stimulus-locked, \n(middle)  response-locked,  and (right) remaining non-phase locked background EEG \nactivity  at site  P03.  Notice  that,  (1)  both the response  latencies and  active dura(cid:173)\ntions of the early stimulus-locked PI and Nl components were very  stable in nearly \nall trials,  (2)  the peak  of the later  P3  component covaried  with  reaction  time, and \n(3) the projections of ICA components accounting for  non-phase locked background \nEEG  activity  contributed  very  little  to  the  averaged  ERP  (right  panel,  bottom \n\n\f122 \n\n(A) \n\nCofT'90nent 1 \n\n(8) \n\nCOfT'90nent 2 \n\nT.-P  lung et al. \n\nII \n\nt=t=\u00b1:::h:h;; \n\n900  0 \n\n900  0 \n\n900 \n\n900  0 \n\n900  0 \n\no \n\n(C) \n\nleftmost \n\nrightmost \n\n~~~~~rt: \n\n900 \n\n900 \n\n900 \n\n900 \n\no \n\n0 \n\n0 \n\n0 \n\nTime (msec) \n\n0 \n\n...L \nIFixation Point \n\n900 \n\nFigure 2:  (A) (left) Scalp topography and 5 consecutive I-sec epochs of the activation time \ncourse  of an leA component  counting  for  blink  artifacts in  641  single  trials  recorded  from \nan  adult autistic subject.  (B) The scalp topography of a second eye-movement  component \nand its averaged  activation  time courses in  response  to target stimuli  presented  at the five \ndifferent  attended  locations.  (C)  Averaged  ERPs  at  site  EOG2  to  targets  presented  at \neach of five  attended locations, before (faint traces) and after (solid traces) artifact removal. \n\ntrace).  These  results  indicate that  ICA  makes possible  the extraction  and separa(cid:173)\ntion of event-related  brain phenomena of all types from single-trial EEG  records. \n\n4.4  Re-aligning single-trial event-related potentials \n\nFigure 3B  (left  pane~ shows the raw  artifact-corrected single-trial ERP epochs (the \nsum  of the  data  in  Fig.  3A).  Response  latency  fluctuations  resulted  in  temporal \nsmearing  of  the  P3  feature  in  the  averaged  ERP  (bottom  left).  Realigning  the \nsingle-trial  ERP  epochs  to  the  median  reaction  time  sharpened  the  averaged  P3 \n(center panel,  P3'), but unfortunately made the early stimulus-locked activity out of \nphase and the early averaged  ERP thus absent  in  the first  200  msec.  Because  ICA \nseparated  stimulus-locked  and  response-locked  activity  into  different  independent \ncomponents, we could realign the time courses of the response-locked  P3' component \nto the median reaction time and project the adjusted data, along with the unaligned \ntime  courses  of stimulus-locked  components  (PI/NI),  back  onto  the scalp  sensors \n(right  pane~.  This  realignment  preserved  the  early  stimulus-locked  PI/NI  while \nsharpening  the  response-locked  P3.  The  method  minimized temporal smearing in \nthe  averaged  ERP arising from  performance fluctuations  (left  (3  right panels). \n\n4.5  Event-related oscillatory EEG activity \n\nICA,  applied  to multichannel single-trial  EEG  records,  can  also separate  multiple \noscillatory components even within a single frequency band.  For example, Figure 3C \nplots scalp topographies and ERP images of activations of two ICA components ac(cid:173)\ncounting for  alpha activity  in  target-response  epochs  from  a  normal subject.  Note \nthat the activity of the first  component (left  pane~ was augmented following stim(cid:173)\nulation,  while the activity of the second  component (middle  pane~ was  blocked  by \nthe subject response.  When the same spatial filter was applied to EEG records from \nanother session in which the subject was instructed to attend to but not to respond \n\n\fAnalyzing and Visualizing Single-Trial Event-Related Potentials \n\n123 \n\n(A) \n\nStimulus-locked Activity at P03  Response-locked Activity at P03 \n\nBackground ActMty at P03 \n\no \n\n0 \n\n-100  100  300  500  700  900 \n\nTime (msec) \n\n-100  100  300  sao  700  900 \n\nTime (msec) \n\n-100  100  300  sao  700  900 \n\nTime (msec) \n\n(8) \n\nSingle-trial ERPs at P03 \no \n\nRe-aligned ERPs \n\nRe--aligned ERPs \n\n100  300  500  700  900 \n\nTime (msec) \n\n(C) \n\nAlpha Component 1 \n\nAlpha component 2 \nMotor-response session \no \n\nAlpha component 2 \nNo-response session \n\n10 \n\n5 \n\no \n\n-5 \n\n-10 \n\n10 \n\no \n\n-10 \n\nFigure  3:  (A)  Projections  of ICA  components  at  site  P03  accounting,  respectively,  for \nstimulus-locked  (left) ,  response-locked  (middle),  and  non-phase  locked  background  EEG \nactivity  (right)  at  one  posterior  site,  P03.  (B)  (left)  Artifact-corrected  single-trial  ERP \nrecords  time-locked  to  stimulus  onsets  (left),  and  subject  responses  (center) .  Note  that \nthe  early  ERP  features  (PI ,  NI)  are  not  in  phase  in  the  response-locked  trials,  and  do \nnot  appear  in  the  response-locked  average  (center  bottom). \n(right)  Projections  of  the \nresponse-locked  components  were  aligned  to median  reaction  time  (355  ms)  and  summed \nwith stimulus-aligned  component  projections,  forming  an enhanced  stimulus-aligned  ERP \n(right  bottom).  (C)  ERP-image  plots  of  activations  of ICA  components  accounting  for \nalpha  activity  in  EEG  recorded  from  a  normal  subject .  The  alpha  activity  extracted  by \nthese  components  were  either  augmented  (left)  or  blocked  (middle)  by  subject  responses. \nWhen  the  spatial  filter  for  the  second  alpha  component  (middle)  was  applied  to  EEG \nrecords  from  another  session  in  which  the  subject  was  asked  only  to  'mentally  note'  the \noccurrence of target stimuli,  blocking  was  replaced  by  continued  phase-locking. \n\n\f124 \n\nT-P lung et al. \n\nto  target  stimuli,  this  alpha activity  was  not  blocked  (right  pane~ .  ICA  identifies \nspatially-overlapping patterns of coherent activity over the entire scalp rather than \nfocusing  on single scalp  channels or channel  pairs. \n\n5  Conclusions \n\nWe  have  developed  analytic  and  visualization  tools  for  analysis  of multichannel \nsingle-trial EEG  records.  Single-trial ERP  analysis  based  on  Independent  Compo(cid:173)\nnent  Analysis  allows  blind  separation  of multichannel  complex  EEG  data  into  a \nsum of temporally independent and spatially fixed  components.  ICA can effectively \nremove  eye  and  muscle  artifacts  without  altering  the  underlying  brain  activity  in \nthe  EEG  records.  lCA  can  also  be  used  to extract  event-related  brain  phenomena \nof all  types  from  EEG  records.  It  can  identify  spatially-overlapping  patterns  of \ncoherent  activity over  the entire scalp , and  can  be used  to realign the time courses \nof response-locked  components to prevent temporal smearing in the average  arising \nfrom  performance fluctuations.  ERP  images  make visible systematic  relations  be(cid:173)\ntween single-trial EEG or MEG records and experimental events, and their relations \nto averaged  ERPs.  ERP images can  also  be  used  to  display  relationships  between \nphase,  amplitude and  timing of event-related  EEG  components  time-locked  to  ei(cid:173)\nther stimuli or subject  responses.  The analysis  and  visualization tools proposed  in \nthis  study  dramatically increase  the  amount  and  quality of information on  event(cid:173)\nor response-related  brain signals that can  be extracted from  ERP data.  Both tools \nappear applicable to electrophyiological research on normal and clinical populations. \n\nReferences \n[1]  T.W.  Lee,  M.  Girolami  and  T.J. Sejnowski  (1999)  Independent  Component  Analysis \nusing  an  Extended  Infomax  Algorithm  for  Mixed  Sub-Gaussian  and  Super-Gaussian \nSources,  Neural  Computation, 11(2):  606-33. \n\n[2]  H.  Yabe,  F.  Satio &  Y.  Fukushima  (1993)  Median  Method for  Detecting  Endogenous \nEvent-related  Brain  Potentials,  Electroencephalog.  din.  Neurophysiolog.  87(6) :403-7. \n[3]  H.  Yabe,  F.  Satio &  Y.  Fukushima (1995)  Classification of Single-trial  ERP Sub-types: \nApplication  of  Globally  Optimal  Vector  Quantization  Using  Simulated  Annealing, \nElectroencephalog.  din.  Neurophysiolog.  94(4):288-97. \n\n[4]  S.  Makeig,  T-P  Jung,  A.J.  Bell,  D.  Ghahremani,  and  T.J.  Sejnowski  (1997)  Blind \nSeparation  of  Event-related  Brain  Responses  into  Independent  Components,  Proc. \nNatl.  Acad.  Sci.  USA,  USA,  94:10979-84. \n\n[5]  A.J.  Bell  &  T.J.  Sejnowski  (1995).  An  information-maximization  approach  to  blind \n\nseparation  and  blind  deconvolution,  Neural  Computation 7:1129-1159. \n\n[6]  S.  Makeig,  M.  Westerfield,  J.  Covington, T-P Jung,  J. Townsend, T.J. Sejnowski,  and \nE.  Courchesne  (in  press)  Functionally  independent  components  of  the  late  positive \nevent-related  potential in  a  visual  spatial  attention  paradigm,  J.  Neuroscience. \n\n[7]  Jung  T-P,  Humphries  C,  Lee  TW,  Makeig  S,  McKeown  MJ,  Iragui  V,  Sejnowski \nTJ (1998)  Extended  ICA  Removes  Artifacts from  Electroencephalographic  Data,  In: \nAdvances in  Neural Information  Processing Systems 10,  894-900. \n\n[8]  J.G.  Small  (1971)  Sensory  Evoked  Responses  of  Autistic  Children,  In: \n\nAutism,  224-39. \n\nInfantile \n\n\f", "award": [], "sourceid": 1574, "authors": [{"given_name": "Tzyy-Ping", "family_name": "Jung", "institution": null}, {"given_name": "Scott", "family_name": "Makeig", "institution": null}, {"given_name": "Marissa", "family_name": "Westerfield", "institution": null}, {"given_name": "Jeanne", "family_name": "Townsend", "institution": null}, {"given_name": "Eric", "family_name": "Courchesne", "institution": null}, {"given_name": "Terrence", "family_name": "Sejnowski", "institution": null}]}