{"title": "Predicting Action Content On-Line and in Real Time before Action Onset \u2013 an Intracranial Human Study", "book": "Advances in Neural Information Processing Systems", "page_first": 872, "page_last": 880, "abstract": null, "full_text": "Predicting Action Content On-Line and in\n\nReal Time before Action Onset \u2014 an\n\nIntracranial Human Study\n\nCalifornia Institute of Technology\n\nCalifornia Institute of Technology\n\nShengxuan Ye\n\nPasadena, CA\n\nsye@caltech.edu\n\nUri Maoz\n\nPasadena, CA\n\nurim@caltech.edu\n\nIan Ross\n\nHuntington Hospital\n\nPasadena, CA\n\nianrossmd@aol.com\n\nAdam Mamelak\n\nCedars-Sinai Medical Center\n\nLos Angeles, CA\n\nadam.mamelak@cshs.org\n\nChristof Koch\n\nCalifornia Institute of Technology\n\nAllen Institute for Brain Science\n\nPasadena, CA\n\nSeattle, WA\n\nkoch@klab.caltech.edu\n\nAbstract\n\nThe ability to predict action content from neural signals in real time before the ac-\ntion occurs has been long sought in the neuroscienti\ufb01c study of decision-making,\nagency and volition. On-line real-time (ORT) prediction is important for under-\nstanding the relation between neural correlates of decision-making and conscious,\nvoluntary action as well as for brain-machine interfaces. Here, epilepsy patients,\nimplanted with intracranial depth microelectrodes or subdural grid electrodes for\nclinical purposes, participated in a \u201cmatching-pennies\u201d game against an opponent.\nIn each trial, subjects were given a 5 s countdown, after which they had to raise\ntheir left or right hand immediately as the \u201cgo\u201d signal appeared on a computer\nscreen. They won a \ufb01xed amount of money if they raised a different hand than\ntheir opponent and lost that amount otherwise. The question we here studied was\nthe extent to which neural precursors of the subjects\u2019 decisions can be detected in\nintracranial local \ufb01eld potentials (LFP) prior to the onset of the action.\nWe found that combined low-frequency (0.1\u20135 Hz) LFP signals from 10 electrodes\nwere predictive of the intended left-/right-hand movements before the onset of the\ngo signal. Our ORT system predicted which hand the patient would raise 0.5 s\nbefore the go signal with 68\u00b13% accuracy in two patients. Based on these results,\nwe constructed an ORT system that tracked up to 30 electrodes simultaneously,\nand tested it on retrospective data from 7 patients. On average, we could predict\nthe correct hand choice in 83% of the trials, which rose to 92% if we let the system\ndrop 3/10 of the trials on which it was less con\ufb01dent. Our system demonstrates\u2014\nfor the \ufb01rst time\u2014the feasibility of accurately predicting a binary action on single\ntrials in real time for patients with intracranial recordings, well before the action\noccurs.\n\n1\n\n\f1\n\nIntroduction\n\nThe work of Benjamin Libet [1, 2] and others [3, 4] has challenged our intuitive notions of the rela-\ntion between decision making and conscious voluntary action. Using electrocorticography (EEG),\nthese experiments measured brain potentials from subjects that were instructed to \ufb02ex their wrist at a\ntime of their choice and note the position of a rotating dot on a clock when they felt the urge to move.\nThe results suggested that a slow cortical wave measured over motor areas\u2014termed \u201creadiness po-\ntential\u201d [5], and known to precede voluntary movement [6]\u2014begins a few hundred milliseconds be-\nfore the average reported time of the subjective \u2018urge\u2019 to move. This suggested that action onset and\ncontents could be decoded from preparatory motor signals in the brain before the subject becomes\naware of an intention to move and of the contents of the action. However, the readiness potential\nwas computed by averaging over 40 or more trials aligned to movement onset after the fact. More\nrecently, it was shown that action contents can be decoded using functional magnetic-resonance\nimaging (fMRI) several seconds before movement onset [7]. But, while done on a single-trial basis,\ndecoding the neural signals took place off-line, after the experiment was concluded, as the sluggish\nnature of fMRI hemodynamic signals precluded real-time analysis. Moreover, the above studies\nfocused on arbitrary and meaningless action\u2014purposelessly raising the left or right hand\u2014while\nwe wanted to investigate prediction of reasoned action in more realistic, everyday situations with\nconsequences for the subject.\nIntracranial recordings are good candidates for single-trial, ORT analysis of action onset and con-\ntents [8, 9], because of the tight temporal pairing of LFP to the underlying neuronal signals. More-\nover, such recordings are known to be cleaner and more robust, with signal-to-noise ratios up to\n100 times larger than surface recordings like EEG [10, 11]. We therefore took advantage of a rare\nopportunity to work with epilepsy patients implanted with intracranial electrodes for clinical pur-\nposes. Our ORT system (Fig. 1) predicts, with far above chance accuracy, which one of two future\nactions is about to occur on this one trial and feeds the prediction back to the experimenter, all\nbefore the onset of the go signal that triggers the patient\u2019s movement (see Experimental Methods).\nWe achieve relatively high prediction performance using only part of the data\u2014learning from brain\nactivity in past trials only (Fig. 2) to predict future ones (Fig. 3)\u2014while still running the analysis\nquickly enough to act upon the prediction before the subject moved.\n\n2 Experimental Methods\n\n2.1 Subjects\n\nSubjects in this experiment were 8 consenting intractable epilepsy patients that were implanted with\nintracranial electrodes as part of their presurgical clinical evaluation (ages 18\u201360, 3 males). They\nwere inpatients in the neuro-telemetry ward at the Cedars Sinai Medical Center or the Huntington\nMemorial Hospital, and are designated with CS or HMH after their patient numbers, respectively. Six\nof them\u2014P12CS, P15CS, P22CS and P29\u201331HMH were implanted with intracortical depth elec-\ntrodes targeting their bilateral anterior-cingulate cortex, amygdala, hippocampus and orbitofrontal\ncortex. These electrodes had eight 40 \u00b5m microwires at their tips, 7 for recording and 1 serving as\na local ground. Two patients, P15CS and P22CS, had additional microwires in the supplementary\nmotor area. We utilized the LFP recorded from the microwires in this study. Two other patients,\nP16CS and P19CS, were implanted with an 8\u00d78 subdural grid (64 electrodes) over parts of their\ntemporal and prefrontal dorsolateral cortices. The data of one patient\u2014P31HMH\u2014was excluded\nbecause microwire signals were too noisy for meaningful analysis. The institutional review boards\nof Cedars Sinai Medical Center, the Huntington Memorial Hospital and the California Institute of\nTechnology approved the experiments.\nDuring the experiment, the subject sat in a hospital bed in a semi-inclined \u201clounge chair\u201d position.\nThe stimulus/analysis computer (bottom left of Fig. 4) displaying the game screen (bottom right\ninset of Fig. 4) was positioned to be easily viewable for the subject. When playing against the\nexperimenter, the latter sat beside the bed. The response box was placed within easy reach of the\nsubject (Fig. 4).\n\n2\n\n\f2.2 Experiment Design\n\nAs part of our focus on purposeful, reasoned action, we had the subjects play a matching-pennies\ngame\u2014a 2-choice version of \u201crock paper scissors\u201d\u2014either against the experimenter or against a\ncomputer. The subjects pressed down a button with their left hand and another with their right on a\nresponse box. Then, in each trial, there was a 5 s countdown followed by a go signal, after which\nthey had to immediately lift one of their hands. It was agreed beforehand that the patient would win\nthe trial if she lifted a different hand than her opponent, and lose if she raised the same hand as her\nopponent. Both players started off with a \ufb01xed amount of money, $5, and in each trial $0.10 was\ndeducted from the loser and awarded to the winner. If a player lifted her hand before the go signal,\ndid not lift her hand within 500 ms of the go signal, or lifted no hand or both hands at the go signal\u2014\nan error trial\u2014she lost $0.10 without her opponent gaining any money. The subjects were shown the\ncountdown, the go signal, the overall score, and various instructions on a stimulus computer placed\nbefore them (Fig. 4). Each game consisted of 50 trials. If, at the end of the game, the subject had\nmore money than her opponent, she received that money in cash from the experimenter.\nBefore the experimental session began, the experimenter explained the rules of the game to the sub-\nject, and she could practice playing the game until she was familiar with it. Consequently, patients\nusually made only few errors during the games (<6% of the trials). Following the tutorial, the sub-\nject played 1\u20133 games against the computer and then once against the experimenter, depending on\ntheir availability and clinical circumstances. The \ufb01rst 2 games of P12CS were removed because\nthe subject tended to constantly raise the right hand regardless of winning or losing. Two patients,\nP15CS and P19CS, were tested in actual ORT conditions. In such sessions\u20143 games each\u2014the\nsubjects always played against the experimenter. These ORT games were different from the other\ngames in two respects. First, a computer screen was placed behind the patient, in a location where\nshe could not see it. Second, the experimenter was wearing earphones (Fig. 1,4). Half a second be-\nfore go-signal onset, an arrow pointing towards the hand that the system predicted the experimenter\nhad to raise to win the trial was displayed on that screen. Simultaneously, a monophonic tone was\nplayed in the experimenter\u2019s earphone ipsilateral to that hand. The experimenter then lifted that hand\nat the go signal (see Supplemental Movie).\n\nFigure 1: A schematic diagram of the on-line real-time (ORT) system. Neural signals \ufb02ow from\nthe patient through the Cheetah machine to the analysis/stimulus computer, which controls the input\nand output of the game and computes the prediction of the hand the patient would raise at the go\nsignal. It displays it on a screen behind the patient and informs the experimenter which hand to raise\nby playing a tone in his ipsilateral ear using earphones.\n\n3\n\nPatient with intracranial electrodesCheetah MachineCollect and save dataDown samplingAnalysis/stimulus machineFilteringBufferResult InterpretationDisplay/SoundExperimenterGame ScreenResponse Box1Gbps RouterTTL SignalAnalysisThe winner is Player 1PLAYER 1PLAYER 2SCORE 1SCORE 2/\f3 The real-time system\n\n3.1 Hardware and software overview\n\nNeural data from the intracranial electrodes were transferred to a recording system (Neuralynx,\nDigital Lynx), where it was collected and saved to the local Cheetah machine, down sampled\nfrom 32 kHz to 2 kHz and buffered. The data were then transferred, through a dedicated 1 Gbps\nlocal-area network, to the analysis/stimulus machine. This computer \ufb01rst band-pass-\ufb01ltered the\ndata to the 0.1\u20135 Hz range (delta and lower theta bands) using a second-order zero-lag elliptic\n\ufb01lter with an attenuation of 40 dB (cf. Figs. 2a and 2b). We found that this frequency range\u2014\ngenerally comparable to that of the readiness potential\u2014resulted in optimal prediction performance.\nIt then ran the analysis algorithm (see below) on the \ufb01ltered data. This computer also controlled\nthe game screen, displaying the names of the players, their current scores and various instructions.\nThe analysis/stimulus computer further\ncontrolled the response box, which con-\nsisted of 4 LED-lit buttons. The but-\ntons of the subject and her opponent\n\ufb02ashed red or blue whenever she or her\nopponent won, respectively. Addition-\nally, the analysis/stimulus computer sent\na unique transistor-transistor logic (TTL)\npulse whenever the game screen changed\nor a button was pressed on the response\nbox, which synchronized the timing of\nthese events with the LFP recordings.\nIn real-time game sessions,\nthe analy-\nsis/stimulus computer also displayed the\nappropriate arrow on the computer screen\nbehind the subject and played the tone\nto the appropriate ear of the experimenter\n0.5 s before go-signal onset (Figs. 1,4).\nThe analysis software was based on a\nmachine-learning algorithm that\ntrained\non past-trials data to predict the current\ntrial and is detailed below. The train-\ning phase included the \ufb01rst 70% of the\ntrials, with the prediction carried out on\nthe remaining 30% using the trained pa-\nrameters, together with an online weight-\ning system (see below). The system ex-\namined only neural activity, and had no\naccess to the subject\u2019s left/right-choice\nhistory. After \ufb01ltering all the training\ntrials (Fig. 2b),\nthe system found the\nmean and standard error over all leftward\nand rightward training trials, separately\n(Fig. 2c, left designated in red).\nIt then\nfound the electrodes and time windows\nwhere the left/right separation was high\n(Fig. 2d,e; see below), and trained the clas-\nsi\ufb01ers on these time windows (Fig. 2f\u2013g).\nThe best electrode/time-window/classi\ufb01er\n(ETC) combinations were then used to\npredict the current trial in the prediction\nphase (Fig. 3). The number of ETCs that\ncan be actively monitored is currently lim-\nited to 10 due to the computational power\nof the real-time system.\n\nFigure 2: The ORT-system\u2019s training phase. Left (in\nred) and right (in blue) raw signals (a) are low-pass \ufb01l-\ntered (b). Mean\u00b1standard errors of signals preceed-\ning left- and right-hand movments (c) are used to com-\npute a left/right separability index (d), from which time\nwindows with good separation are found (e). Seven\nclassi\ufb01ers are then applied to all the time windows (f)\nand the best electrode/time-window/classi\ufb01er combi-\nnations are selected (g) and used in the prediction phase\n(Fig. 3).\n\n4\n\nEl49\u2212T2El49\u2212T3Classifier Cf2Classifier Cf1Classifier Cf6...El49\u2212T1\u2212Cf1     El49\u2212T2\u2212Cf1El49\u2212T3\u2212Cf1El49\u2212T1\u2212Cf2     El49\u2212T2\u2212Cf249\u2212T3\u2212Cf2...El49\u2212T1\u2212Cf6     El49\u2212T2\u2212Cf6El49\u2212T3\u2212Cf6\u22125\u22124\u22123\u22122\u221210600800\u00b5V\u22125\u22124\u22123\u22122\u221210\u2212200\u22121000100\u00b5V\u22125\u22124\u22123\u22122\u221210\u2212200\u22121000100\u00b5V\u22125\u22124\u22123\u22122\u221210\u2212101\u22125\u22124\u22123\u22122\u221210\u2212101El49\u2212T1 Combination El49-T1-Cf2...Combination El49-T2-Cf2Combination El49-T2-Cf6Countdown to go signal at t=0 (seconds)(a)(b)(c)(d)(e)(f)(g)El\fFigure 3: The ORT-system\u2019s prediction phase. A new signal\u2014from 5 to 0.5 seconds before the\ngo signal\u2014is received in real time, and each electrode/time-window/classi\ufb01er combination (ETC)\nclassi\ufb01es it as resulting in left- or right-hand movement. These predictions are then compared to the\nactual hand movement, with the weights associated with ETCs that correctly (incorrectly) predicted\nincreasing (decreasing).\n\n3.2 Computing optimal left/right-separating time windows\n\nThe algorithm focused on \ufb01nding the time windows with the best left/right separation for the dif-\nferent recording electrodes over the training set (Fig. 2c\u2013e). That is, we wanted to predict whether\nthe signal aN (t) on trial N will result in a leftward or rightward movement\u2014i.e., whether the la-\nbel of the N th trial will be Lt or Rt, respectively. For each electrode, we looked at the N \u2212 1\nprevious trials a1(t), a2(t), . . . , aN\u22121(t), and their associated labels as l1, l2, . . . , lN\u22121. Now, let\nL(t) = {ai(t)| li = Lt}N\u22121\ni=1 be the set of previous leftward and\nrightward trials in the training set, respectively. Furthermore, let Lm(t) (Rm(t)) and Ls(t) (Rs(t))\nbe the mean and standard error of L(t) (R(t)), respectively. We can now de\ufb01ne the normalized\nrelative left/right separation for each electrode at time t (see Fig. 2d):\n\ni=1 and R(t) = {ai(t)| li = Rt}N\u22121\n\n\uf8f1\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f2\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f4\uf8f3\n\n\u03b4(t) =\n\n[Lm(t) \u2212 Ls(t)] \u2212 [Rm(t) + Rs(t)]\n\nLm(t) \u2212 Rm(t)\n\n\u2212 [Rm(t) \u2212 Rs(t)] \u2212 [Lm(t) + Ls(t)]\n\nRm(t) \u2212 Lm(t)\n\nif\n\nif\n\n[Lm(t) \u2212 Ls(t)] \u2212 [Rm(t) + Rs(t)] > 0\n\n[Rm(t) \u2212 Rs(t)] \u2212 [Lm(t) + Ls(t)] > 0\n\n0\n\notherwise\n\napart are combined into one. Then, for each time window from t1 to t2 we de\ufb01ne a =(cid:82) t2\n\nThus, \u03b4(t) > 0 (\u03b4(t) < 0) means that the leftward trials tend to be considerably higher (lower)\nthan rightward trials for that electrode at time t, while \u03b4(t) = 0 suggests no left/right separation at\ntime t. We de\ufb01ne a consecutive time period of |\u03b4(t)| > 0 for t < prediction time (the time before\nthe go signal when we want the system to output a prediction; -0.5 s for the ORT trials) as a time\nwindow (Fig. 2e). After all time windows are found for all electrodes, time windows less than M ms\n|\u03b4(t)|dt.\nWe then eliminate all time windows satisfying a < A. We found the values M = 200 ms and\nA = 4, 500 \u00b5V \u00b7 ms to be optimal for real-time analysis. This resulted in 20\u201330 time windows over\nall 64 electrodes that we monitored.\n\nt1\n\n5\n\nCombination El49\u2212T1\u2212Cf2Weight = 1&Combination El49\u2212T2\u2212Cf2Weight = 1Combination El49\u2212T2\u2212Cf6Weight = 1Trained classifiersL&LLLLRLRLPredicted resultLReal result==Adjust the weights\u00b5V\u22125\u22124\u22123\u22122\u221210\u2212200\u22121000100\fFigure 4: The experimental setup in the clinic. At 400 ms before the go signal, the patient and\nexperimenter are watching the game screen (inset on bottom right) on the analysis/stimulus computer\n(bottom left) and still pressing down the buttons of the response box. The realtime system already\ncomputed a prediction, and thus displays an arrow on the screen behind the patient and plays a tone\nin the experimenter\u2019s ear ipsilateral to the hand it predicts he should raise to beat the patient (see\nSupplemental Movie).\n\n3.3 Classi\ufb01ers selection and ETC determination\n\nWe used ensemble learning with 7 types of relatively simple binary classi\ufb01ers (due to real-time\nprocessing considerations) on every electrode\u2019s time windows (Fig. 2f). Classi\ufb01ers A to G would\nclassify aN (t) as Lt if:\n\n(cid:1) (cid:54)= sign(cid:0)aN,M\n(i) sign(cid:0)Rm,M\n(cid:1) = sign(cid:0)aN,M\n(ii) sign(cid:0)Rm,M\n(iii) sign(cid:0)Rm(t)(cid:1) (cid:54)= sign(cid:0)SN,M\n\n(A) De\ufb01ning aN,M , Lm,M and Rm,M as(cid:80) aN (t),(cid:80) Lm(t) and(cid:80) Rm(t) over time window M,\n(cid:1) = sign(cid:0)Lm,M\n(cid:1), or\n(cid:1) and(cid:12)(cid:12)Lm,M\n(cid:1) = sign(cid:0)Lm,M\n(cid:1) (cid:54)= sign(cid:0)Lm(t)(cid:1) and(cid:12)(cid:12)Lm,M\n(B) (cid:12)(cid:12)mean(cid:0)aN (t)(cid:1) \u2212 mean(cid:0)Lm(t)(cid:1)(cid:12)(cid:12) <(cid:12)(cid:12)mean(cid:0)aN (t)(cid:1) \u2212 mean(cid:0)Rm(t)(cid:1)(cid:12)(cid:12);\n(C) (cid:12)(cid:12)median(cid:0)aN (t)(cid:1) \u2212 median(cid:0)Lm(t)(cid:1)(cid:12)(cid:12) < (cid:12)(cid:12)median(cid:0)aN (t)(cid:1) \u2212 median(cid:0)Rm(t)(cid:1)(cid:12)(cid:12) over the time\n(D) (cid:12)(cid:12)aN (t) \u2212 Lm(t)(cid:12)(cid:12)L2 <(cid:12)(cid:12)aN (t) \u2212 Rm(t)(cid:12)(cid:12)L2 over the time window;\n\n(cid:12)(cid:12) >(cid:12)(cid:12)Rm,M\n(cid:12)(cid:12), or\n(cid:12)(cid:12);\n(cid:12)(cid:12) <(cid:12)(cid:12)Rm,M\n\nwindow;\n\n(E) aN (t) is convex/concave like Lm(t) while Rm(t) is concave/convex, respectively;\n(F) Linear support-vector machine (SVM) designates it as so; and\n(G) k-nearest neighbors (KNN) with Euclidean distance designates it as so.\n\nEach classi\ufb01er is optimized for certain types of features. To estimate how well its classi\ufb01cation\nwould generalize from the training to the test set, we trained and tested it using a 70/30 cross-\nvalidation procedure within the training set. We tested each classi\ufb01er on every time window of every\nelectrode, discarding those with accuracy <0.68, which left 12.0 \u00b1 1.6% of the original 232 \u00b1 18\nETCs, on average (\u00b1standard error). The training phase therefore ultimately output a set of S binary\nETC combinations (Fig. 2g) that were used in the prediction phase (Fig. 3).\n\n3.4 The prediction-phase weighting system\nIn the prediction phase, each of the overall S binary ETCs calculates a prediction, ci \u2208 {\u22121, 1} (for\nright and left, respectively), independently at the desired prediction time. All classi\ufb01ers are initially\n\n6\n\n$4.80P15CS$5.20Uri1\fgiven the same weight, w1 = w2 = \u00b7\u00b7\u00b7 = wS = 1. We then calculate \u03be =(cid:80)S\n\ni=1 wi \u00b7 ci and predict\nleft (right) if \u03be > d (\u03be < \u2212d), or declare it an undetermined trial if \u2212d < \u03be < d. Here d is the\ndrop-off threshold for the prediction. Thus the larger d is, the more con\ufb01dent the system needs to be\nto make a prediction, and the larger the proportion of trials on which the system abstains\u2014the drop-\noff rate. Weight wi associated with ETCi is increased (decreased) by 0.1 whenever ETCi predicts\nthe hand movement correctly (incorrectly). A constantly erring ETC would therefore be associated\nwith an increasingly small and then increasingly negative weight.\n\n3.5\n\nImplementation\n\nThe algorithm was implemented in MATLAB 2011a (MathWorks, Natick, MA) as well as in C++\non Visual Studio 2008 (Microsoft, Redmond, WA) for enhanced performance. The neural signals\nwere collected by the Digital Lynx S system using Cheetah 5.4.0 (Neuralynx, Redmond, WA). The\nsimulated-ORT system was also implemented in MATLAB 2011a. The simulated-ORT analyses\ncarried out in this paper used real patient data saved on the Digital Lynx system.\n\nFigure 5: Across-subjects average of the prediction accuracy of simulated-ORT versus time before\nthe go signal. The mean accuracies over time when the system predicts on every trial, is allowed\nto drop 19% or 30% of the trials, are depicted in blue, green and red, respectively (\u00b1standard error\nshaded). Values above the dashed horizontal line are signi\ufb01cant at p = 0.05.\n\n4 Results\n\nWe tested our prediction system in actual real time on 2 patients\u2014P15CS and P19CS (a depth\nand grid patient, respectively), with a prediction time of 0.5 s before the go signal (see Sup-\nplementary Movie). Because of computational limitations, the ORT system could only track 10\nelectrodes with just 1 ETC per electrode in real time. For P15CS, we achieved an accuracy of\n72\u00b12% (\u00b1standard error; accuracy = number of accurately predicted trials / [total number of tri-\nals - number of dropped trials]; p = 10\u22128, binomial test) without modifying the weights on-\nline during the prediction (see Section 3.4). For P19CS we did not run patient-speci\ufb01c train-\ning of the ORT system, and used parameter values that were good on average over previous pa-\ntients instead. The prediction accuracy was signi\ufb01cantly above chance 63\u00b12% (\u00b1standard er-\nror; p = 7\u00b7 10\u22124, binomial test). To understand how much we could improve our accuracy\nwith optimized hardware/software, we ran the simulated-ORT at various prediction times along\n\n7\n\nDrop rate:Go-signalonsetSignificant accuracy(p=0.05)\u22125\u22124.5\u22124\u22123.5\u22123\u22122.5\u22122\u22121.5\u22121\u22120.500.50.60.70.80.91 Time (s)Prediction accuracyNone0.180(cid:17)(cid:22)(cid:19) \fthe 5 s countdown leading to the go signal. We further tested 3 drop-off rates\u20140, 0.19 and\n0.30 (Fig. 5; drop-off rate = number of dropped trials / total number of trials; these resulted\nfrom 3 drop-off thresholds\u20140, 0.1 and 0.2\u2014respectively, see Section 3.4:). Running of\ufb02ine,\nwe were able to track 20\u201330 ETCs, which resulted in considerably higher accuracies (Figs. 5,6).\nAveraged over all subjects, the accu-\nracy rose from about 65% more than\n4 s before the go signal to 83\u201392%\nclose to go-signal onset, depending\non the allowed drop-off rate. In par-\nticular, we found that for a predic-\ntion time of 0.5 s before go-signal\nonset, we could achieve accuracies\nof 81\u00b15% and 90\u00b13% (\u00b1standard\nerror) for P15CS and P19CS, re-\nspectively, with no drop off (Fig. 6).\nWe also analyzed the weights that\nour weighting system assigned to the\ndifferent ETCs. We found that the\nempirical distribution of weights to\nETCs associated with classi\ufb01ers A to\nG was, on average: 0.15, 0.12, 0.16,\n0.22, 0.01, 0.26 and 0.07, respec-\ntively. This suggests that the linear\nSVM and L2-norm comparisons (of\naN to Lm and Rm) together make up\nnearly half of the overall weights at-\ntributed to the classi\ufb01ers, while the\ncurrent concave/convex measure is of\nlittle use as a classi\ufb01er.\n\nFigure 6: Simulated-ORT accuracy over time for individual\npatients with no drop off.\n\n5 Discussion\n\nWe constructed an ORT system that, based on intracranial recordings, predicted which hand a per-\nson would raise well before movement onset at accuracies much greater than chance in a com-\npetitive environment. We further tested this system off-line, which suggested that with optimized\nhardware/software, such action contents would be predictable in real time at relatively high accu-\nracies already several seconds before movement onset. Both our prediction accuracy and drop-off\nrates close to movement onset are superior to those achieved before movement onset with non-\ninvasive methods like EEG and fMRI [7, 12\u201314]. Importantly, our subjects played a matching pen-\nnies game\u2014a 2-choice version of rock-paper-scissors [15]\u2014to keep their task realistic, with minor\nthough real consequences, unlike the Libet-type paradigms whose outcome bears no consequences\nfor the subjects. It was suggested that accurate online, real-time prediction before movement onset\nis key to investigating the relation between the neural correlates of decisions, their awareness, and\nvoluntary action [16, 17]. Such prediction capabilities would facilitate many types of experiments\nthat are currently infeasible. For example, it would make it possible to study decision reversals on\na single-trial basis, or to test whether subjects can guess above chance which of their action con-\ntents are predictable from their current brain activity, potentially before having consciously made up\ntheir mind [16, 18]. Accurately decoding these preparatory motor signals may also result in earlier\nand improved classi\ufb01cation for brain-computer interfaces [13, 19, 20]. The work we present here\nsuggests that such ORT analysis might well be possible.\n\nAcknowledgements\n\nWe thank Ueli Rutishauser, Regan Blythe Towel, Liad Mudrik and Ralph Adolphs for meaningful\ndiscussions. This research was supported by the Ralph Schlaeger Charitable Foundation, Florida\nState University\u2019s \u201cBig Questions in Free Will\u201d initiative and the G. Harold & Leila Y. Mathers\nCharitable Foundation.\n\n8\n\n\u22125\u22124.5\u22124\u22123.5\u22123\u22122.5\u22122\u22121.5\u22121\u22120.500.40.50.60.70.80.91  P15CSP16CSP19CSP29HMHP30HMHP12CSAccuracyTime before go signal (at t=0) (seconds)Patients:P22CS\fReferences\n\n[1] B. Libet, C. Gleason, E. Wright, and D. Pearl. Time of conscious intention to act in relation to\nonset of cerebral activity (readiness-potential): The unconscious initiation of a freely voluntary\nact. Brain, 106:623, 1983.\n\n[2] B. Libet. Unconscious cerebral initiative and the role of conscious will in voluntary action.\n\nBehavioral and brain sciences, 8:529\u2013539, 1985.\n\n[3] P. Haggard and M. Eimer. On the relation between brain potentials and the awareness of\n\nvoluntary movements. Experimental Brain Research, 126:128\u2013133, 1999.\n\n[4] A. Sirigu, E. Daprati, S. Ciancia, P. Giraux, N. Nighoghossian, A. Posada, and P. Haggard.\nAltered awareness of voluntary action after damage to the parietal cortex. Nature Neuroscience,\n7:80\u201384, 2003.\n\n[5] H. Kornhuber and L. Deecke. Hirnpotenti\u00a8alanderungen bei Willk\u00a8urbewegungen und passiven\nBewegungen des Menschen: Bereitschaftspotential und reafferente Potentiale. P\ufb02\u00a8ugers Archiv\nEuropean Journal of Physiology, 284:1\u201317, 1965.\n\n[6] H. Shibasaki and M. Hallett. What is the Bereitschaftspotential? Clinical Neurophysiology,\n\n117:2341\u20132356, 2006.\n\n[7] C. Soon, M. Brass, H. Heinze, and J. Haynes. Unconscious determinants of free decisions in\n\nthe human brain. Nature Neuroscience, 11:543\u2013545, 2008.\n\n[8] I. Fried, R. Mukamel, and G. Kreiman. Internally generated preactivation of single neurons in\n\nhuman medial frontal cortex predicts volition. Neuron, 69:548\u2013562, 2011.\n\n[9] M. Cerf, N. Thiruvengadam, F. Mormann, A. Kraskov, R. Quian Quiorga, C. Koch, and\nI. Fried. On-line, voluntary control of human temporal lobe neurons. Nature, 467:1104\u20131108,\n2010.\n\n[10] T. Ball, M. Kern, I. Mutschler, A. Aertsen, and A. Schulze-Bonhage. Signal quality of simul-\n\ntaneously recorded invasive and non-invasive EEG. Neuroimage, 46:708\u2013716, 2009.\n\n[11] G. Schalk, J. Kubanek, K. Miller, N. Anderson, E. Leuthardt, J. Ojemann, D. Limbrick,\nD. Moran, L. Gerhardt, and J. Wolpaw. Decoding two-dimensional movement trajectories\nusing electrocorticographic signals in humans. Journal of Neural engineering, 4:264, 2007.\n\n[12] O. Bai, V. Rathi, P. Lin, D. Huang, H. Battapady, D. Y. Fei, L. Schneider, E. Houdayer, X. Chen,\nand M. Hallett. Prediction of human voluntary movement before it occurs. Clinical Neuro-\nphysiology, 122:364\u2013372, 2011.\n\n[13] O. Bai, P. Lin, S. Vorbach, J. Li, S. Furlani, and M. Hallett. Exploration of computational\nmethods for classi\ufb01cation of movement intention during human voluntary movement from\nsingle trial EEG. Clinical Neurophysiology, 118:2637\u20132655, 2007.\n\n[14] U. Maoz, A. Arieli, S. Ullman, and C. Koch. Using single-trial EEG data to predict laterality\n\nof voluntary motor decisions. Society for Neuroscience, 38:289.6, 2008.\n\n[15] C. Camerer. Behavioral game theory: Experiments in strategic interaction. Princeton Univer-\n\nsity Press, 2003.\n\n[16] J. D. Haynes. Decoding and predicting intentions. Annals of the New York Academy of Sci-\n\nences, 1224:9\u201321, 2011.\n\n[17] P. Haggard. Decision time for free will. Neuron, 69:404\u2013406, 2011.\n[18] J. D. Haynes. Beyond libet. In W. Sinnott-Armstrong and L. Nadel, editors, Conscious will\n\nand responsibility, pages 85\u201396. Oxford University Press, 2011.\n\n[19] A. Muralidharan, J. Chae, and D. M. Taylor. Extracting attempted hand movements from EEGs\n\nin people with complete hand paralysis following stroke. Frontiers in neuroscience, 5, 2011.\n\n[20] E. Lew, R. Chavarriaga, S. Silvoni, and J. R. Milln. Detection of self-paced reaching movement\n\nintention from EEG signals. Frontiers in Neuroengineering, 5:13, 2012.\n\n9\n\n\f", "award": [], "sourceid": 4513, "authors": [{"given_name": "Uri", "family_name": "Maoz", "institution": null}, {"given_name": "Shengxuan", "family_name": "Ye", "institution": null}, {"given_name": "Ian", "family_name": "Ross", "institution": null}, {"given_name": "Adam", "family_name": "Mamelak", "institution": null}, {"given_name": "Christof", "family_name": "Koch", "institution": null}]}