{"title": "Information-based Adaptive Stimulus Selection to Optimize Communication Efficiency in Brain-Computer Interfaces", "book": "Advances in Neural Information Processing Systems", "page_first": 4820, "page_last": 4830, "abstract": "Stimulus-driven brain-computer interfaces (BCIs), such as the P300 speller, rely on using a sequence of sensory stimuli to elicit specific neural responses as control signals, while a user attends to relevant target stimuli that occur within the sequence. In current BCIs, the stimulus presentation schedule is typically generated in a pseudo-random fashion. Given the non-stationarity of brain electrical signals, a better strategy could be to adapt the stimulus presentation schedule in real-time by selecting the optimal stimuli that will maximize the signal-to-noise ratios of the elicited neural responses and provide the most information about the user's intent based on the uncertainties of the data being measured. However, the high-dimensional stimulus space limits the development of algorithms with tractable solutions for optimized stimulus selection to allow for real-time decision-making within the stringent time requirements of BCI processing. We derive a simple analytical solution of an information-based objective function for BCI stimulus selection by transforming the high-dimensional stimulus space into a one-dimensional space that parameterizes the objective function - the prior probability mass of the stimulus under consideration, irrespective of its contents. We demonstrate the utility of our adaptive stimulus selection algorithm in improving BCI performance with results from simulation and real-time human experiments.", "full_text": "Information-based Adaptive Stimulus Selection to\n\nOptimize Communication Ef\ufb01ciency in\n\nBrain-Computer Interfaces\n\nBoyla O. Mainsah,1 Dmitry Kalika,2 Leslie M. Collins,1,\u2217\n\nSiyuan Liu,1 Chandra S. Throckmorton1\n\n1Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA\n\n2Johns Hopkins University Applied Physics Laboratory, Laurel, MD, USA\n\n\u2217Corresponding author: leslie.collins@duke.edu\n\nAbstract\n\nStimulus-driven brain-computer interfaces (BCIs), such as the P300 speller, rely\non using a sequence of sensory stimuli to elicit speci\ufb01c neural responses as con-\ntrol signals, while a user attends to relevant target stimuli that occur within the\nsequence. In current BCIs, the stimulus presentation schedule is typically gene-\nrated in a pseudo-random fashion. Given the non-stationarity of brain electrical\nsignals, a better strategy could be to adapt the stimulus presentation schedule in\nreal-time by selecting the optimal stimuli that will maximize the signal-to-noise\nratios of the elicited neural responses and provide the most information about\nthe user\u2019s intent based on the uncertainties of the data being measured. How-\never, the high-dimensional stimulus space limits the development of algorithms\nwith tractable solutions for optimized stimulus selection to allow for real-time\ndecision-making within the stringent time requirements of BCI processing. We\nderive a simple analytical solution of an information-based objective function for\nBCI stimulus selection by transforming the high-dimensional stimulus space into a\none-dimensional space that parameterizes the objective function - the prior proba-\nbility mass of the stimulus under consideration, irrespective of its contents. We\ndemonstrate the utility of our adaptive stimulus selection algorithm in improving\nBCI performance with results from simulation and real-time human experiments.\n\n1\n\nIntroduction\n\nBrain-computer interfaces (BCIs) acquire brain signals in real-time, process the signals to extract\nrelevant neural information and translate this information into commands that convey a user\u2019s state\nor intent to control external devices [1]. BCIs are typically de\ufb01ned by the speci\ufb01c neural signal\ncomponents that are used to control the BCI [2]. The generation of these neural signals can either\nbe self-initiated, such as with motor imagery, or elicited with sensory stimuli, such as is the case\nwith BCIs based on event-related potentials (ERPs). BCIs can replace or restore control abilities\nin individuals with severe motor disabilities such as amyotrophic lateral sclerosis (ALS) or spinal\ncord injury [3]. The P300 ERP-based BCI speller [4] is one of the most widely researched BCIs for\ncommunication for individuals whose severe neuromuscular limitations preclude their use of most\ncommercially available assistive technologies, such as individuals with late stage ALS [1, 5]. The\nP300 speller relies on eliciting and detecting ERP responses embedded in electroencephalography\n(EEG) data while a user attends to relevant target visual, tactile or auditory stimuli occurring within\na sequence of non-relevant (background) stimuli. In a typical visual P300 speller, a user focuses\non a desired character displayed on an interface with M possible options, such as in Figure 1(a),\nwhile groups of characters, termed \ufb02ash groups, are sequentially illuminated on the screen. Ideally,\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00b4eal, Canada.\n\n\fFigure 1: (a) Example P300 speller interface. (b) Stimulus codebook associated with the correspon-\nding row-column paradigm (RCP). Each column in the codebook represents a \ufb02ash group with the\ncharacters highlighted in white. For example, the illuminated row \ufb02ash group in (a) is \ufb02ash group 6\nin (b). Each row represents the presentation pattern or codeword of a character.\n\na P300 ERP is elicited when the target character is presented or \u201c\ufb02ashed.\u201d After multiple stimulus\npresentations, the BCI analyzes the user\u2019s EEG data with an automated algorithm to make a decision\nabout the user\u2019s intended character.\nUnfortunately, current stimulus-driven BCIs are limited by their slow communication rates due to the\ninherent limitations of relying on sensory stimulation and noisy EEG data to control the BCI. Multiple\ndata measurements for each character being spelled are needed to increase the signal-to-noise ratios\n(SNRs) of the ERPs to facilitate their detection accuracy in noisy EEG data. Also, ERPs exhibit high\ntemporal variability and are susceptible to psychophysical factors, such as refractory effects, user\ndistractions and fatigue, each of which negatively impact the ERP elicitation process [6, 7]. The\nability to modulate ERP responses via the stimulus presentation schedule [8] provides a mechanism\nby which to improve BCI performance. However, conventional BCI stimulus paradigms rely on\npseudo-randomly generated stimulus presentation schedules, where the design process is often based\non simplicity. A simple way to design a stimulus presentation paradigm is to group characters\nby rows and columns of a grid and present them in random order, which is commonly known as\nthe row-column (RC) paradigm [4] (see Figure 1(b)). However, the RC paradigm is susceptible to\npsychophysical factors that negatively impact BCI performance. Refractory effects, for example,\noccur when there is a short time interval between the target character\u2019s presentation, or the target-to-\ntarget interval (TTI), which reduces the ERP SNR [7]. Due to random stimulus selection, a signi\ufb01cant\nproportion of characters in the RC paradigm have short TTIs, and this exacerbates refractory effects.\nAlso, users report more visual discomfort with the RC paradigm when characters adjacent to the\ntarget character are \ufb02ashed together with the target [9].\nOther stimulus presentation paradigms have been developed to address the limitations of the RC\nparadigm by using heuristics with the goal of reducing known sources of errors during ERP elici-\ntation. The checkerboard paradigm [9] uses a set of heuristics on characters in a grid interface to\nprevent adjacent characters from \ufb02ashing together and impose a minimum TTI between character\npresentations. Other research groups have investigated optimizing the stimulus presentation schedule\nin advance using error-correcting codes, where the Hamming distance between character stimulus\npresentation patterns, or codewords, are maximized to increase error recovery during the decoding\nprocess. However, maximizing Hamming distances favors the selection of character presentation\npatterns with short TTIs, which increases refractory effects. Hence, the use of error-correcting codes\nis unlikely to improve BCI performance [10, 11, 12], unless the impact of refractory effects are\nfactored into the codebook design process [13]. Overall, all of these previously considered stimulus\nparadigms do not necessarily provide the best approach to maximize BCI performance as not every\nstimulus provides the same amount of utility to estimate the user\u2019s intent based on the currently\nobserved data. Despite its limitations, the RC paradigm, which was used in the original P300 speller\nstudy [4], still remains the standard stimulus paradigm that is used in the BCI literature [14].\nAlternatively, the BCI stimulus selection process can be adapted in real-time based on the data\nthat is currently measured to maximize the elicited neural responses [15, 16]. Adaptive stimulus\nselection is achieved by solving a combinatorial optimization problem with an objective function that\nquanti\ufb01es the usefulness of a stimulus-response pair to estimate the user\u2019s target character given the\n\n2\n\n(a) P300 Speller Interface(b) Stimulus Codebook for the RCPFlash group (randomized, without replacement)123456789101112Grid characterABCDEFGHIJKLMNOPQRSTUVWXYZSp123456789\funcertainties in the data. The stimulus selection process is subject to real-time BCI system constraints\nas the selection process from a space of 2M possible \ufb02ash groups needs to be executed within the time\ninterval between two consecutive stimuli (known as the inter-stimulus interval, ISI), which typically\nranges from 60-250ms in most BCI systems. A previous P300 speller study that utilized a partially\nobserved Markov decision process for adaptive stimulus selection restricted the search space to just\nrow or column \ufb02ash groups to obtain a more tractable solution [17]. A better strategy is to allow the\nstimulus selection process the \ufb02exibility to explore the whole stimulus space to dynamically create\n\ufb02ash groups to maximize the objective function, which requires a computationally ef\ufb01cient algorithm.\nWe hypothesize that the limited development of adaptive stimulus selection algorithms in the current\nBCI literature [18] is likely due to the lack of objective functions with tractable solutions (the \u201ccurse\nof dimensionality\u201d) to allow for real-time algorithm implementation. We have developed a simple,\nyet powerful analytical solution to an objective function, which allows for computational ef\ufb01ciency\nin exploring the high dimensional BCI stimulus space: the objective function is parameterized by\nthe prior probability mass of a future stimulus under consideration, irrespective of its content. The\nmain contributions of this work are as follows: 1) We introduce an adaptive BCI stimulus selection\nalgorithm, where the objective is to maximize the information content elicited from future stimuli\nin estimating the user\u2019s intent, based on the neural responses being measured and the BCI\u2019s current\nbelief regarding the user\u2019s intent; 2) We outline considerations and practical steps for real-time\nimplementation of our adaptive BCI stimulus selection algorithm; and 3) We present preliminary\nresults from simulation and human BCI experiments that demonstrate the potential to obtain signi\ufb01cant\nperformance improvements with our adaptive stimulus selection algorithm.\n\n2 Adaptive Stimulus Selection Algorithm\n\n2.1 P300 BCI Speller Overview\n\nA user can select one of M choices using the P300 speller. To make a selection, the user focuses on\na desired target character, C\u2217, on a computer screen (such as the one shown in Figure 1(a)) while\nsubsets of characters or \ufb02ash groups are sequentially illuminated. Let f t = [ft,1, ..., ft,M ] represent\na binary M-dimensional vector where each bit indicates whether a character is (ft,m = 1) or is not\n(ft,m = 0) present within a \ufb02ash group. The user\u2019s EEG response to a \ufb02ash group presentation is\nprocessed and scored with a user-speci\ufb01c classi\ufb01er, yielding a score, yt. The classi\ufb01er score is used to\nupdate a decoding function that quanti\ufb01es the possibility of each of the BCI choices to be the user\u2019s\ntarget character given the observed data. A stopping rule is used to determine when to terminate data\ncollection; the standard approach is to collect a \ufb01xed amount of data, termed static stopping. After\ndata collection, the character with the maximum score is selected as the target character estimate, \u02c6C\u2217.\nFor the decoding function, we use the naive Bayesian dynamic stopping algorithm developed in [19]\nto quantify the probability of each possible character being the target character after each \ufb02ash group\npresentation. Dynamic stopping (DS) is achieved by setting a probability threshold, Pth, to terminate\ndata collection; hence the quality of data is used to balance the trade-off between minimizing the\namount of data collection to increase spelling speed and maximizing the amount of data collection to\nincrease accuracy. The character probabilities at time index t are updated accordingly:\n\nP (C\u2217 = m|yt, ft) =\n(cid:52)\n= lt,m(yt) =\n\np(yt|C\u2217 = m, ft)\n\n(cid:80)\np(yt|C\u2217 = m, ft)P (C\u2217 = m|yt\u22121, ft\u22121)\nj p(yt|C\u2217 = j, ft)P (C\u2217 = j|yt\u22121, ft\u22121)\n\n(cid:26) l0(yt), if ft,m = 0\n\n,\n\n(1)\n\nl1(yt), if ft,m = 1\n\n(2)\nwhere P (C\u2217 = m|yt, ft) is the posterior probability that character m is the target character, C\u2217,\ngiven the presented \ufb02ash groups, ft = [f 1, ..., f t], and classi\ufb01er scores, yt = [y1, ..., yt]; P (C\u2217 =\nm|yt\u22121, ft\u22121) is the prior probability; p(yt|C\u2217 = m, f t) is the likelihood of generating the classi\ufb01er\nscore, yt, given character m is the target character and the current \ufb02ash group sequence, ft; l0 and l1\nare the class conditional classi\ufb01er score probability density functions (pdfs) for non-target and target\nstimulus events, which are estimated for each BCI user during system calibration. The denominator\nterm in (1) represents the marginal probability of the classi\ufb01er score conditioned on the current data:\n\n,\n\np(yt|yt\u22121, ft) =\n\np(yt|C\u2217 = j, ft)P (C\u2217 = j|yt\u22121, ft\u22121).\n\n(3)\n\nM(cid:88)\n\nj=1\n\n3\n\n\fThis alternative expression for the marginal probability, p(yt|yt\u22121, ft), will be useful in simplifying\nthe objection function that is used in our adaptive stimulus selection algorithm.\n\n2.2 Objective Function\n\nThe information encoded in the user\u2019s EEG response to a stimulus is condensed into a classi\ufb01er score,\nwhich is used to update the BCI system\u2019s current belief regarding the user\u2019s target character. Due to\nnoise, multiple character-speci\ufb01c presentation patterns can generate the same sequence of classi\ufb01er\nscores, which can lead to BCI decoding errors. The more information about the target character that\nis contained in the classi\ufb01er scores, the more likely it is that the target character will be correctly\nidenti\ufb01ed. To facilitate correct target character estimation in as few stimulus presentations as possible,\nwe use an information-based criterion [20] to bias the stimulus selection process towards stimuli that\nprovide the most information to the BCI to correctly estimate the user\u2019s intent given the current data.\nMutual information is a non-negative measure in information theory that quanti\ufb01es the amount of\ninformation about a random variable that can be obtained from another random variable [22]. It\nis typically denoted as I(A; B), where A and B represent two random variables. The utility of a\nhypothetical future \ufb02ash group, f h\nt+1, can be estimated by evaluating the mutual information between\nthe target character, C\u2217, and a hypothetical classi\ufb01er score, Y h\nt+1, that is generated in response to the\npresentation of f h\nt+1. This amount of information can also be evaluated within the context of how\nmuch the currently observed data (i.e., the previously observed \ufb02ash groups, ft, and classi\ufb01er scores,\nyt) reduce the uncertainty about the target character estimate. If A and B represent continuous and\ndiscrete random variables, respectively, and r represents a speci\ufb01c value of another random variable,\nR, the mutual information between A and B conditioned on r is calculated accordingly:\n\n(cid:90) \u221e\n\n\u2212\u221e\n\n(cid:34)(cid:88)\n\nb\n\n(cid:19)(cid:35)\n\n(cid:18) P (b|a, r)\n\nP (b|r)\n\nI(A; B|r) =\n\np(a|r)\n\nP (b|a, r) log\n\nda|r.\n\n(4)\n\nAssuming the Bayesian algorithm (1)-(3), we use the following objective function based on mutual\ninformation to select a \ufb02ash group at each time step accordingly:\n\nI(Y h\n\nt+1; C\u2217|yt, fh\n\nt+1) =\n\n=\n\n(cid:34) M(cid:88)\n(cid:18) P h(C\u2217 = m|yh\n\nt+1)\n\nm=1\n\nt+1|yt, fh\n\n(cid:32)\nP (C\u2217 = m|yt, ft)\n\nlt+1,m(zh\n\nt+1)pt,m log\n\n(cid:90) \u221e\n\n\u2212\u221e\n\n(cid:90) \u221e\n\n\u2212\u221e\n\np(yh\n\nlog\n\n(cid:34) M(cid:88)\n\nm=1\n\nt+1, ft+1)\n\nP h(C\u2217 = m|yh\n\n(cid:19)(cid:21)\n(cid:80)M\nt+1 \u2208 F ,\nt+1), \u2200f h\n\nt+1, ft+1) \u00d7\n\nt+1|yt, fh\n\ndyh\n\nt+1\n\nlt+1,m(zh\nc=1 lt+1,c(zh\n\nt+1)\nt+1)pt,c\n\nf s\n\nt+1 = argmax\n\nf h\n\nt+1\n\nI(Y h\n\nt+1; C\u2217|yh\n\nt , fh\n\n(cid:33)(cid:35)\n\ndzh\n\nt+1,\n\n(5)\n\n(6)\n\nt+1; C\u2217|yh\n\nt+1, fh\n\nt+1 = [ft, f h\n\nwhere I(Y h\nt+1) is the mutual information between the future classi\ufb01er score Y h\nand the target character C\u2217 conditioned on the current classi\ufb01er scores, yt, and hypothetical future\nt+1\n(cid:52)\n\ufb02ash group sequence, fh\n=\nt+1|yt, fh\nt+1 is the \ufb02ash group that maximizes the objective\nyh\nfunction, which is selected for presentation. Alternatively, the mutual information can be expressed as\nan expectation of the Kullback-Leibler divergence (DKL) between the hypothetical posterior (ph\nt+1)\nand current (pt) character probability distributions taken over the future conditional classi\ufb01er score,\nt+1 [20, 21],\nzh\n\nt+1]; F is the search space of all possible \ufb02ash groups; zh\n\nt+1 for notational simplicity; and f s\n\nt+1\n\nI(Y h\n\nt+1; C\u2217|yt, fh\n\nt+1)\n\n(cid:52)\n= Ezh\n\nt+1\n\n(cid:2)DKL(ph\n\nt+1||pt)(cid:3) .\n\n(7)\n\nThe conditional mutual information in (5) is not the most convenient formulation to solve the\noptimization problem due to the lack of a closed form solution. When using gradient-based methods\nin a high dimensional space (i.e., in terms of the M-dimensional space, F) where the solution to the\nobjective function is not tractable, convergence within a time limit (which is vital for BCI real-time\n\n4\n\n\foperation) is not guaranteed and there is the possibility to be stuck in a local maximum. Alternatively,\nwe can exploit the binary choice in the character likelihood assignments (2) to group together \ufb02ashed\nand non-\ufb02ashed characters in the \ufb02ash group under consideration, f h\nt+1. With this grouping, the\ndenominator of the logarithm term in (5) can be expressed as:\n\nM(cid:88)\n\nc=1\n\nlt+1,c(zh\n\nt+1)pt,c =\n\nP 1t(f h\n\nt+1) =\n\n\uf8f9\uf8fb +\n\n\uf8ee\uf8f0l1(zh\n\n(cid:88)\n\npt,c\n\nt+1)\n\n\u2200c:ft,c=1\nt+1)P 1t(f h\n\nt+1)\n\nt+1)) + l1(zh\n\n\uf8f9\uf8fb\n\n(cid:88)\n\n\uf8ee\uf8f0l0(zh\nt+1)\n(cid:88)\nt+1)(1 \u2212 P 1t(f h\n\n\u2200c:ft,c=0\n\npt,c\n\n= l0(zh\n\npt,c\n\n\u2200c:f h\n\nt+1,c=1\n\n(8)\n\n(9)\n\n(10)\n\n(cid:19)\n\n(11)\n\nt+1) is the sum of prior probabilities at time t for characters that are \ufb02ashed in f h\n\nwhere P 1t(f h\nt+1,\nwhich we will denote as P 1t for simplicity. Similarly, we group together \ufb02ashed and non-\ufb02ashed\ncharacters in the discrete integral in (5) to obtain the following expression for the mutual information:\n\n(cid:90) \u221e\nM(cid:88)\n\n\u2212\u221e\n\nI(Y h\n\nt+1; C\u2217|yt, fh\n\nt+1) =\n\nI(zh\n\nt+1) =\n\nI(zh\n\nt+1)dzh\n\nt+1\n\n(cid:18)\n\nlt+1,m(zh\n\nt+1)pt,m log\n\nm=1\n\n= P 1tl1(zh\n\nt+1) log\n\n(cid:18)\n\nl0(zh\n\nt+1)\n\nlt+1,m(zh\n\n(cid:19)\nt+1)(1 \u2212 P 1t) + l1(zh\nl1(zh\n\nt+1)\n\nt+1)P 1t\n\nl0(zh\n\nt+1)(1 \u2212 P 1t) + l1(zh\n(cid:18)\n\nl0(zh\n\nt+1)\n\nt+1)P 1t\n\nl0(zh\n\nt+1)(1 \u2212 P 1t) + l1(zh\n\nt+1)P 1t\n\n+\n\n(cid:19)\n\n(1 \u2212 P 1t)l0(zh\n\n+1) log\n\nThe mutual information in (10) is now expressed as an integral function of l0 and l1, and P 1t\nover zh\nt+1. If the BCI classi\ufb01er score pdfs, l0 and l1, are \ufb01xed, after performing the integral over\nt+1, the mutual information is only a function of P 1t. The objective function is now conveniently\nzh\nparameterized by P 1t, the total prior probability of characters within a \ufb02ash group.\nThe mutual information function of a BCI user can be esti-\nmated from their respective calibrated classi\ufb01er score pdfs.\nFigure 2 shows examples of mutual information functions\nestimated for different BCI users. Given a user\u2019s mutual\ninformation function, the P 1t value that maximizes the\nobjective function, which we denote as P 1opt, can be ex-\ntrapolated. The advantage of an analytical solution is that,\nwhile the \ufb02ash group that maximizes the objective func-\ntion varies at every time step given the observed data, the\nstimulus that maximizes the objective function can sim-\nply be de\ufb01ned by its prior probability. Hence, the \ufb02ash\ngroup whose prior probability mass (9) is closest to P 1opt\ncan be determined and selected for presentation. With a\nknown solution, we can more easily explore a much larger\nstimulus space in a time ef\ufb01cient manner by using a greedy\nsearch strategy to iterate over an ordered list of character\nprobabilities to construct a close-to-optimal \ufb02ash group.\n\nFigure 2: Examples of user-speci\ufb01c\nmutual information (MI) functions esti-\nmated using their respective calibrated\nBCI non-target (l0) and target (l1)\nclassi\ufb01er score pdfs (n = 29) [19].\n\n2.3 Considerations\nfor Real-time Algorithm Implementation\n\nSystem Constraints So far, it is assumed in (6) that the classi\ufb01er score, yt, that is generated after\npresenting the \ufb02ash group, f t, is available instantaneously prior to selecting the \ufb02ash group for the\nnext time step, f s\nt+1. However, this is typically not the case during real-time BCI implementation as\nthere is a delay between presenting a stimulus and computing the corresponding classi\ufb01er score due\n\n5\n\nStimuluspriorprobability,P1t00.20.40.60.81MI(l0;l1;P1t)00.10.20.30.40.5Mutual Information Functions\fto processing a time window of EEG data, as illustrated in Figure 3. Due to the shorter ISI duration\nrelative to the data processing window, stimulus presentation is still ongoing while accumulating\nthe EEG data buffer associated with a current \ufb02ash group, f t. For example, in Figure 3, the \ufb02ash\ngroups [f t+1, ..., f t+6] will have already been presented prior to computing yt. We will de\ufb01ne the\ndata processing delay by the number of additional \ufb02ash groups prior to being able to compute the\nclassi\ufb01er score after each \ufb02ash group presentation, denoted as \u03c4. This means with adaptive stimulus\nselection, the \ufb01rst \u03c4 \ufb02ash groups have to be initialized and the data available at time step t is used to\nselect the \ufb02ash group at t + \u03c4 + 1.\nTo accommodate the data processing delay, the mutual\ninformation function is modi\ufb01ed accordingly:\n\n(cid:2)DKL(ph\n\nt+\u03c4 +1||pt)(cid:3) .\n\nI(Y h\n\nt+\u03c4 +1)\n\nt+\u03c4 +1\n\n= E\n\nt+\u03c4 +1; C\u2217|yt, fh\nt+\u03c4 +1|yt,fh\nyh\nt+\u03c4 +1|yt, fh\n\nt+1, ..., yh\n\nt+1, ..., f h\n\nHowever, computing the conditional hypothetical classi-\n\ufb01er score, yh\nt+\u03c4 +1, is computationally expen-\nit involves a series of nested integrals over all\nsive:\npossible \ufb02ash groups, [f h\nt+\u03c4 ], and classi\ufb01er\nscores, [yh\nt+\u03c4 ]. For computational simplicity,\nwe approximate the data available at t + \u03c4 with that\navailable at t, i.e. \u02c6yt+\u03c4 = yt and \u02c6pt+\u03c4 = pt. While\nthese approximations are not the best estimates for their\nrespective values at t+\u03c4, the substitutions enable the use\nof (5) when implementing adaptive stimulus selection\nwith a data processing delay:\n\nf s\n\nt+\u03c4 +1 = arg max\n\nf h\n\nt+\u03c4 +1\n\nE\nt+\u03c4 +1|\u02c6yt+\u03c4 ,fh\nyh\n\nt+\u03c4 +1\n\nFigure 3: Illustration of the BCI data pro-\ncessing delay between when a stimulus, ft,\nis presented and when the resulting classi-\n\ufb01er score, yt, is computed.\n\n(cid:2)DKL(ph\n\nt+\u03c4 +1||\u02c6pt+\u03c4 )(cid:3) , \u2200f h\n\nt+\u03c4 +1 \u2208 F c\n\n(12)\n\nPhysiological Constraints We also consider mitigating the impact of psychophysical factors during\nstimulus selection, which is achieved by placing restrictions in the stimulus space. Refractory effects\nare typically mitigated by imposing a minimum time interval between any character\u2019s presentation to\nincrease the likelihood of a long TTI to elicit ERPs with high SNRs [19]. Alternatively, refractory\neffects can be mitigated by using a non-naive classi\ufb01cation algorithm that models the relationship\nbetween TTI and classi\ufb01er scores to bias the stimulus selection process towards TTI values that will\nmaximize the discriminability between target and non-target classi\ufb01er scores. (See supplementary\nmaterial, Section A for the corresponding objective function.) However, other psychophysical factors,\nsuch as adjacency distractions and visual fatigue, are more complex to model for mitigation. Using\ninsight from previous studies, we imposed additional constraints to mitigate the impact of these other\neffects. For example, the spatial distance between characters in large-sized \ufb02ash groups usually\ntends to be small, which may exacerbate adjacency distractions. Also, consecutive presentations\nof highly correlated \ufb02ash groups can lead to visual fatigue. We mitigated refractory effects and\nadjacency distractions by imposing a minimum TTI and limiting the \ufb02ash group size, respectively.\n\n2.4 Pseudo-code\nfor Adaptive Stimulus Selection\n\nGiven an initialization \ufb02ash group set,\n[s1, s2, ...., s\u03c4 ] and classi\ufb01er score pdfs l0 and\nl1, the adaptive stimulus selection algorithm\nassuming a data processing delay \u03c4 (12) and a\nsearch space of \ufb02ash groups with stimulus space\nconstraints, F c, is outlined in Algorithm 1.\n\n3 Simulation Experiment\n\nAlgorithm 1: Adaptive BCI Stimulus Selection\nUsing Mutual Information (MI).\nOf\ufb02ine Solution\n\nMI(l0, l1, P 1t)\n\n(cid:46) (10)\n\nP 1opt = arg max\nP 1t\n\nOnline Encoder\n\nif 1 \u2264 t \u2264 \u03c4 then\n\nf s\n\nt = st\n\nelse\n\nf s\n\nt = arg min\nt \u2208F c\nf h\n\n(cid:46) Initialization set;\n\n(cid:12)(cid:12)(cid:12)P 1opt \u2212 P 1t\u22121\u2212\u03c4 (f h\n\nt )\n\n(cid:12)(cid:12)(cid:12)\n\nUsing the framework outlined in [23], we per-\nformed simulations (in MATLAB) to compare our adaptive stimulus selection algorithm with two\n\n6\n\nTimeOffOnftft+1ft+2ft+3ft+4ft+5ft+6ft+7ytyt+1EEG data window at time index, tStimulus indicatorClassifier score indicatorEEG signals\fconventional random BCI stimulus selection methods, the RC [4] and the checkerboard (CB) [9]\nparadigms. The following scenarios with the adaptive stimulus selection algorithm were simulated:\n\n(i) An ideal scenario with a greedy search strategy and no constraints imposed (6) to obtain a\n\nperformance upper bound.\n\n(ii) A scenario with a greedy search strategy and imposed constraints (12) to obtain more realistic\nperformance bounds associated with real-time algorithm implementation. The constraints\nincluded: a data processing delay \u03c4 = 6; a \ufb02ash group size limit of 9; and a stochastic TTI\nrestriction using a pre-de\ufb01ned probability distribution with a minimum value of 3.\n\n(iii) A scenario with a search space of only row and column \ufb02ash groups and the same constraints\nas in (ii) above (excluding the \ufb02ash group limit) to obtain performance bounds when using a\nmore restricted search space. The character selection process occurred in two stages with the\nselection of the row \ufb02ash group followed by the selection of the column \ufb02ash group, or vise\nversa. We denote this two-stage selection process with row and column \ufb02ash groups as RC2.\n\nIn each simulation run, the target character was uniformly drawn from 72 characters assuming an\n8 \u00d7 9 grid, and \ufb02ash groups were generated based on the speci\ufb01ed stimulus paradigm. The classi\ufb01er\nscores associated with non-target and target \ufb02ash groups were assumed to be normally distributed and\ndrawn with parameters d = (\u00b51\u2212\u00b50)/\u03c3, where d is the detectability index, which quanti\ufb01es classi\ufb01er\nperformance level; \u00b50 and \u00b51 are the means of the classi\ufb01er score pdfs l0 and l1, respectively, and \u03c3\nis the assumed common standard deviation. In the Bayesian DS algorithm, character probabilities\nwere initialized uniformly, and the stopping threshold and data collection limit were set to Pth = 0.9\nand 120 stimulus \ufb02ashes, respectively.\nSelection accuracy and the average number of \ufb02ash group presentations prior to character selection,\ndenoted as the expected stopping time (EST), were estimated with results averaged over 1500\nsimulation runs. Results are shown in Figure 4. (See supplementary material, Section B for example\nstimulus presentation schedules generated for the stimulus presentation paradigms.) In general, the\nadaptive stimulus paradigms performed signi\ufb01cantly better than the random paradigms, even given\nthe performance drop from the ideal condition. The advantage of a larger search space during adaptive\nstimulus selection is evident when comparing the performances between the RC2 and greedy adaptive\nparadigms with constraints. Overall, these results show the signi\ufb01cant margins of improvements in\nBCI accuracy and spelling speed that can potentially be obtained with adaptive stimulus selection.\n\nFigure 4: Results from simulations showing the performance of the Bayesian dynamic stopping\nalgorithm with various BCI stimulus presentation paradigms as a function of detectability index, d.\n\n4 Online Experiment\n\nA preliminary study was conducted to investigate the utility of adaptive stimulus selection during\nreal-time BCI use, as compared to a conventional BCI stimulus selection algorithm. Eight healthy\nparticipants were recruited from the student population at Duke University for a study approved by\nthe Institutional Review Board. Participants gave informed consent prior to their experiment session.\nThe open source BCI2000 software [24] was used to implement the P300 speller with the Bayesian\nDS algorithm, with the stopping probability threshold and data collection limit set to Pth = 0.9 and\n145 stimulus \ufb02ashes, respectively. The \ufb02ash duration, ISI and time pause between character selections\nwere set to 62.5 ms, 62.5 ms and 3.5 s, respectively. Data collected from electrodes {Fz, Cz, P3, Pz,\n\n7\n\nDetectability Index, d00.511.522.53Percent correct (%)020406080100(a) AccuracyRow-column (RC) RandomCheckerboard RandomGreedy Adaptive - IdealGreedy Adaptive with ConstraintsRC2 Adaptive with ConstraintsDetectability Index, d00.511.522.53Stimulus presentations per character (SPC)020406080100120(b) Expected Stopping Time\fP4, PO7, PO8, Oz} were used for signal processing [25]. EEG signals were sampled at a rate of 256\nHz and \ufb01ltered between 0.5-30 Hz [19]. Features were extracted from an 800 ms segment of EEG\ndata at each electrode following each \ufb02ash, by down-sampling to a rate of 20 Hz using bin averaging\n[25]. The averaged samples were concatenated across channels to form the feature vectors supplied to\na classi\ufb01er. Two stimulus paradigms were tested: the CB paradigm and the greedy adaptive paradigm\nwith constraints (Section 3 (ii)).\nParticipants performed word copy-spelling tasks (three 6-letter words) with the P300 speller. In a\ncopy-spelling task, the user is instructed by the BCI as to the character in the grid to focus on. The\nBCI evaluation protocol for a stimulus presentation paradigm included: a calibration block, where the\nuser performs copy-spelling with no classi\ufb01er use or BCI feedback, to collect EEG data to estimate\nuser-speci\ufb01c BCI parameters (classi\ufb01er weight vector, classi\ufb01er likelihood functions and the objective\nfunction for adaptive stimulus selection); and a test block, where the user performs copy-spelling\nwith use of the estimated BCI parameters and BCI feedback to evaluate performance. Details on how\nwe calibrated the adaptive stimulus paradigm are provided in the supplemental material, Section C.\nThe testing order of the stimulus paradigms was randomized across participants to avoid order bias.\nParticipant accuracy and EST results are shown in Figure 5(a) and (b), respectively. Statistical\nsigni\ufb01cance tests were not performed due to the small sample size. On average, a small drop in\naccuracy was experienced with the adaptive paradigm (M \u00b1 SD = 72.50 \u00b1 26.11%) when compared\nto the CB paradigm (77.50 \u00b1 23.95%), with half of the participants maintaining similar accuracy\nlevels in both stimulus presentation paradigms. Across participants, signi\ufb01cant reductions in the EST\nwere obtained with the adaptive paradigm (55.77\u00b125.70 stimulus presentations per character (SPC))\nwhen compared to the CB paradigm (87.12\u00b146.54 SPC). User feedback revealed that in the adaptive\nparadigm, there were persistent presentations of characters in the spatial vicinity of the target character,\nwhich was distracting to most users while they focused on the target character.\n\nFigure 5: Participant performance, (a) expected stopping time, and (b) accuracy, with the checkerboard\nand adaptive stimulus paradigms. The mean for each measure is shown on the far right of each panel.\n\nThe common user experience of adjacency distractions with the adaptive paradigm condition indicated\nthat the constraints we imposed to minimize the impact of these effects were inadequate and this\nmay have limited the ability to obtain the full performance bene\ufb01ts with the adaptive stimulus\nparadigm. Based on the stochastic progression of the classi\ufb01er scores, the mean probability of\nthe target character should increase progressively over time, with a faster rate of convergence in a\nbetter-performing stimulus paradigm, assuming the same performance level across paradigms. To\ninform future algorithm re\ufb01nements, we performed post-hoc analysis to gain insight into how the\nrelationship between user behavior and stimulus paradigm performance were re\ufb02ected in the data.\nFigure 6 shows the progression of the mean target character probabilities in both paradigms. Half\nof the participants achieved higher spelling speeds with our adaptive stimulus paradigm while\nmaintaining similar accuracy levels as with the CB paradigm, which is generally illustrated in Figure\n6 by the faster algorithm convergence of the mean target character probability in the participants. In\nthe other participants, we observed an initial increase in the rate of convergence with our adaptive\nalgorithm. This indicates the algorithm is still useful in quickly narrowing down the potential\nchoices for the target character and biases the stimulus selection process accordingly by presenting\ncharacters in the vicinity of the target character. However, at higher stopping times, the target\ncharacter probability may occasionally decline and not always recover. We hypothesize that the\noccasional decline in the mean target character probability at higher stopping times is a negative\n\n8\n\nParticipant (sorted by CB accuracy)12345678meanPercent correct (%)020406080100(a) AccuracyCheckerboard (CB)AdaptiveParticipant (sorted by CB accuracy)12345678meanStimulus presentations per character (SPC)050100150(b) Expected Stopping Time\fconsequence of the increased adjacency distractions and user fatigue that occur as a result of the\nrepetitive presentations of characters around the target character in the adaptive stimulus paradigm.\n\nFigure 6: Progression of the mean target character probability for the adaptive and checkerboard (CB)\nstimulus presentation paradigms, as a function of stimulus \ufb02ash number for each participant.\n\nWe previously highlighted the need to impose constraints in the stimulus space to mitigate the negative\nimpact of psychophysical effects (Section 2.3). The interaction between algorithm and application is\ncrucial for understanding the likelihood of success of proposed methods due to theoretical assumptions\nthat are made during algorithm development. Adjacency distractions are more complex to model and\nmitigate, and the constraint we imposed of limiting \ufb02ash group size was not enough to overcome\nthe adjacency issue. Better approaches to mitigate adjacency distractions include imposing spatial\nconstraints in the stimulus space based on the interface geometry to prevent characters that are directly\nadjacent to each other from \ufb02ashing together, or using a center interface layout where singletons or\ngroups of characters are sequentially presented in the center of a screen [26].\n\n5 Conclusions\n\nWe have developed and tested an adaptive stimulus selection algorithm for the P300 BCI speller\nthat utilizes previous user responses to select future stimuli that are maximally informative of the\nuser\u2019s intent to improve BCI communication ef\ufb01ciency. To our knowledge, this is the \ufb01rst adaptive\nBCI stimulus selection method with a tractable analytical solution that provides the \ufb02exibility to\nef\ufb01ciently sample the high dimensional stimulus space in a real-time feasible manner. We provide\na simple parameterization of our objective function in a one-dimensional space that quanti\ufb01es the\nprior probability mass of a stimulus under consideration, irrespective of its content. We outlined\npractical steps to account for BCI system computational limitations and the potential impact of\npsychophysical factors during stimulus selection with real-time algorithm implementation. Although\nthis work focuses on the P300 speller, the methodology developed here for adaptive stimulus selection\nis applicable to other BCIs that execute a number of possible action queries prior to decision-making.\nWhile the performance trends in the simulations were not entirely re\ufb02ected with measures of online\nBCI use, our preliminary \ufb01ndings are promising as they show the potential bene\ufb01t of using a data-\ndriven stimulus selection strategy. In general, faster convergence was achieved with the adaptive\nstimulus selection algorithm; from the post-hoc analysis, we hypothesize that the initial increase of\nthe target character probability indicates that the stimulus selection process was able to quickly narrow\ndown to the vicinity of the target character. We believe that the limitations associated with a grid\ninterface layout can be addressed with additional re\ufb01nements of the stimulus space constraints, such\nas imposing spatial restrictions on characters within a \ufb02ash group to minimize adjacency distractions.\nThe ability to search a larger stimulus space with our algorithm allows for greater \ufb02exibility to achieve\nthe best possible implementation of an adaptive stimulus paradigm with a given BCI con\ufb01guration.\nIt should be noted that while BCI studies are typically conducted in a non-disabled population for time\nef\ufb01ciency and practicality during algorithm development, results from a non-disabled population may\nnot necessarily be applicable to individuals with severe neuromuscular limitations due to differences\nin disease cause and level of neuromuscular control. We expect different tolerance levels to stimulus\npresentation properties in a disabled population where there is a more limited ability to easily navigate\nthe user interface. Future work includes additional re\ufb01nements and testing of the adaptive stimulus\nselection algorithm with a larger sample size of non-disabled participants, and validating the algorithm\nin a study with individuals with ALS.\n\n9\n\n Flash Number, t05010015000.51Participant 600.51Participant 405010015000.51Participant 800.5(clipped at p = 0.5)Participant 100.51Participant 205010015000.51Participant 7050100150 Mean Target Probability00.51Participant 5Adaptive CB00.51Participant 3\fAuthor Contributions DK conceptualized and developed the adaptive stimulus selection algorithm,\ndesigned and conducted the simulation and the real-time human BCI experiments. BOM, LMC and\nCST provided technical contributions to the theoretical development and implementation of this work.\nSL performed the post-hoc analysis. All authors contributed to writing and editing the manuscript.\n\nReferences\n[1] J. R. Wolpaw and E. W. Wolpaw, Brain-computer interfaces: principles and practice. New York: Oxford\n\nUniversity Press, 2012.\n\n[2] R. A. Ramadan and A. V. Vasilakos, \u201cBrain computer interface: control signals review,\u201d Neurocomputing,\n\nvol. 223, pp. 26\u201344, 2017.\n\n[3] S. Moghimi, A. Kushki, A. M. Guerguerian, and T. Chau, \u201cA review of EEG-based brain-computer\ninterfaces as access pathways for individuals with severe disabilities,\u201d Assistive Technology, vol. 25, no. 2,\npp. 99\u2013110, 2013.\n\n[4] L. A. Farwell and E. Donchin, \u201cTalking off the top of your head: toward a mental prosthesis utilizing\nevent-related brain potentials,\u201d Electroencephalography and Clinical Neurophysiology, vol. 70, no. 6, pp.\n510\u2013523, 1988.\n\n[5] B. O. Mainsah, L. M. Collins, K. A. Colwell, E. W. Sellers, D. B. Ryan, K. Caves, and C. S. Throckmorton,\n\u201cIncreasing BCI communication rates with dynamic stopping towards more practical use: an als study,\u201d\nJournal of Neural Engineering, vol. 12, no. 1, p. 016013, 2015.\n\n[6] J. M. Clearwater, C. C. Kerr, C. J. Rennie, and P. A. Robinson, \u201cNeural mechanisms of ERP change: com-\nbining insights from electrophysiology and mathematical modeling,\u201d Journal of Integrative Neuroscience,\nvol. 7, no. 04, pp. 529\u2013550, 2008.\n\n[7] S. M. M. Martens, N. J. Hill, J. Farquhar, and B. Sch\u00a8olkopf, \u201cOverlap and refractory effects in a brain-\ncomputer interface speller based on the visual P300 event-related potential,\u201d Journal of Neural Engineering,\nvol. 6, no. 2, 2009.\n\n[8] K. Squires, S. Petuchowski, C. Wickens, and E. Donchin, \u201cThe effects of stimulus sequence on event\nrelated potentials: A comparison of visual and auditory sequences,\u201d Perception & Psychophysics, vol. 22,\nno. 1, pp. 31\u201340, 1977.\n\n[9] G. Townsend, B. K. LaPallo, C. B. Boulay, D. J. Krusienski, G. E. Frye, C. K. Hauser, N. E. Schwartz,\nT. M. Vaughan, J. R. Wolpaw, and E. W. Sellers, \u201cA novel P300-based brain-computer interface stimulus\npresentation paradigm: moving beyond rows and columns,\u201d Clinical Neurophysiology, vol. 121, no. 7, pp.\n1109\u201320, 2010.\n\n[10] J. Hill, J. Farquhar, S. Martens, F. Biessmann, and B. Sch\u00a8olkopf, \u201cEffects of stimulus type and of error-\ncorrecting code design on BCI speller performance,\u201d in Advances in Neural Information Processing\nSystems. Curran Associates, Inc., 2009, pp. 665\u2013672.\n\n[11] S. M. M. Martens, J. M. Mooij, N. J. Hill, J. Farquhar, and B. Sch\u00a8olkopf, \u201cA graphical model framework\nfor decoding in the visual ERP-based BCI speller,\u201d Neural Computation, vol. 23, no. 1, pp. 160\u2013182, 2011.\n[12] T. Verhoeven, P. Buteneers, J. R. Wiersema, J. Dambre, and P. Kindermans, \u201cTowards a symbiotic brain-\ncomputer interface: exploring the application-decoder interaction,\u201d Journal of Neural Engineering, vol. 12,\nno. 6, p. 066027, 2015.\n\n[13] B. O. Mainsah, G. Reeves, L. M. Collins, and C. S. Throckmorton, \u201cOptimizing the stimulus presentation\nparadigm design for the P300-based brain-computer interface using performance prediction,\u201d Journal of\nNeural Engineering, vol. 14, no. 4, p. 046025, 2017.\n\n[14] J. N. Mak, Y. Arbel, J. W. Minett, L. M. McCane, B. Yuksel, D. Ryan, D. Thompson, L. Bianchi, and\nD. Erdogmus, \u201cOptimizing the P300-based brain\u2013computer interface: current status, limitations and future\ndirections,\u201d Journal of Neural Engineering, vol. 8, no. 2, p. 025003, 2011.\n\n[15] C. DiMattina and K. Zhang, \u201cAdaptive stimulus optimization for sensory systems neuroscience,\u201d Frontiers\n\nin Neural Circuits, vol. 7, p. 101, 2013.\n\n[16] B. Cowley, R. Williamson, K. Clemens, M. Smith, and M. Y. Byron, \u201cAdaptive stimulus selection for\noptimizing neural population responses,\u201d in Advances in Neural Information Processing Systems, 2017, pp.\n1395\u20131405.\n\n[17] J. Park and K.-E. Kim, \u201cA POMDP approach to optimizing P300 speller BCI paradigm,\u201d IEEE Transactions\n\non Neural Systems and Rehabilitation Engineering, vol. 20, no. 4, pp. 584\u2013594, 2012.\n\n[18] P. Varona, D. A. Guarde\u02dcno, T. Nowotny, and F. d. B. R. Ortiz, \u201cOnline event detection requirements in\n\nclosed-loop neuroscience,\u201d in Closed Loop Neuroscience, 2016, pp. 81\u201391.\n\n10\n\n\f[19] C. S. Throckmorton, K. A. Colwell, D. B. Ryan, E. W. Sellers, and L. M. Collins, \u201cBayesian approach\nto dynamically controlling data collection in P300 spellers,\u201d IEEE Transactions on Neural Systems and\nRehabilitation Engineering, vol. 21, no. 3, pp. 508\u201317, 2013.\n\n[20] K. Kastella, \u201cDiscrimination gain to optimize detection and classi\ufb01cation,\u201d IEEE Transactions on Systems,\n\nMan, and Cybernetics-Part A: Systems and Humans, vol. 27, no. 1, pp. 112\u2013116, 1997.\n\n[21] D. Kalika, L. M. Collins, C. S. Throckmorton, and B. O. Mainsah, \u201cAdaptive stimulus selection in ERP-\nbased brain-computer interfaces by maximizing expected discrimination gain,\u201d in IEEE International\nConference on Systems, Man, and Cybernetics, 2017, pp. 1405\u20131410.\n\n[22] T. M. Cover and J. A. Thomas, Elements of Information Theory.\n[23] B. O. Mainsah, L. M. Collins, and C. S. Throckmorton, \u201cUsing the detectability index to predict P300\n\nJohn Wiley & Sons, 2006.\n\nspeller performance,\u201d Journal of Neural Engineering, vol. 13, no. 6, p. 066007, 2016.\n\n[24] G. Schalk and J. Mellinger, A Practical Guide to Brain-Computer Interfacing with BCI2000: General-\nPurpose Software for Brain-Computer Interface Research, Data Acquisition, Stimulus Presentation, and\nBrain Monitoring. Springer, 2010.\n\n[25] D. J. Krusienski, E. W. Sellers, D. J. McFarland, T. M. Vaughan, and J. R. Wolpaw, \u201cToward enhanced\n\nP300 speller performance,\u201d Journal of Neuroscience Methods, vol. 167, no. 1, pp. 15\u201321, 2008.\n\n[26] S. Lees, N. Dayan, H. Cecotti, P. Mccullagh, L. Maguire, F. Lotte, and D. Coyle, \u201cA review of rapid serial\nvisual presentation-based brain\u2013computer interfaces,\u201d Journal of Neural Engineering, vol. 15, no. 2, p.\n021001, 2018.\n\n11\n\n\f", "award": [], "sourceid": 2336, "authors": [{"given_name": "Boyla", "family_name": "Mainsah", "institution": "Duke University"}, {"given_name": "Dmitry", "family_name": "Kalika", "institution": "Johns Hopkins Applied Physics Laboratory"}, {"given_name": "Leslie", "family_name": "Collins", "institution": "Duke University"}, {"given_name": "Siyuan", "family_name": "Liu", "institution": "Duke University"}, {"given_name": "Chandra", "family_name": "Throckmorton", "institution": "Duke University"}]}