{"title": "Efficient characterization of electrically evoked responses for neural interfaces", "book": "Advances in Neural Information Processing Systems", "page_first": 14444, "page_last": 14458, "abstract": "Future neural interfaces will read and write population neural activity with high spatial and temporal resolution, for diverse applications. For example, an artificial retina may restore vision to the blind by electrically stimulating retinal ganglion cells. Such devices must tune their function, based on stimulating and recording, to match the function of the circuit. However, existing methods for characterizing the neural interface scale poorly with the number of electrodes, limiting their practical applicability. This work tests the idea that using prior information from previous experiments and closed-loop measurements may greatly increase the efficiency of the neural interface. Large-scale, high-density electrical recording and stimulation in primate retina were used as a lab prototype for an artificial retina. Three key calibration steps were optimized: spike sorting in the presence of stimulation artifacts, response modeling, and adaptive stimulation. For spike sorting, exploiting the similarity of electrical artifact across electrodes and experiments substantially reduced the number of required measurements. For response modeling, a joint model that captures the inverse relationship between recorded spike amplitude and electrical stimulation threshold from previously recorded retinas resulted in greater consistency and efficiency. For adaptive stimulation, choosing which electrodes to stimulate based on probability estimates from previous measurements improved efficiency. Similar improvements resulted from using either non-adaptive stimulation with a joint model across cells, or adaptive stimulation with an independent model for each cell. Finally, image reconstruction revealed that these improvements may translate to improved performance of an artificial retina.", "full_text": "Ef\ufb01cient characterization of electrically evoked\n\nresponses for neural interfaces\n\nNishal P. Shah \u2217\nStanford University\n\nSasidhar Madugula\nStanford University\n\nPawel Hottowy\n\nAGH University of Science and Technology\n\nAlexander Sher\n\nUniversity of California, Santa Cruz\n\nAlan Litke\n\nUniversity of California, Santa Cruz\n\nLiam Paninski\n\nColumbia University\n\nE.J. Chichilnisky\nStanford University\n\nAbstract\n\nFuture neural interfaces will read and write population neural activity with high\nspatial and temporal resolution, for diverse applications. For example, an arti\ufb01cial\nretina may restore vision to the blind by electrically stimulating retinal ganglion\ncells. Such devices must tune their function, based on stimulating and recording, to\nmatch the function of the circuit. However, existing methods for characterizing the\nneural interface scale poorly with the number of electrodes, limiting their practical\napplicability. This work tests the idea that using prior information from previous\nexperiments and closed-loop measurements may greatly increase the ef\ufb01ciency of\nthe neural interface. Large-scale, high-density electrical recording and stimulation\nin primate retina were used as a lab prototype for an arti\ufb01cial retina. Three key\ncalibration steps were optimized: spike sorting in the presence of stimulation arti-\nfacts, response modeling, and adaptive stimulation. For spike sorting, exploiting\nthe similarity of electrical artifact across electrodes and experiments substantially\nreduced the number of required measurements. For response modeling, a joint\nmodel that captures the inverse relationship between recorded spike amplitude and\nelectrical stimulation threshold from previously recorded retinas resulted in greater\nconsistency and ef\ufb01ciency. For adaptive stimulation, choosing which electrodes to\nstimulate based on probability estimates from previous measurements improved ef-\n\ufb01ciency. Similar improvements resulted from using either non-adaptive stimulation\nwith a joint model across cells, or adaptive stimulation with an independent model\nfor each cell. Finally, image reconstruction revealed that these improvements may\ntranslate to improved performance of an arti\ufb01cial retina.\n\nIntroduction\n\n1\nRecent advances in large-scale electrical and optical recording have made it possible to record and\nstimulate neural circuits at unprecedented scale and resolution [Jun et al., 2017, Kipke et al., 2008,\nKerr and Denk, 2008, Stosiek et al., 2003]. These advances suggest the possibility of using electronic\ndevices to restore functions lost to disease, or to augment human capacities [Wilson et al., 1991,\nSchwartz, 2004]. One such application is an arti\ufb01cial retina, which can provide a treatment for\nincurable blindness by electrically stimulating retinal ganglion cells, the output neurons of the retina\n[Stingl et al., 2013, Humayun et al., 2012, Lorach et al., 2015]. A high-\ufb01delity device must encode a\nvisual scene by electrically stimulating retinal neurons in a way that produces accurate and useful\nvisual perception (Figure 1A).\n\n\u2217Code: https://github.com/Chichilnisky-Lab/shah-neurips-2019\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fHowever, to achieve this goal, the device must control the precise, asynchronous patterns of activity\ntransmitted by multiple ganglion cell types to the brain. This will require \ufb01rst identifying the location\nand type of many individual ganglion cells in the patient\u2019s retina, then characterizing their responses\nto electrical stimulation. Previously, it has been shown that the location and types of individual retinal\nganglion cells can be identi\ufb01ed using recorded activity [Richard et al., 2015]. However, ef\ufb01cient\ncharacterization of electrical responses remains unsolved.\nFor example, in ex vivo experiments with primate retina, intended as a lab prototype for an arti\ufb01cial\nretina, characterization of neural response to stimulation of each of 512 electrodes individually\n(to avoid nonlinear interactions) requires about an hour, and scales linearly with the number of\nelectrodes. Thus, with the advent of arrays that can stimulate thousands of electrodes [Dragas et al.,\n2017] with multi-electrode current patterns [Fan et al., 2018], naive response calibration may be too\ntime-consuming for the clinic.\n\nFigure 1: (A) Functional components of a retinal prosthesis. (B) Different steps in adaptive character-\nization of electrical response properties.\nIn the present work, new methods are proposed to ef\ufb01ciently characterize the electrical stimulation\nproperties of a retinal interface, in a manner that may extend to other neural systems. Three novel\nsteps are presented to calibrate the interface, based on the voltage recorded in response to electrical\nstimulation (Figure 1B):\n\n\u2022 Spike sorting in the presence of electrical artifact: We develop a novel approach to\nestimate electrical artifacts, a key hurdle in spike sorting [Mena et al., 2017], in a subspace\nidenti\ufb01ed from past experimental data.\n\u2022 Response modeling: We develop a model of electrical response properties of the target\ncells; this model incorporates as a prior the observed relationship between recorded spike\namplitude and the threshold for electrical stimulation of a cell on a given electrode, from\npast experimental data.\n\u2022 Adaptive stimulation: Inspired by previous work [Lewi et al., 2009, Shababo et al., 2013],\nwe develop a method to exploit the data already recorded from a retina to optimize the\nchoice of stimulation patterns for the next batch of measurements.\n\nFinally, to make these methods most relevant for arti\ufb01cial vision, a modi\ufb01cation is presented that\nminimizes error in the reconstructed visual stimulus, a more meaningful indicator of performance\nthan the recorded spike counts.\nBelow, each algorithm is described separately in detail, with the results presented in aggregate after.\nNote that for simplicity, distinct notation is used for each algorithm; this notation is de\ufb01ned in the\nbeginning of each section.\n2 Spike sorting in the presence of stimulation artifacts\nThe goal of spike sorting is to identify spikes \ufb01red in response to electrical stimulation, and to distin-\nguish spikes produced by different neurons. Spike sorting in the presence of electrical stimulation\nis dif\ufb01cult because the recorded spike voltages are corrupted by large stimulation artifacts with\nmagnitude and duration comparable to those of the spike waveforms (see for [Mena et al., 2017] for\nexamples). Hence, artifacts must be subtracted before identifying spikes. After artifact subtraction,\nspikes are identi\ufb01ed based on waveforms previously recorded in the absence of electrical stimulation.\nPrevious work from [O\u2019Shea and Shenoy, 2018] estimated the stimulation artifact by exploiting\nthe artifact similarity for a given stimulation electrode, across different pulses, trials and recording\n\n2\n\nABdecodingencodingelectrical responsestarget electrical stimulation neural responses perceptionspike sortingresponse modelingadaptive stimulation\felectrodes, but did not assign spikes to cells. [Mena et al., 2017] used previously recorded spike\nwaveforms to jointly estimate the cellular activity and artifacts, assuming a smooth change in artifact\nwith increasing currents. In comparison, this work identi\ufb01es spikes by exploiting artifact similarity\nacross stimulating electrodes and experiments, eliminating the need to track and separate spikes and\nartifact over a range of current levels, substantially reducing data requirements.\nLet (cid:126)ya,r \u2208 RL be the L dimensional recorded data on electrode r when the stimulating electrode\nhas amplitude a (L is the number of timesteps considered following the electrical stimulus). Using\nthe artifacts estimated by applying the algorithm in [Mena et al., 2017] on previous experiments,\nan n dimensional subspace Aa,d(r,e) \u2208 RL\u00d7n is estimated for each stimulation amplitude (a) and\ndistance d(r, e) between the recording and stimulating electrodes. Hence the artifact is modeled as\nAa,d(r,e)(cid:126)ba,r, with (cid:126)ba,r \u2208 Rn estimated from the recorded data.\nLet (cid:126)xc,a \u2208 {0, 1}L be the spiking activity of cell c and Wc,r \u2208 RL\u00d7L be the Toeplitz matrix\nconsisting of shifted copies of a previously identi\ufb01ed spike waveform on electrode r. Each neuron has\nat most one spike during the recording interval after stimulation, and the amplitude is exactly 1 when\nit spikes. This constraint is incorporated approximately by a softmax parameterization of (cid:126)xc,a with\nan auxillary parameter qc,a allowing for the possibility of no spike and temperature \u03c4 determining the\nquality of approximation. Since neurons \ufb01re sparsely, an L1 norm penalty is applied on xc,a as well.\nThe artifact parameters (cid:126)b and spike assignments (cid:126)x are estimated by minimizing the penalized\nreconstruction error (Lspike-sort) for a particular stimulating electrode e, the recorded voltage traces on\nmultiple recording electrodes, and all the stimulating amplitudes simultaneously:\n\n(cid:88)\nc (cid:107)(cid:126)xc,a(cid:107)1.\n\nWc,r(cid:126)xc,a)(cid:107)2\n\n2 + \u03bbL1\n\n(1)\n\n(cid:88)\n\na\n\n(cid:88)\nr (cid:107)(cid:126)ya,r \u2212 (Aa,d(r,e)(cid:126)ba,r +\n\n(cid:88)\n\nc\n\nLspike-sort =\n\nFor the results presented here, L = 55, n = 9, and cells with large amplitude on the stimulating\nelectrode were used for spike sorting (roughly 10 cells per electrode). See Results and Appendix for\ndetails.\n\n3 Response modeling\nGiven the spikes recorded in response to electrical stimulation, the goal of response modeling is\nto estimate the activation probability for each cell and electrode pair. The standard method, which\ninvolves estimating the response probability for each cell-electrode pair independently is presented\n\ufb01rst, followed by a joint model that incorporates priors from previous experiments.\nn=1 are\nFor estimating these models, N samples of electrical stimulus-response pairs {en, an, cn}n=N\ngiven, with stimulating electrode en \u2208 {1,\u00b7\u00b7\u00b7 , Ne}, activated cell cn \u2208 {1,\u00b7\u00b7\u00b7 , Nc}, and current\nlevel an \u2208 {1,\u00b7\u00b7\u00b7 , Na}.\n3.1\nIndependent model\nThis model assumes that there is no consistent relationship between recorded spikes and stimulation\nthreshold across cell-electrode pairs. Thus, for each sample, the spiking probability is modeled as\na Bernoulli distribution P (Rn = 1) := \u03b3en,an,cn =\n1+e\u2212(pen ,cn (an\u2212qen,cn )) , where pen,cn , qen,cn\nare the parameters of the sigmoidal activation curve for the stimulating electrode en and cell cn.\nThe parameters are inferred independently by using standard methods for maximizing the logistic\nlog-likelihood for each cell electrode pair.\n\n1\n\nJoint model\n\n3.2\nUsing prior data on the relationship between recording and stimulation from previously recorded\nretinas could lead to more ef\ufb01cient characterization of activation probabilities. Previous work\n[Madugula et al., 2017] suggests that for a given electrode, the recorded spike amplitude and the\nstimulation threshold are inversely related. The inverse relationship lies on different curves for axonal\nand somatic activation due to differences in channel density and geometry (Figure 3A). This section\npresents a model that jointly models this relationship across multiple cell-electrode pairs.\nIn the model, the activation threshold qe,c is related to the spike amplitude (Ee,c) using a reciprocal\nrelationship, different for somatic or axonal activation (Te,c) but common for all cell-electrode pairs\n\n3\n\n\fin a given retina (Equation 2). A Gaussian prior on the parameters of the reciprocal relationship (x, y)\nis derived from previously recorded retinas (Equation 3, see Results):\n\nqe,c \u223c N (xTe,c +\n\nyTe,c\nEe,c\n\n, \u03bd2)\n\n{xT , yT} \u223c N (\u00b5T , \u03a3T ); T \u2208 {soma, axon}.\n\nHence, the parameters of the model are given by\n\u0398 = {{pe,c, qe,c}e=Ne,c=Nc\nand the resulting model likelihood (Lmodel) given by\n\ne=1,c=1\n\n;{xj, yj}j\u2208{soma,axon}, \u03bd}\n\n(2)\n\n(3)\n\n(4)\n\n(5)\n\nLmodel = \u03a0nP (Rn|an; pen,cn , qen,cn )\u03a0e,cP (qe,c|Ee,c; xTe,c , yTe,c, \u03bdTe,c )\n\n\u03a0i\u2208{soma,axon}P (xi, yi|\u00b5i, \u03a3i).\nn=1 ). \u03bd is learned but\nThe goal is to estimate the posterior over the parameters P (\u0398|{Rn, en, an, cn}n=N\nnon-random, and other parameters are estimated by variational approximation of the posterior [Blei\net al., 2017, Wainwright et al., 2008]. The posterior is approximated using a Gaussian mean-\ufb01eld\nvariational distribution. This approximation is estimated by maximizing the evidence lower bound\n(ELBO) on the log-likelihood using the reparametrization trick [Kingma and Welling, 2013]. See\nAppendix for details.\n4 Adaptive stimulation\nIn this section, the goal is to develop an algorithm that uses responses from prior stimulation within\na retina to choose subsequent stimulation patterns, in closed loop. Since real-time closed loop\nexperiments are generally not feasible using existing hardware, the experiment is assumed to run in\nmultiple phases, with the algorithm choosing the entire collection of current patterns to stimulate in\nthe next phase. The \ufb01rst phase is non-adaptive: each electrode and amplitude is stimulated T times,\nfor a total of NeNaT . In subsequent phases, parameter estimates from earlier phases are used to\nallocate a total of NeNaT stimuli unevenly across electrodes and amplitudes.\nThe number of stimuli for each electrode and amplitude, Te,a \u2208 Z+, is computed by minimizing a loss\nfunction L, that depends on the estimation accuracy of the stimulation probabilities. In this paper, the\nloss function is chosen as total variance in the estimate of response probability across electrodes and\ne,a,c var(\u03b3e,a,c), where var(\u03b3e,a,c) denotes the variance in estimate of activation\nprobability. This condition is identical to A-optimality in optimal design literature [Atkinson et al.,\n2007], departing from the commonly used information-theoretic methods in neuroscience [Lewi\net al., 2009, Paninski et al., 2007]; since the stimulation algorithms considered here choose only\none electrode-amplitude combination at a time (Section 5), it is not necessary to account for the\ndependence of estimation error between different probabilities as in D-optimality.\nFor adaptively choosing the stimulations, the estimation error in response probabilities \u03b3e,a after\nTe,a additional stimulations must be computed. Given the variance in estimate of the sigmoid\nparameters (\u03b8e,c), the error in probabilities at individual current levels is given using Taylor expansion\ne,a,c, where f(cid:48) is the sigmoid derivative. If T (cid:48)\ne,a are the number\nas var(\u03b3e,a,c) \u2248 f\nof previous stimulations, the variance of sigmoid parameters after (Te,a + T (cid:48)\ne,a) stimulations is\napproximated using the inverse of the resulting Fisher information I(\u03b8)\u22121. More concretely, the\nasymptotic variance of maximum likelihood estimate \u02c6\u03b8 is related to the inverse Fisher information\ncomputed using true parameter \u03b8. However as \u03b8 is unknown, the inverse Fisher information is\napproximated using the estimated parameters I(\u02c6\u03b8e,c). The resulting optimization problem is:\n\namplitudes L =(cid:80)\n\ne,a,cvar(\u03b8e,c)f(cid:48)\n(cid:48)T\n\n(cid:88)\n\n(cid:48)T\ne,a,c[I(\u02c6\u03b8e,c)]\nf\n\n\u22121f\n\n(cid:48)\ne,a,c\n\nLadapt-stim =\n\nTe,a\n\nminimize\n\nsubject to (cid:88)\n\ne,a,c\n\n(6)\n\nTe,a \u2264 NeNaT,\n\nTe,a \u2265 0 \u2200e, a.\n\ne,a\n\nA soft-max representation of Te,a is used to minimize the unconstrained problem and the \ufb01nal\napproximate solution is quantized to give an exact (integer) solution (full details in Appendix).\n\n4\n\n\f5 Evaluation for arti\ufb01cial retina application\n\nTo optimize and evaluate these techniques in a manner that most accurately captures the function of\nan arti\ufb01cial retina, a metric for approximating the expected impact on visual function is developed.\nRecent work [Shah et al., 2019] proposed that the perception from a collection of current patterns\nmay be added linearly when they are combined by sequential stimulation at a rate faster than visual\nintegration times. The stimulation sequence is chosen such that the response to each stimulation\npattern is independent, and the \ufb01nal perception depends only on the total number of spikes generated in\na temporal integration window. In this framework, the contribution of response probability estimation\nerror from each cell-electrode pair to the accuracy of perception is estimated. Let R\u03b3 denote the\nobserved spike response, given that the spiking probability is \u03b3. When (cid:126)dc denotes the linear decoding\n\ufb01lter associated with activation of cell c, the cell\u2019s contribution to perception is given by (cid:126)dcR\u03b3. The\nexpected change in perception when the cell response is generated by estimated probability \u02c6\u03b3 instead\nof the true probability \u03b3 is given by E\u02c6\u03b3,R(cid:107) (cid:126)dc(R\u02c6\u03b3 \u2212 R\u03b3)(cid:107)2 = (cid:107)(cid:126)dc(cid:107)2(var(R\u02c6\u03b3) + var(R\u03b3) + var(\u02c6\u03b3)),\nassuming R\u03b3 and R\u02c6\u03b3 are independent. The effect of inaccurate response probability estimation is\nmainly accounted for by the last term: (cid:107) (cid:126)dc(cid:107)2var(\u02c6\u03b3).\nIn section 6.4, the performance for the retinal prosthesis application is evaluated using the above mea-\nsure. The variance is either computed from the independent response model or the variational approx-\nimation of the joint response model. For adaptive stimulation, the optimization problem in Equation\n2var(\u02c6\u03b3e,a,c).\n\n6 is modi\ufb01ed by weighing each term by the strength of the decoder(cid:80)\n\n(cid:80)\n\n(cid:80)\nc (cid:107)dc(cid:107)2\n\ne\n\na\n\n6 Results\nExtracellular recording and stimulation of primate retinal ganglion cells ex vivo using a 512-electrode\ntechnology [Litke et al., 2004, Frechette et al., 2005] were used to evaluate the performance of the\nalgorithms. First, recorded voltages from 30 minutes of visual stimulation were spike sorted using\ncustom software. The estimated spatio-temporal spike waveform for each cell was identi\ufb01ed by\naveraging the recorded voltage waveforms over thousands of recorded spikes. Spike amplitude was\nmeasured on each electrode as the maximum negative voltage deviation, and spike shape was used\nto determine if the electrode was recording from the soma (biphasic waveform) or axon (triphasic\nwaveform). Subsequently, electrical stimulation experiments were performed [Jepson et al., 2013]\nby passing brief (\u223c 0.1ms), weak (\u223c 1\u00b5A)) current pulses repeatedly through each electrode\nindividually to identify the probability of eliciting a spike.\n\nFigure 2: Spike sorting (A) Artifact recorded for a 0.68\u00b5A triphasic current pulse, on electrodes at\ndifferent distances on a 60\u00b5m hexagonal grid. Lines in same column correspond to artifacts from\ndifferent retinal preparations but the same electrode, lines with the same color indicate recordings\nfrom different pieces of retina from the same animal. (B) Comparison of estimated current index\nfor 50% spiking probability for the algorithm (y-axis) and manual analysis (x-axis) for the new\nmethod (red) and the simpli\ufb01ed method from Mena et al. [2017] (baseline, blue). Each of the 241\ndots represents a cell-electrode pair. (C) Comparison of spike sorting results on 5 trials, when the\n5 trials were analyzed independently, and as part of a total of 24 trials. Histogram across multiple\ncell-electrode pairs for the new method (red) and a simple form of a previous approach (blue) [Mena\net al., 2017].\n\n5\n\nBACelec 1elec 2elec 1elec 2elec 1elec 20 micronsretinas01020304005101520253035400102030400510152025303540manual analysis (current index)algorithm (current index)new methodbaseline60 microns120 micronsdistancesfraction of trials with matched results between 5 and 24 trialsnew methodbaseline10-210-11001010.40.50.60.70.80.91.0number of cell-electrode pairs (log10)comparison of activation tresholdrobustness to fewer trialsartifact similarity \f6.1 Spike sorting in the presence of stimulation artifacts\nThe performance of spike sorting was evaluated with voltage traces recorded in response to repeated\nelectrical stimulation. For each of the recorded traces, stimulation artifacts were estimated by applying\nthe simpli\ufb01ed algorithm previously proposed in [Mena et al., 2017]. Brie\ufb02y, the artifact estimate is\ninitialized to the results obtained with a lower current amplitude, and then is updated by iterating\nbetween greedy spike estimation and artifact estimation. The estimated artifacts for different relative\nlocations of stimulating and recording electrodes, across multiple experiments, are shown in Figure\n2A. For a given stimulation current and a \ufb01xed distance between stimulating and recording electrodes,\nthe artifact was similar across different stimulating electrodes, and across recordings.\nImproved spike sorting was therefore implemented using the reduced space to regularize the artifact\nestimates. Performance was tested on the responses of one retina, with the artifact waveform\nbasis estimated from a 9-dimensional approximation of data from 22 recordings (>99% of variance\nexplained; see Appendix, Figure 6). Analysis of 24 repetitions for each electrode and amplitude\nrevealed that the estimated activation threshold (current value that elicits a spike with probability\n0.5) matched the value obtained with human analysis to within 67% in 161/ 241 of cell-electrode\npairs (Figure 2B, left). In some cases the algorithm produced higher thresholds than the human, and\nvice-versa, but no large overall bias was observed. These results were comparable to the value of\n173/241 of pairs obtained with the baseline method (Figure 2B, right). Thus, the new and baseline\nmethods exhibited similar accuracy. To test whether the new method showed improved ef\ufb01ciency,\nboth algorithms were applied in two cases: using 5 trials, or 24 trials, per stimulation pattern. The\nnew approach resulted in greater consistency: spike times identi\ufb01ed using fewer trials matched the\nspike times identi\ufb01ed using more trials more closely than with the baseline method (Figure 6C) Thus,\nincorporating priors on artifact waveforms across recordings can allow for effective spike detection\nwith limited data.\n6.2 Response modeling\n\nFigure 3: Response models (A) Relationship between observed spike amplitude (horizontal axis) and\nactivation threshold (vertical axis) for electrodes recording from soma (blue) or axon (red). Each dot\ncorresponds to a cell-electrode pair. Thin lines indicate \ufb01ts obtained by different random samplings\nof a subset of cells (B) Squared error in probability estimates, averaged over multiple cell-electrode\npairs (vertical axis) with different number of measurements using different methods. Measurements\nare done in batches, with a total of 2NeNa measurements per batch. (C) Estimated spike probability\nfor a few cell-electrode pairs, showing improvement using the proposed methods.\n\nTo improve the ef\ufb01ciency of response model estimation, the effect of imposing priors on electrical\nproperties of cells was examined. To test the impact of priors with ground truth and separately from\nspike sorting, neural responses were simulated based on activation probabilities estimated from a\npreviously recorded retina. Two models were compared: (a) an independent model, in which spike\nprobabilities are estimated without priors; (b) a joint model, in which spike probabilities are estimated\nwith priors.\nThe key concept used for priors is that electrical stimulation thresholds for a cell on different electrodes\nshould bear some relationship to the magnitude of the spike recorded on those electrodes, because\nboth threshold and spike amplitude should depend inversely on the physical distance to the spike\ninitiation region of the cell. Thus, for the joint model, the relationship between threshold and spike\namplitude for both axonal and somatic activation was learned from a set of 199 cell-electrode pairs\nacross 3 data sets (Figure 3A), using the spike sorting algorithm in [Mena et al., 2017]. The mean and\nvariance of the parameters of the inverse relationship were identi\ufb01ed by \ufb01tting curves with randomly\nsampled cell-electrode pairs.\n\n6\n\nAB510152025!0.020.040.060.080.10average number of trials<(\u02c6pp)2>(null)(null)(null)(null)C0501001502002501521273339activation thresholdspike amplitudeactivation probabilitycurrent index10adaptivejointindependentground truth24323223adaptivejointindependentdataperformanceexamplessoma!axon3226\fThe impact of using this prior was evaluated by measuring the responses from the simulated retina in\nbatches, where an average of 2 measurements per electrode and amplitude were delivered in each\nbatch. As the number of measurements increased, the joint model (green, Figure 3B, examples\nin Figure 3C and additional data set in Figure 7A) produced estimates of spike probability which\nwere more accurate than estimates made by the independent model (black curve). Thus, a prior\ncapturing the inverse relationship between spike amplitude and threshold can improve estimation of\nthe response model.\n6.3 Adaptive stimulation\n\nFigure 4: Adaptive stimulation (A) Adaptive stimulation with different batch sizes (red lines) and\nnon-adaptive stimulation (black), both with the independent response model. Subsequent panels\nindicate spatial distribution of the electrodes selected by the adaptive algorithm. (B) First phase: All\nelectrodes and amplitudes are selected uniformly (2 times each). Cell soma (red circle) and axon\ndirection (red lines) are estimated from detected spike waveform across electrodes. (C) Second phase:\nThe increase (yellow) and decrease (blue) in number of measurements in second (\ufb01rst adaptive) phase\ncompared to the \ufb01rst (non-adaptive) phase. Size of circle indicates the magnitude of deviation. (D, E)\nSuccessive difference in the number of measurements for each electrode.\n\nAn alternative to regularizing based on previous retinas is to use feedback from measurements already\nmade in the target retina to select the next electrical stimulus, and thus potentially improve estimation\nin closed loop. The effectiveness of this approach was tested using simulated data, to allow for\nmore repetitions of electrical stimulation than were present in recorded data. An adaptive algorithm\nwas developed using two stimuli on average per electrode and amplitude for each batch (2NeNa\nmeasurements in total). After the \ufb01rst non-adaptive phase, the adaptive algorithm divides all the\navailable capacity in the next batch across stimulations to minimize estimation error. With the simpler\nindependent model, adaptive stimulation gave lower estimation error compared to the non-adaptive\nmethod (Figure 3B, red). The adaptive method with the independent model and the non-adaptive\nmethod with the joint model exhibited similar performance, suggesting that, with suf\ufb01cient data, the\ncomputationally expensive adaptive stimulation can be replaced with better priors in non-adaptive\nstimulation. The adaptive method with the joint model performed similarly to the adaptive model\nwith the independent model (not shown), possibly because the contribution of prior from previous\nexperiments is reduced with better stimulus selection. The estimated response probabilities as a\nfunction of stimulation current for a few cell-electrode pairs after three phases of each approach are\nshown in Figure 3C, again showing similar performance of the two approaches.\nIn any adaptive method, a reduction in the number of adaptive phases reduces the computational\nburden, but could also reduce performance. To explore this tradeoff, the algorithm was tested with a\nsmaller number of adaptive phases, and a corresponding increase in stimulation capacity per phase. A\nsmall number of adaptive phases typically yielded high estimation accuracy; for example, two phases\nwith batch size 5 each yielded similar accuracy as 10 phases with batch size 1 each (Figure 4A).\nAdaptive stimulation also revealed systematic spatial structure in the electrodes selected for stim-\nulation. Compared to the uniform stimulation in the \ufb01rst (non-adaptive) phase, the electrodes on\nlower left side of the array were stimulated more frequently in the second (adaptive) phase (Figure\n4C). This could potentially be explained by the geometry of the axons in the recording, which cross\nthe array in a particular direction as they head toward the optic nerve (Figure 4B). Since cells could\neither be stimulated directly at the soma or indirectly at the axon, electrodes on the side of the array\n\n7\n\nphase 1: uniformphase 2 - phase 1phase 3 - phase 2phase 4 - phase 3+-BCDEspatial pattern of selected electrodes5101520250.0250.0750.1250.175average number of trials<(\u02c6pp)2>(null)(null)(null)(null)non-adaptiveadaptive batch: 1adaptive batch: 2adaptive batch: 3adaptive batch: 4adaptive batch: 5adaptive batch: 7adaptive batch: 10Aperformance\fwith more axons would potentially stimulate more cells on average. Thus, these electrodes would\ncontribute more to reducing error in estimation of response probabilities, and the adaptive algorithm\npreferentially selects them. However, in the third (adaptive) phase, the algorithm corrects itself and\nselects electrodes with fewer stimulated cells, presumably because estimation error remains high\nin those cells (Figure 4D). For subsequent phases, there is no obvious spatial structure in selected\nelectrodes, presumably because the residual estimation is now similar across electrodes (Figure 4E).\nThese observations were replicated in another retina (Figure 7B,C,D,E).\n6.4 Performance for neural interface\n\n<(cid:107)d(cid:107)2>\n\nFigure 5: Impact of electrical calibration on decoded stimulus (A) Contribution of imperfect prob-\nability estimates to error in linear decoding (y-axis), measured by variance in estimated probabilities,\nweighted by decoder norm ( <(cid:107)d(cid:107)2(p\u2212 \u02c6p)2>\n) as a function of the average number of stimulation pulses\n(per electrode and amplitude, x-axis) for the three methods presented in the text. Independent and\njoint model same as Figure 3, but stimuli for adaptive method chosen using the modi\ufb01ed objective.\nBlack arrow indicates the trials at which the estimates are compared in (B). (B) Expected mean\nsquared error of linearly decoded stimulus when probabilities from different methods are used for\nchoosing the stimulation pattern. Each dot corresponds to a different white noise image. (C, D, E)\nThe improvement in the expected linearly decoded stimulus using different probability estimates\n(over the independent model).\n\nerror in the estimate of response probability is given by(cid:80)\n\nWhile the above techniques improve spike identi\ufb01cation, response model estimation, and stimulus\nselection, a larger issue is how effective these improvements are for the function of the neural\ninterface. In the case of vision restoration, performance ultimately depends on how each targeted cell\ncontributes to vision, and on how well an actual image can be represented in the collection of cells.\nTo test functional impact in a way that accounts for how cells contribute to perception, the adaptive\nmethod was used with a modi\ufb01ed error metric. Previous work [Shah et al., 2019] proposes that\nan arti\ufb01cial retina could linearly combine the expected perception from stimulation of different\nelectrodes by temporally multiplexing within the integration time of the brain. Expected perception\nis inferred by assuming that the brain performs optimal linear reconstruction of the visual stimulus\nfrom retinal inputs. When electrode e is stimulated at amplitude a, the change in perception due to\nc (cid:107)dc(cid:107)2var(\u02c6\u03b3e,ac), where dc is the optimal\nlinear reconstruction \ufb01lter for cell c. To evaluate adaptive estimation in this framework, the adaptive\nalgorithm was modi\ufb01ed to minimize the error in visual stimulus reconstruction, across electrodes\nand amplitudes (see Section 5). Applying the modi\ufb01ed algorithm to simulated data revealed a faster\ndecrease in the stimulus reconstruction error that is attributable to response probability estimation,\ncompared to non-adaptive stimulation (Figure 5A). As before, the joint model with non-adaptive\nstimulation also outperformed the independent model.\nTo test functional impact in a way that captures variation in visual image structure, the spatial\nreconstruction of 20 different target images based on electrical stimulation was examined. Optimizing\nthe electrical stimulation using estimated response probabilities for both the adaptive and joint\n\n8\n\nadaptive - independentjoint - independentground truth - independentCDA510152025!0.0010.0020.0030.0040.0050.0060.007independentjointadaptiveaverage number of trialsEcontribution to decoding errorchange in achieved decodingperformanceB0.5500.6000.6500.7000.7500.550.600.650.700.75jointadaptiveground truthdecoding error with optimized stimulationindependent (relative MSE)different methods (relative MSE)\fcalibration algorithms (using the method in [Shah et al., 2019]) resulted in more accurate stimulus\nreconstruction compared to the independent model (Figure 5B,C,D). Thus, the gains from ef\ufb01cient\ncharacterization of electrical response properties are likely to translate into improved arti\ufb01cial vision.\n7 Summary\nThis paper presents three novel methods to optimize the function of a neural interface, speci\ufb01cally, an\narti\ufb01cial retina for treating blindness. Using large scale multi-electrode recordings from primate retina\nas a lab prototype, prior information and closed-loop approaches improved the accuracy and ef\ufb01ciency\nof spike sorting, response modeling, and stimulus selection. Notably, the computationally expensive\nclosed-loop stimulation approach exhibited similar performance to a much simpler non-adaptive\napproach that uses prior information from previous experiments, highlighting the value of using\nlarge data sets to improve device function. Evaluation of image reconstruction revealed that these\napproaches improved overall function in terms of the quality of the visual image transmitted to the\nbrain.\nIn principle, similar approaches may be useful in other neural systems (e.g. intra-Cortical micro-\nstimulation [Salzman et al., 1990] for proprioceptive feedback in motor prostheses [Salas et al., 2018])\nand in other neural interfaces (e.g. optical recording and stimulation [Shababo et al., 2013]). With the\nadvent of large-scale data sets, as well as the availability of motor and visual prosthesis technologies\nin many subjects, the methods developed here may be helpful in capturing similarities and differences\namong individuals and experiments.\n\n8 Future work\nFuture improvements in electrical response calibration are possible by addressing current technical\nlimitations and incorporating additional priors. For spike sorting, prior information about monotonic\nincrease of activation probabilities with increasing currents could be incorporated. For wider ap-\nplicability, the artifact estimation method should be extended to stimulation patterns that were not\ndelivered in previous experiments. For response modeling, priors on the relationship between spike\namplitude on multiple electrodes and activation threshold/slope differences between soma and axon\nmay be useful [Jepson et al., 2013, Fan et al., 2018].\nSeveral enhacements may be important in future work. First, for understanding the impact on arti\ufb01cial\nretina, analysis of linear image reconstruction [Warland et al., 1997, Stanley et al., 1999] could be\nextended to exploit more powerful nonlinear methods [Parthasarathy et al., 2017]. Second, it may be\nuseful to evaluate response modeling, adaptive stimulation, and spike sorting together rather than\nindependently. Finally, future work could focus on reducing computational cost in addition to the\nduration of electrical calibration.\n\nReferences\nA. Atkinson, A. Donev, and R. Tobias. Optimum Experimental Designs, with SAS. 01 2007. ISBN\n\n978-0-19-929659-0.\n\nD. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians.\n\nJournal of the American Statistical Association, 112(518):859\u2013877, 2017.\n\nJ. Dragas, V. Viswam, A. Shadmani, Y. Chen, R. Bounik, A. Stettler, M. Radivojevic, S. Geissler,\nM. E. J. Obien, J. M\u00fcller, et al.\nIn vitromulti-functional microelectrode array featuring 59\n760 electrodes, 2048 electrophysiology channels, stimulation, impedance measurement, and\nneurotransmitter detection channels. IEEE journal of solid-state circuits, 52(6):1576\u20131590, 2017.\n\nV. H. Fan, L. E. Grosberg, S. S. Madugula, P. Hottowy, W. Dabrowski, A. Sher, A. M. Litke, and\nE. Chichilnisky. Epiretinal stimulation with local returns enhances selectivity at cellular resolution.\nJournal of neural engineering, 2018.\n\nE. S. Frechette, A. Sher, M. I. Grivich, D. Petrusca, A. M. Litke, and E. Chichilnisky. Fidelity of the\nensemble code for visual motion in primate retina. Journal of neurophysiology, 94(1):119\u2013135,\n2005.\n\n9\n\n\fM. S. Humayun, J. D. Dorn, L. Da Cruz, G. Dagnelie, J.-A. Sahel, P. E. Stanga, A. V. Cideciyan,\nJ. L. Duncan, D. Eliott, E. Filley, et al. Interim results from the international trial of second sight\u2019s\nvisual prosthesis. Ophthalmology, 119(4):779\u2013788, 2012.\n\nL. H. Jepson, P. Hottowy, K. Mathieson, D. E. Gunning, W. D \u02dbabrowski, A. M. Litke, and\nE. Chichilnisky. Focal electrical stimulation of major ganglion cell types in the primate retina for\nthe design of visual prostheses. Journal of Neuroscience, 33(17):7194\u20137205, 2013.\n\nJ. J. Jun, N. A. Steinmetz, J. H. Siegle, D. J. Denman, M. Bauza, B. Barbarits, A. K. Lee, C. A.\nAnastassiou, A. Andrei, \u00c7. Ayd\u0131n, et al. Fully integrated silicon probes for high-density recording\nof neural activity. Nature, 551(7679):232, 2017.\n\nJ. N. Kerr and W. Denk. Imaging in vivo: watching the brain in action. Nature Reviews Neuroscience,\n\n9(3):195, 2008.\n\nD. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,\n\n2014.\n\nD. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114,\n\n2013.\n\nD. R. Kipke, W. Shain, G. Buzs\u00e1ki, E. Fetz, J. M. Henderson, J. F. Hetke, and G. Schalk. Advanced\nneurotechnologies for chronic neural interfaces: new horizons and clinical opportunities. Journal\nof Neuroscience, 28(46):11830\u201311838, 2008.\n\nJ. Lewi, R. Butera, and L. Paninski. Sequential optimal design of neurophysiology experiments.\n\nNeural Computation, 21(3):619\u2013687, 2009.\n\nA. Litke, N. Bezayiff, E. Chichilnisky, W. Cunningham, W. Dabrowski, A. Grillo, M. Grivich,\nP. Grybos, P. Hottowy, S. Kachiguine, et al. What does the eye tell the brain?: Development of\na system for the large-scale recording of retinal output activity. IEEE Transactions on Nuclear\nScience, 51(4):1434\u20131440, 2004.\n\nH. Lorach, G. Goetz, R. Smith, X. Lei, Y. Mandel, T. Kamins, K. Mathieson, P. Huie, J. Harris,\nA. Sher, et al. Photovoltaic restoration of sight with high visual acuity. Nature medicine, 21(5):\n476, 2015.\n\nS. Madugula, G. E. Mena, L. E. Grosberg, V. H. Fan, P. Hottowy, W. Dabrowski, A. Sher, A. Litke,\nL. Paninski, and E. Chichilnisky. Large-scale analysis of patterned epiretinal stimulation for\nprosthesis design. TEATC, 2017.\n\nG. E. Mena, L. E. Grosberg, S. Madugula, P. Hottowy, A. Litke, J. Cunningham, E. Chichilnisky,\nand L. Paninski. Electrical stimulus artifact cancellation and neural spike detection on large\nmulti-electrode arrays. PLoS computational biology, 13(11):e1005842, 2017.\n\nD. J. O\u2019Shea and K. V. Shenoy. Eraasr: an algorithm for removing electrical stimulation artifacts\n\nfrom multielectrode array recordings. Journal of neural engineering, 15(2):026020, 2018.\n\nL. Paninski, J. Pillow, and J. Lewi. Statistical models for neural encoding, decoding, and optimal\n\nstimulus design. Progress in brain research, 165:493\u2013507, 2007.\n\nN. Parthasarathy, E. Batty, W. Falcon, T. Rutten, M. Rajpal, E. Chichilnisky, and L. Paninski. Neural\nnetworks for ef\ufb01cient bayesian decoding of natural images from retinal neurons. pages 6434\u20136445,\n2017.\n\nE. Richard, G. A. Goetz, and E. Chichilnisky. Recognizing retinal ganglion cells in the dark. In\n\nAdvances in Neural Information Processing Systems, pages 2476\u20132484, 2015.\n\nM. A. Salas, L. Bashford, S. Kellis, M. Jafari, H. Jo, D. Kramer, K. Shan\ufb01eld, K. Pejsa, B. Lee,\nC. Y. Liu, et al. Proprioceptive and cutaneous sensations in humans elicited by intracortical\nmicrostimulation. Elife, 7:e32904, 2018.\n\nC. D. Salzman, K. H. Britten, and W. T. Newsome. Cortical microstimulation in\ufb02uences perceptual\n\njudgements of motion direction. Nature, 346(6280):174, 1990.\n\n10\n\n\fA. B. Schwartz. Cortical neural prosthetics. Annu. Rev. Neurosci., 27:487\u2013507, 2004.\n\nB. Shababo, B. Paige, A. Pakman, and L. Paninski. Bayesian inference and online experimental\ndesign for mapping neural microcircuits. In Advances in Neural Information Processing Systems,\npages 1304\u20131312, 2013.\n\nN. P. Shah, S. Madugula, G. Mena, P. Tandon, P. Hottowy, A. Sher, A. Litke, S. Mitra, and\nE. Chichilnisky. Optimization of electrical stimulation for a high-\ufb01delity arti\ufb01cial retina. In-\nternational IEEE EMBS conference on Neuroengineering, 2019.\n\nG. B. Stanley, F. F. Li, and Y. Dan. Reconstruction of natural scenes from ensemble responses in the\n\nlateral geniculate nucleus. Journal of Neuroscience, 19(18):8036\u20138042, 1999.\n\nK. Stingl, K. U. Bartz-Schmidt, D. Besch, A. Braun, A. Bruckmann, F. Gekeler, U. Greppmaier,\nS. Hipp, G. H\u00f6rtd\u00f6rfer, C. Kernstock, et al. Arti\ufb01cial vision with wirelessly powered subretinal\nelectronic implant alpha-ims. Proceedings of the Royal Society B: Biological Sciences, 280(1757):\n20130077, 2013.\n\nC. Stosiek, O. Garaschuk, K. Holthoff, and A. Konnerth. In vivo two-photon calcium imaging of\nneuronal networks. Proceedings of the National Academy of Sciences, 100(12):7319\u20137324, 2003.\n\nM. J. Wainwright, M. I. Jordan, et al. Graphical models, exponential families, and variational\n\ninference. Foundations and Trends R(cid:13) in Machine Learning, 1(1\u20132):1\u2013305, 2008.\n\nD. K. Warland, P. Reinagel, and M. Meister. Decoding visual information from a population of retinal\n\nganglion cells. Journal of neurophysiology, 78(5):2336\u20132350, 1997.\n\nB. S. Wilson, C. C. Finley, D. T. Lawson, R. D. Wolford, D. K. Eddington, and W. M. Rabinowitz.\n\nBetter speech recognition with cochlear implants. Nature, 352(6332):236, 1991.\n\n11\n\n\f9 Appendix\n\n9.1 Details on spike sorting in the presence of stimulation artifacts\n\nHere, the spike sorting procedure presented in Section 2 is discussed in detail.\nLet (cid:126)ya,r \u2208 RL be the recorded waveform of length L. The artifact is approximated in a subspace\nlearned from previous experiments. For a given amplitude a, and relative distance to recording\nelectrode d(s, r), an n dimensional basis is learned Aa,d(s,r). The artifact is thus reconstructed using\nan n dimensional learned parameter (cid:126)b \u2208 Rn as Aa,d(s,r)(cid:126)ba,r.\nLet (cid:126)xc,a \u2208 {0, 1}L be the spiking activity and Wc,r \u2208 RL\u00d7L be the matrix consisting of shifted\ncopies of a previously identi\ufb01ed spike waveform recorded on electrode r. The contribution of neural\nactivity from cell c to the recorded data is given by Wc,r(cid:126)xc,a.\nEach cell has at most one spike during this recording interval, and when it spikes the amplitude is\nexactly 1. This constraint is incorporated by parameterizing xc,a as a softmax function of real valued\n(cid:126)zc,a with temperature \u03c4 :\n\n(cid:80)\n\n(cid:126)xc,a,t =\n\nezc,a,t/\u03c4\n\nt ezc,a,t/\u03c4 + eqc,a/\u03c4\n\nwhere qc,a is an auxiliary parameter. Since only a few cells are activated in response to electrical\nstimulation, a sparsity enforcing L1 norm penalty is applied on (cid:126)x.\nThe artifact parameters (cid:126)b and spike assignments (cid:126)x are estimated by minimizing the penalized\nreconstruction error (Lspike-sort) for a particular stimulating electrode e, the recorded voltage traces on\nmultiple recording electrodes and all the stimulating amplitudes simultaneously:\n\n(cid:88)\n\na\n\n(cid:88)\nr (cid:107)(cid:126)ya,r \u2212 (Aa,d(r,e)(cid:126)ba,r +\n\n(cid:88)\n\nc\n\nLspike-sort =\n\n(cid:88)\nc (cid:107)(cid:126)xc,a(cid:107)1\n\nWc,r(cid:126)xc,a)(cid:107)2\n\n2 + \u03bbL1\n\nOptimization is performed using Adam (learning rate = 0.01), with \u03bbL1 chosen using cross-validation\nand temperature \u03c4 is reduced to 0.8 times its previous value every time the loss converges.\n\n9.2 Details on the joint response model\n\nHere, the joint response modeling procedure presented in Section 3.2 is discussed in detail.\n\n9.2.1 Model\nThe responses are denoted by Rn \u2208 {0, 1}. Similar to the independent model,\n\nP (Rn = 1) =\n\n1\n\n1 + e\u2212(pen,cn (an\u2212qen ,cn ))\n\n(7)\n\nwhere pen,cn, qen,cn are the parameters of the sigmoidal activation curve for the stimulating electrode\nen and cell cn.\nFor each cell c and electrode e, Ee,c \u2208 R+ denotes the recorded spike amplitude and Te,c denotes\nwhether the electrode e is recording from the soma or axon, as determined by the spike shape. The\nspike threshold qe,c is modeled as a Gaussian distribution, with a separate relationship for soma and\naxons:\n\nqe,c \u223c N (xTe,c +\n\nyTe,c\nEe,c\n\n, \u03bd2)\n\nFurther, the prior for {x, y} is modeled with a two dimensional Gaussian\n{xT , yT} \u223c N (\u00b5T , \u03a3T ); T \u2208 {soma, axon}\n\n12\n\n(8)\n\n(9)\n\n\fthe parameters of the model for electrically evoked responses are given by \u0398 =\n\nThe parameters for prior distribution (\u00b5T , \u03a3T ) are estimated from stimulated electrode and cell pairs\nin previous experiments.\nHence,\n{{pe,c, qe,c}e=Ne,c=Nc\n\u03a0nP (Rn|an; pen,cn, qen,cn )\u03a0e,cP (qe,c|Ee,c; xTe,c, yTe,c , \u03bdTe,c )\u03a0i\u2208{soma,axon}P (xi, yi|\u00b5i, \u03a3i).\n9.2.2 Inference\n\n;{xj, yj}j\u2208{soma,axon}, \u03bd} and the resulting model likelihood (Lmodel) is\n\ne=1,c=1\n\nThe goal is to estimate the posterior distribution of model parameters given the recorded data\nn=1 ). During inference, \u03bd is treated as non-random and other parame-\nP (\u0398|{Rn, en, an, cn}n=N\nters are estimated by variational approximation [Blei et al., 2017, Wainwright et al., 2008]. Let\nzpe,c , zqe,c, zxi, zyi represent the variational parameters corresponding to parameters in \u0398. A mean-\n\ufb01eld variational approximation of the posterior is learned\n\nP (\u0398|{Rn, en, an, cn}n=N\n\nn=1 ) \u2248 \u03a0e,cq(zpe,c )q(zqe,c )\u03a0i\u2208{soma,axon}q(zxi)q(zyi).\n\nThe parameters of the variational distribution (\u03c6) are estimated by maximizing the evidence lower\nbound (ELBO) on the log-likelihood (\u2212 log Lmodel):\n\n\u2212 log Lmodel \u2265 Eq(z) log P (R, Z) + H(q(z)).\n\n(10)\n\n(11)\n\nAs shown below, the \ufb01rst term of the joint probability corresponds to modeling electrically evoked\nspikes, the second term corresponds to modeling spike threshold from spike amplitudes, and the\nthird term corresponds to the relationship between spike threshold and spike amplitude for all the\ncell-electrode pairs within a retina:\n\nn=N(cid:88)\n\nn=1\n\n(cid:88)\n\ne,c\n\n+\n\nEq(z) log P (R, Z) =\n\nEq\u03c6(zpen ,cn ),q\u03c6(zqen ,cn ) log P (Rn, zpen,cn , zqen ,cn|an)\n\nEq\u03c6(zqe,c ) log P (zqe,c , zxTe,c , zyTe,c|Ee,c, \u03bdTe,c ) +\n\nEq\u03c6(zxi ),q\u03c6(zyi ) log P (zxi, zyi|\u00b5i, \u03c3i).\n(12)\n\n(cid:88)\n\ni\n\nqe,c\n\u00b5 , \u03c6\n\nqe,c\n\n\u00b5 , \u03c6xi\n\n\u03c32 ); q\u03c6(zxi) = N (\u03c6xi\n\nThe variational distributions are parameterized as Gaussians: q\u03c6(zpe,c ) = N (\u03c6\n\u03c32) and q\u03c6(zyi ) = N (\u03c6yi\nq\u03c6(zqe,c ) = N (\u03c6\nthe sum of Gaussian entropy corresponding to each variational parameter.\nFor maximizing the ELBO, the variational parameters are sampled using the re-parametrization\ntrick [Kingma and Welling, 2013]: z = \u03c6\u00b5 + \u03c6\u03c32 \u0001, \u0001 \u223c N (0, I). The empirical approximation of\nELBO is computed by averaging 10 samples for p, q and one sample for x, y. This approximation is\nmaximized by stochastic gradient descent with norm clipping. At each step of the stochastic gradient\ndescent, the objective function is evaluated over all samples, resulting in randomness only due to\nsampling of variational parameters. Finally, the posterior activation probabilties are estimated by\naveraging over 1000 random samples of zp, zq.\n\npe,c\n\u03c32 );\n\u03c32), with H(q) being\n\n\u00b5 , \u03c6yi\n\npe,c\n\u00b5 , \u03c6\n\n9.3 Details on adaptive stimulation\n\nHere, adaptive stimulus selection procedure presented in Section 4 is described in detail.\nThe goal is to minimize the total uncertainty in activation probability estimates over all electrodes,\namplitudes and cells (\u03b3e,a,c). The adaptive stimulation proceeds in batches, where a total of NeNaT\nstimulations must be unevenly divided across electrodes and amplitudes. Let Te,a \u2208 Z+ denote\nthe number of stimulations for electrode e and amplitude a in the next phase of the closed loop\nexperiment. Hence, the optimization problem to be solved after each batch is given by :\n\n(cid:88)\n\nvar(\u03b3e,a,c)\n\nLadapt-stim =\n\nTe,a\n\nminimize\n\nsubject to (cid:88)\n\ne,a\n\nTe,a \u2264 NeNaT,\n\nTe,a \u2265 0 \u2200e, a.\n\n13\n\ne,a,c\n\n(13)\n\n\f1\n\nDe\ufb01ne Xa = [a, 1] , \u03b8e,c = [pe,c,\u2212pe,cqe,c]. Under this notation, the activation probability \u03b3e,a,c =\ne,a denote the previous number of measurements for electrode\n1+exp(\u2212\u03b8T\ne and amplitude a. The Fisher information of \u03b8e,c after (Te,a + T (cid:48)\ne,a) stimulations is computed using\nthe chain rule as\n\ne,cXa) = f (\u2212\u03b8T\n\ne,cXa). Let T (cid:48)\n\nI(\u03b8e,c) =\n\n(Te,a + T\n\n(cid:48)\ne,a)\u03b3e,a,c(1 \u2212 \u03b3e,a,c)XaX T\na .\n\n(cid:88)\n\na\n\nThe asymptotic variance of the maximum likelihood estimate \u02c6\u03b8 is given by the inverse Fisher\ninformation at the true parameter values I(\u03b8e,c)\u22121. Finally, the \ufb01rst-order expansion of \u03b3e,a,c gives the\nvariance of individual parameter estimates as var(\u03b3e,a,c) \u2248 (f(cid:48))2var(\u03b8e,c) = QT\ne,a,cvar(\u03b8e,c)Qe,a,c,\nwhere Qe,a,c = \u03b3e,a,c(1 \u2212 \u03b3e,a,c)Xa. Normalizing the current levels a between \u22121 and 1 leads\nto better condition number for Fisher information and better approximation of the variance. After\nrelaxing the integer constraint on Te,a, the optimization problem is:\n\n(cid:88)\n\n(cid:48)\nQ\ne,a,c[\n\n(cid:88)\n\ne,a,c\n\nLadapt-stim =\n\nTe,a\n\nminimize\n\nsubject to (cid:88)\n\nTe,a \u2264 NeNaT,\n\ne,a\n\n(Te,a(cid:48) + T\n\na(cid:48)\nTe,a \u2265 0 \u2200e, a.\n\n(14)\n\n(15)\n\n(cid:48)\ne,a(cid:48))\u03b3e,a(cid:48),c(1 \u2212 \u03b3e,a(cid:48),c)Xa(cid:48)X\n\n(cid:48)\na(cid:48)]\n\n\u22121Qe,a,c\n\nAs the true parameter values are not unknown, the estimated probabilities in Equation 15 are\nused instead. These are estimated either by maximum likelihood on the independent model, or\nperforming variational inference on the joint model. Re-parameterization of Te,a converts the\nconstrained optimization problem into an unconstrained optimization problem. Speci\ufb01cally, a\nsoft-max representation of Te,a = (NeNaT )\ne(cid:48) /\u03c4 is used, where \u03c4 = 1000 is\nthe temperature parameter and \u03b4e is an auxillary parameter to allow for loose constraints. After\nminimizing the unconstrained problem using Adam optimization [Kingma and Ba, 2014], the exact\ninteger solution for Te,a is obtained by rounding.\n\nete,a /\u03c4\ne(cid:48),a(cid:48) /\u03c4\nt\n\n(cid:80)\n\ne(cid:48) ,a(cid:48) e\n\n+e\u03b4\n\n9.4 Approximation of artifacts in low dimensional space\n\nFigure 6: (A) Recorded artifacts (same as Figure 2A), (B) Reconstruction of artifacts from a 9\ndimensional subspace. The projection retains most of the structure in the variation of artifacts.\n\n14\n\nelec 1elec 2elec 1elec 2elec 1elec 20 microns60 microns120 micronsdistancesprojected artifactsretinaselec 1elec 2elec 1elec 2elec 1elec 20 microns60 microns120 micronsdistancesoriginal artifactsretinasAB\f9.5 Additional dataset\n\nFigure 7: Results for another retina. (A) Conventions as in Figure 3B. (B-E) Conventions as in Figure\n4(B-E).\n\n15\n\n+51015200.000.020.040.060.080.100.120.14independentjointadaptive<(\u02c6pp)2>(null)(null)(null)(null)average number of trials-spatial pattern of selected electrodesperformancephase 1: uniformphase 2 - phase 1phase 3 - phase 2phase 4 - phase 3ECBDA\f", "award": [], "sourceid": 8189, "authors": [{"given_name": "Nishal", "family_name": "Shah", "institution": "Stanford University"}, {"given_name": "Sasidhar", "family_name": "Madugula", "institution": "Stanford University"}, {"given_name": "Pawel", "family_name": "Hottowy", "institution": "AGH University of Science and Technology in Krak\u00f3w"}, {"given_name": "Alexander", "family_name": "Sher", "institution": "Santa Cruz Institute for Particle Physics, University of California, Santa Cruz"}, {"given_name": "Alan", "family_name": "Litke", "institution": "Santa Cruz Institute for Particle Physics, University of California, Santa Cruz"}, {"given_name": "Liam", "family_name": "Paninski", "institution": "Columbia University"}, {"given_name": "E.J.", "family_name": "Chichilnisky", "institution": "Stanford University"}]}