{"title": "Computation of Smooth Optical Flow in a Feedback Connected Analog Network", "book": "Advances in Neural Information Processing Systems", "page_first": 706, "page_last": 712, "abstract": null, "full_text": "Computation of Smooth Optical Flow in a \n\nFeedback Connected Analog Network \n\nAlan Stocker * \n\nInstitute of Neuroinforrnatics \nUniversity and ETH Zi.irich \n\nWinterthurerstrasse 190 \n8057 Zi.irich, Switzerland \n\nRodney Douglas \n\nInstitute of Neuroinforrnatics \nUniversity and ETH Zi.irich \n\nWinterthurerstrasse 190 \n8057 Zi.irich, Switzerland \n\nAbstract \n\nIn 1986, Tanner and Mead [1] implemented an interesting constraint sat(cid:173)\nisfaction circuit for global motion sensing in a VLSI. We report here a \nnew and improved a VLSI implementation that provides smooth optical \nflow as well as global motion in a two dimensional visual field. The com(cid:173)\nputation of optical flow is an ill-posed problem, which expresses itself as \nthe aperture problem. However, the optical flow can be estimated by the \nuse of regularization methods, in which additional constraints are intro(cid:173)\nduced in terms of a global energy functional that must be minimized . We \nshow how the algorithmic constraints of Hom and Schunck [2] on com(cid:173)\nputing smooth optical flow can be mapped onto the physical constraints \nof an equivalent electronic network. \n\n1 Motivation \n\nThe perception of apparent motion is crucial for navigation. Knowledge of local motion of \nthe environment relative to the observer simplifies the calculation of important tasks such as \ntime-to-contact or focus-of-expansion. There are several methods to compute optical flow. \nThey have the common problem that their computational load is large. This is a severe \ndisadvantage for autonomous agents, whose computational power is restricted by energy, \nsize and weight. Here we show how the global regularization approach which is necessary \nto solve for the ill-posed nature of computing optical flow, can be formulated as a local \nfeedback constraint, and implemented as a physical analog device that is computationally \nefficient. \n\n* correspondence to: alan@ini.phys.ethz.ch \n\n\fComputation of Optical Flow in an Analog Network \n\n707 \n\n2 Smooth Optical Flow \n\nHorn and Schunck [2] defined optical flow in relation to the spatial and temporal changes \nin image brightness. Their model assumes that the total image brightness E(x, y, t) does \nnot change over time; \n\nExpanding equation (1) according to the chain rule of differentiation leads to \n\no \n\nF == ox E(x, y, t)u + oy E(x, y, t)v + 8t E(x, y, t) = 0, \n\nd \ndt E(x, y, t) = O. \n\n0 \n\n0 \n\n(I) \n\n(2) \n\nwhere u = dx / dt and v = dy / dt represent the two components of the local optical flow \nvector. \nSince there is one equation for two unknowns at each spatial location, the problem is \nill-posed, and there are an infinite number of possible solutions lying on the constraint \nline for every location (x, y). However, by introducing an additional constraint the prob(cid:173)\nlem can be regularized and a unique solution can be found. \nFor example, Horn and Schunck require the optical flow field to be smooth. As a measure \nof smoothness they choose the squares of of the spatial derivatives of the flow vectors, \n\n(3) \n\nOne can also view this constraint as introducing a priori knowledge: the closer two points \nare in the image space the more likely they belong to the projection of the same object. Un(cid:173)\nder the assumption of rigid objects undergoing translational motion, this constraint implies \nthat the points have the same, or at least very similar motion vectors. This assumption is \nobviously not valid at boundaries of moving objects, and so this algorithm fails to detect \nmotion discontinuities [3]. \nThe computation of smooth optical flow can now be formulated as the minimization prob(cid:173)\nlem of a global energy functional, \n\nJ J ~ dx dy ---7 min \n\n(4) \n\nwith F and 8 2 as in equation (2) and (3) respectively. Thus, we exactly apply the approach \nof standard regularization theory [4]: \n\nL \n\nAx=y \nx = A -Iy \nII Ax - y II +.x II P 11= min \n\ny: data \ninverse problem, ill-posed \nregularization \n\nThe regularization parameter, .x, controls the degree of smoothing of the solution and its \ncloseness to the data. The norm, II . II, is quadratic. A difference in our case is that A \nis not constant but depends on the data. However, if we consider motion on a discrete \ntime-axis and look at snapshots rather than continuously changing images, A is quasi(cid:173)\nstationary.1 The energy functional (4) is convex and so, a simple numerical technique \nlike gradient descent would be able to find the global minimum. To compute optical flow \nwhile preserving motion discontinuities one can modify the energy functional to include \na binary line process that prevents smoothing over discontinuities [4]. However, such an \nfunctional will not be convex. Gradient descent methods would probably fail to find the \nglobal amongst all local minima and other methods have to be applied. \n\n1 In the a VLSI implementation this requires a much shorter settling time constant for the network \n\nthan the brightness changes in the image. \n\n\f708 \n\nA. Stocker and R. Doug/as \n\n3 A Physical Analog Model \n\n3.1 Continuous space \n\nStandard regularization problems can be mapped onto electronic networks consisting of \nconductances and capacitors [5]. Hutchinson et al. [6] showed how resistive networks can \nbe used to compute optical flow and Poggio et al. [7] introduced electronic network so(cid:173)\nlutions for second-order-derivative optic flow computation. However, these proposed net(cid:173)\nwork architectures all require complicated and sometimes negative conductances although \nHarris et al. [8] outlined a similar approach as proposed in this paper independently. Fur(cid:173)\nthennore, such networks were not implemented practically, whereas our implementation \nwith constant nearest neighbor conductances is intuitive and straightforward. \nConsider equation (4): \n\nL = L(u, v, '\\lu, '\\lv, x, y). \n\nThe Lagrange function L is sufficiently regular (L E C 2 ), and thus it follows from cal(cid:173)\nculus of variation that the solution of equation (4) also suffices the linear Euler-Lagrange \nequations \n\nA '\\l2u - Ex (Exu + Eyv + E t ) \nA'\\l2v - Ey(Exu + Eyv + E t ) \n\no \nO. \n\n(5) \n\nThe Euler-Lagrange equations are only necessary conditions for equation (4). The suffi(cid:173)\ncient condition for solutions of equations (5) to be a weak minimum is the strong Legendre(cid:173)\ncondition, that is \n\nL'ilu'ilu > 0 \n\nand \n\nL'ilv'ilv > 0, \n\nwhich is easily shown to be true. \n\n3.2 Discrete Space - Mapping to Resistive Network \n\nBy using a discrete five-point approximation of the Laplacian \\7 2 on a regular grid, equa(cid:173)\ntions (5) can be rewritten as \nA(Ui+1 )' + Ui-1 )' + Ui )'+1 + Ui )-1 - 4Ui )') - Ex, ,(Ex \nA(Vi+1)' +Vi - 1)' +Vi)'+1 +Vi)'-1 - 4Vi)' ) -Ey' (Ex, ,Ui)' +Ey' ,Vi)' +Et, ,)=0 \nwhere i and j are the indices for the sampling nodes. Consider a single node of the resistive \nnetwork shown in Figure 1: \n\n,Ui)' + E y' Vi)' + E t \n\n1 , ) ' . J ' \n\n1 ,1 ' \n\nl ,J ' \n\n' . ]\n\n' \n\n, \n\n, \n\n, \n\n, \n\n, \n\n, \n\n, \n\n, \n\n, \n\n, \n\n1,] \n\nt,] \n\n,) =0 (6) \n\n1 , J \n\nFigure 1: Single node of a resistive network. \n\nFrom Kirchhoff's law it follows that \n\ndV,\u00b7 , \n\nC d~') = G(Vi+1 ,j + Vi-I ,j + Vi,HI + Vi,j-1 - 4Vi,j) + lini.j \n\n(7) \n\n\fComputation of Optical Flow in an Analog Network \n\n709 \n\nwhere Vi ,j represents the voltage and l in', i the input current. G is the conductance between \ntwo neighboring nodes and C the node capacitance. \nIn steady state, equation (7) becomes \n\nG(Vi+I ,j + Vi - I ,j + Vi, j+! + Vi ,j- I - 4Vi ,j) + lini\" = O. \n\nThe analogy with equations (6) is obvious: \n\nG ~ .A \n\nlUin \u00b7\u00b7 ~ -Ex\u00b7 . (Ex \u00b7 UiJ' +Ey , ViJ' +Et\nlVin \" ~ -Ey . , (Ex \" UiJ, +Ey\" Vi),+Et\n\nt , ] ' \n\nt t ] \n\nt. ) \n\nt t )\n\n1 , )\n\n' \n\n' \n\n' \n\nt\n\n, } \n\nt\n\n, } \n\n1 , )\n\n\u00b7 ) \n\n1 , ) \n\n, ) \n\nI , J \n\n(8) \n\n(9) \n\nTo create the full system we use two parallel resistive networks in which the node voltages \nUi, j and Vi,j represent the two components of the optical flow vector U and v . The input \ncurrents lUini,i and lVini\" are computed by a negative recurrentfeedback loop modulated \nby the input data, which are the spatial and temporal intensity gradients. \nNotice that the input currents are proportional to the deviation of the local brightness con(cid:173)\nstraint: the less the local optical flow solution fits the data the higher the current lini.j will \nbe to correct the solution and vice versa. \nStability and convergence of the network are guaranteed by Maxwell 's minimum power \nprinciple [4, 9]. \n\n4 The Smooth Optical Flow Chip \n\n4.1 \n\nImplementation \n\n-CP\\~}1J\u00ad\n~tf)~ \n! I ~ \n\nFigure 2: A single motion cell within the three layer network. For simplicity only one \nresistive network is shown. \n\nThe circuitry consists of three functional layers (Figure 2). The input layer includes an \narray of adaptive photoreceptors [10] and provides the derivatives of the image brightness \nto the second layer, The spatial gradients are the first-order linear approximation obtained \nby subtracting the two neighboring photoreceptor outputs. The second layer computes the \ninput current to the third layer according to equations (9). Finally these currents are fed \ninto the two resistive networks that report the optical flow components. \nThe schematics of the core of a single motion cell are drawn in Figure 3. The photoreceptor \nand the temporal differentiator are not shown as well as the other half of the circuitry that \ncomputes the y-component of the flow vector. \n\n\f710 \n\nA. Stocker and R. Doug/as \n\nA few remarks are appropriate here: First, the two components of the optical flow vector \nhave to be able to take on positive and negative values with respect to some reference po(cid:173)\ntential. Therefore, a symmetrical circuit scheme is applied where the positive and negative \n(reference voltage) values are carried on separate signal lines. Thus, the actual value is \nencoded as the difference of the two potentials. \n\nE (E V + E) \nx \n\nx x \n\nt \n\n~.\" .... \" ....... \" ....... \" ......... : \n\ntemporal \ndifferentiator \n\nExl \n_ f-VViBias ! \nl \nv+ \nI:\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7 .. \u00b7\u00b7 .. \u00b7 .. \u00b7\u00b7\u00b7\u00b7\u00b7 .. \u00b7\u00b7\u00b7 .. \u00b7: \nX DiffBias \n\n1 \n\nOpBias \n\nFigure 3: Cell core schematics; only the circuitry related to the computation of the \nx-component of the flow vector is shown. \n\nSecond, the limited linear range of the Gilbert multipliers leads to a narrow span of flow ve(cid:173)\nlocities that can be computed reliably. However, the tuning can be such that the operational \nrange is either at high or very low velocities. Newer implementations are using modified \nmultipliers with a larger linear range. \nThird, consider a single motion cell (Figure 2). In principle, this cell would be able to sat(cid:173)\nisfy the local constraint perfectly. In practice (see Figure 3), the finite output impedance of \nthe p-type Gilbert multiplier slightly degrades this ideal solution by imposing an effective \nconductance G load . Thus, a constant voltage on the capacitor representing a non-zero mo(cid:173)\ntion signal requires a net output current of the mUltiplier to maintain it. This requirement \nhas two interesting consequences: \ni) The reported optical flow is dependent on the spatial gradients (contrast). A single un(cid:173)\ncoupled cell according to Figure 2 has a steady state solution with \n\nU \ni ,j '\" (G load + E;i .j + E~iJ \n\nI , ] \n\n.Ex \n\n. \n' .J \n\n-Et \n\nand \n\n'Y: \ni,j '\" (Gload + E; . + Ey2) \n\n-EtEy .. \n\n1 , J \n\n1,) \n\n1,) \n\n1,) \n\nrespectively. For the same object speed, the chip reports higher velocity signals for higher \nspatial gradients. Preferably, Gload should be as low as possible to minimize its influence \non the solution. \nii) On the other hand, the locally ill-posed problem is now well-posed because G load im(cid:173)\nposes a second constraint. Thus, the chip behaves sensibly in the case of low contrast \ninput (small gradients), reporting zero motion where otherwise, unreliable high values \nwould occur. This is convenient because the signal-to-noise ratio at low contrast is very \npoor. Furthermore, a single cell is forced to report the velocity on the constraint line with \nsmallest absolute value, which is normal to the spatial gradient. That means that the chip \n\n\fComputation of Optical Flow in an Analog Network \n\n711 \n\nreports normal flow when there is no neighbor connection. Since there is an trade-off be(cid:173)\ntween the robustness of the optical flow computation and a low conductance Glaad, the \nfollower-connected transconductance amplifier in our implementation allows us to control \nG laad above its small intrinsic value. \n\n4.2 Results \n\nThe results reported below were obtained from a MOSIS tinychip containing a 7x7 array \nof motion cells each 325x325 A 2 in size. The chip was fabricated in 1.2 J.,tm technology at \nAMI. \n\n- \\ \n\n......... \n\n\"'-\n\"- \"- \"-\n\n-, \" \"\"-\n,,,-\n\"\" \n-~\" , \n.\" \"-\n'f-~' ~ , \n\n,\"\"\",-- ~ ~ \n\n3 \n\n\"-\n\n... \n\n\"-\n\n\"-\n\n, \" \n, \"-\n\n, ,\" \"-\n\"- ,~\" , \n.' , ,\" \"- , \n\" ,~\" , , \n.' ,\" \"- , \n\" \n, , , \n,1'-'\" \n\na \n\nb \n\nc \n\nFigure 4: Smooth optical flow response of the chip to an left-upwards moving edge. \na: photoreceptor output, the arrow indicates the actual motion direction. b: weak coupling \n(small conductance G). c: strong coupling. \n\n3 \n\n\\ --\n, - , \n, \" \n'r-- /\" \n. , I I \n'I / \n\n,.- ,~ \n\n\" \n.,.- ,/ \n\n\\ \n\n, \n\nlr- ~~~~~~ \n\nI \n,- - / 2F--- ~ ~~ ~ -E-- ~ \n\"-\n,, '.... \n\n\"'--\n/ .F--~~~~~~ \n\nSr- ~ ~ ~ ~ '4--\n&r-- ~ 'E--- ~ -E--\n\n3F-- ~ ~ ~ \"'E--- ~ -E--\n\n--\\ \n\n'E-- ~ \n\n