{"title": "Efficient Formal Safety Analysis of Neural Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 6367, "page_last": 6377, "abstract": "Neural networks are increasingly deployed in real-world safety-critical domains such as autonomous driving, aircraft collision avoidance, and malware detection. However, these networks have been shown to often mispredict on inputs with minor adversarial or even accidental perturbations. Consequences of such errors can be disastrous and even potentially fatal as shown by the recent Tesla autopilot crash. Thus, there is an urgent need for formal analysis systems that can rigorously check neural networks for violations of different safety properties such as robustness against adversarial perturbations within a certain L-norm of a given image. An effective safety analysis system for a neural network must be able to either ensure that a safety property is satisfied by the network or find a counterexample, i.e., an input for which the network will violate the property. Unfortunately, most existing techniques for performing such analysis struggle to scale beyond very small networks and the ones that can scale to larger networks suffer from high false positives and cannot produce concrete counterexamples in case of a property violation. In this paper, we present a new efficient approach for rigorously checking different safety properties of neural networks that significantly outperforms existing approaches by multiple orders of magnitude. Our approach can check different safety properties and find concrete counterexamples for networks that are 10x larger than the ones supported by existing analysis techniques. We believe that our approach to estimating tight output bounds of a network for a given input range can also help improve the explainability of neural networks and guide the training process of more robust neural networks.", "full_text": "Ef\ufb01cient Formal Safety Analysis of Neural Networks\n\nShiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, Suman Jana\n\nColumbia University, NYC, NY 10027, USA\n\n{tcwangshiqi, kpei, jaw2228, junfeng, suman}@cs.columbia.edu\n\nAbstract\n\nNeural networks are increasingly deployed in real-world safety-critical domains\nsuch as autonomous driving, aircraft collision avoidance, and malware detection.\nHowever, these networks have been shown to often mispredict on inputs with minor\nadversarial or even accidental perturbations. Consequences of such errors can be\ndisastrous and even potentially fatal as shown by the recent Tesla autopilot crashes.\nThus, there is an urgent need for formal analysis systems that can rigorously check\nneural networks for violations of different safety properties such as robustness\nagainst adversarial perturbations within a certain L-norm of a given image. An\neffective safety analysis system for a neural network must be able to either ensure\nthat a safety property is satis\ufb01ed by the network or \ufb01nd a counterexample, i.e.,\nan input for which the network will violate the property. Unfortunately, most\nexisting techniques for performing such analysis struggle to scale beyond very\nsmall networks and the ones that can scale to larger networks suffer from high\nfalse positives and cannot produce concrete counterexamples in case of a property\nviolation. In this paper, we present a new ef\ufb01cient approach for rigorously checking\ndifferent safety properties of neural networks that signi\ufb01cantly outperforms existing\napproaches by multiple orders of magnitude. Our approach can check different\nsafety properties and \ufb01nd concrete counterexamples for networks that are 10\u00d7\nlarger than the ones supported by existing analysis techniques. We believe that our\napproach to estimating tight output bounds of a network for a given input range\ncan also help improve the explainability of neural networks and guide the training\nprocess of more robust neural networks.\n\n1\n\nIntroduction\n\nOver the last few years, signi\ufb01cant advances in neural networks have resulted in their increasing\ndeployments in critical domains including healthcare, autonomous vehicles, and security. However,\nrecent work has shown that neural networks, despite their tremendous success, often make dangerous\nmistakes, especially for rare corner case inputs. For example, most state-of-the-art neural networks\nhave been shown to produce incorrect outputs for adversarial inputs speci\ufb01cally crafted by adding\nminor human-imperceptible perturbations to regular inputs [36, 14]. Similarly, seemingly minor\nchanges in lighting or orientation of an input image have been shown to cause drastic mispredictions\nby the state-of-the-art neural networks [29, 30, 37]. Such mistakes can have disastrous and even\npotentially fatal consequences. For example, a Tesla car in autopilot mode recently caused a fatal\ncrash as it failed to detect a white truck against a bright sky with white clouds [3].\nA principled way of minimizing such mistakes is to ensure that neural networks satisfy simple\nsafety/security properties such as the absence of adversarial inputs within a certain L-norm of a given\nimage or the invariance of the network\u2019s predictions on the images of the same object under different\nlighting conditions. Ideally, given a neural network and a safety property, an automated checker\nshould either guarantee that the property is satis\ufb01ed by the network or \ufb01nd concrete counterexamples\ndemonstrating violations of the safety property. The effectiveness of such automated checkers hinges\non how accurately they can estimate the decision boundary of the network.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fHowever, strict estimation of the decision boundary of a neural network with piecewise linear\nactivation functions such as ReLU is a hard problem. While the linear pieces of each ReLU node can\nbe partitioned into two linear constraints and ef\ufb01ciently check separately, the total number of linear\npieces grow exponentially with the number of nodes in the network [25, 27]. Therefore, exhaustive\nenumeration of all combinations of these pieces for any modern network is prohibitively expensive.\nSimilarly, sampling-based inference techniques like blackbox Monte Carlo sampling may need an\nenormous amount of data to generate tight accurate bounds on the decision boundary [11].\nIn this paper, we propose a new ef\ufb01cient approach for rigorously checking different safety properties\nof neural networks that signi\ufb01cantly outperform existing approaches by multiple orders of magnitude.\nSpeci\ufb01cally, we introduce two key techniques. First, we use symbolic linear relaxation that combines\nsymbolic interval analysis and linear relaxation to compute tighter bounds on the network outputs\nby keeping track of relaxed dependencies across inputs during interval propagation when the actual\ndependencies become too complex to track. Second, we introduce a novel technique called directed\nconstraint re\ufb01nement to iteratively minimize the errors introduced during the relaxation process\nuntil either a safety property is satis\ufb01ed or a counterexample is found. To make the re\ufb01nement\nprocess ef\ufb01cient, we identify the potentially overestimated nodes, i.e., the nodes where inaccuracies\nintroduced during relaxation can potentially affect the checking of a given safety property, and use\noff-the-shelf solvers to focus only on those nodes to further tighten their output ranges.\nWe implement our techniques as part of Neurify, a system for rigorously checking a diverse set\nof safety properties of neural networks 10\u00d7 larger than the ones that can be handled by existing\ntechniques. We used Neurify to check six different types of safety properties of nine different\nnetworks trained on \ufb01ve different datasets. Our experimental results show that on average Neurify is\n5, 000\u00d7 faster than Reluplex [17] and 20\u00d7 than ReluVal [39].\nBesides formal analysis of safety properties, we believe our method for ef\ufb01ciently estimating tight\nand rigorous output ranges of a network will also be useful for guiding the training process of robust\nnetworks [42, 32] and improving explainability of the decisions made by neural networks [34, 20, 23].\nRelated work. Several researchers have tried to extend and customize Satis\ufb01ability Modulo Theory\n(SMT) solvers for estimating decision boundaries with strong guarantees [17, 18, 15, 10, 31]. Another\nline of research has used Mixed Integer Linear Programming (MILP) solvers for such analysis [38,\n12, 7]. Unfortunately, the ef\ufb01ciency of both of these approaches is severely limited by the high\nnonlinearity of the resulting formulas.\nDifferent convex or linear relaxation techniques have also been used to strictly approximate the\ndecision boundary of neural networks. While these techniques tend to scale signi\ufb01cantly better than\nsolver-based approaches, they suffer from high false positive rates and struggle to \ufb01nd concrete\ncounterexamples demonstrating violations of safety properties [42, 32, 13, 9]. Similarly, existing\nworks on \ufb01nding lower bounds of adversarial perturbations to fool a neural network also suffer\nfrom the same limitations [28, 41]. Note that concurrent work of Weng et al. [40] uses similar\nlinear relaxation method as ours but it alone struggles to solve such problems as shown in Table\n6. Also, their follow-up work [44] that provides a generic relaxation method for general activation\nfunctions does not address this issue either. In contrast, we mainly use our relaxation technique to\nidentify crucial nodes and iteratively re\ufb01ne output approximations over these nodes with the help of\nlinear solver. Another line of research has focused on strengthening network robustness either by\nincorporating these relaxation methods into training process [43, 8, 24] or by leveraging techniques\nlike differential privacy [22]. Our method, essentially providing a more accurate formal analysis of a\nnetwork, can potentially be incorporated into training process to further improve network robustness.\nRecently, ReluVal, by Wang et al. [39], has used interval arithmetic [33] for rigorously estimating\na neural network\u2019s decision boundary by computing tight bounds on the outputs of a network for\na given input range. While ReluVal achieved signi\ufb01cant performance gain over the state-of-the-art\nsolver-based methods [17] on networks with a small number of inputs, it struggled to scale to larger\nnetworks (see detailed discussions in Section 2).\n\n2 Background\n\nWe build upon two prior works [10, 39] on using interval analysis and linear relaxations for analyzing\nneural networks. We brie\ufb02y describe them and refer interested readers to [10, 39] for more details.\n\n2\n\n\fSymbolic interval analysis. Interval arithmetic [33] is a \ufb02exible and ef\ufb01cient way of rigorously\nestimating the output ranges of a function given an input range by computing and propagating the\noutput intervals for each operation in the function. However, naive interval analysis suffers from\nlarge overestimation errors as it ignores the input dependencies during interval propagation. To\nminimize such errors, Wang et al. [39] used symbolic intervals to keep track of dependencies by\nmaintaining linear equations for upper and lower bounds for each ReLU and concretizing only for\nthose ReLUs that demonstrate non-linear behavior for the given input intervals. Speci\ufb01cally, consider\nan intermediate ReLU node z = Relu(Eq), (l, u) = (Eq, Eq), where Eq denotes the symbolic\nrepresentation (i.e., a closed-form equation) of the ReLU\u2019s input in terms of network inputs X and\n(l, u) denote the concrete lower and upper bounds of Eq, respectively. There are three possible output\nintervals that the ReLU node can produce depending on the bounds of Eq: (1) z = [Eq, Eq] when\nl \u2265 0, (2) z = [0, 0] when u \u2264 0, or (3) z = [l, u] when l < 0 < u. Wang et al. will concretize the\noutput intervals for this node only if the third case is feasible as the output in this case cannot be\nrepresented using a single linear equation.\nBisection of input features. To further minimize overestimation, [39] also proposed an iterative\nre\ufb01nement strategy involving repeated input bisection and output reunion. Consider a network\nF taking d-dimensional input, and the i-th input feature interval is Xi and network output in-\nterval is F (X) where X = {X1, ..., Xd}. A single bisection on Xi will create two children:\nX\u2032 = {X1, ..., [Xi, Xi+Xi\n2 , Xi], ..., Xd}. The reunion of the\ncorresponding output intervals F (X\u2032)! F (X\u2032\u2032), will be tighter than the original output interval, i.e.,\nF (X\u2032)! F (X\u2032\u2032) \u2286 F (X), as the Lipschitz continuity of the network ensures that the overestimation\n\nerror decreases as the width of input interval becomes smaller. However, the ef\ufb01ciency of input\nbisection decreases drastically as the number of input dimensions increases.\nLinear relaxation. Ehlers et al. [10] used lin-\near relaxation of ReLU nodes to strictly over-\napproximate the non-linear constraints intro-\nduced by each ReLU. The generated linear con-\nstraints can then be ef\ufb01ciently solved using a\nlinear solver to get bounds on the output of a\nneural network for a given input range. Consider\nthe simple ReLU node taking input z\u2032 with an\nupper and lower bound u and l respectively and\nproducing output z as shown in Figure 1. Linear relaxation of such a node will use the following\nthree linear constraints: (1) z \u2265 0, (2) z \u2265 z\u2032, and (3) z \u2264 u(z\u2032\u2212l)\nto expand the feasible region to the\nu\u2212l\ngreen triangle from the two original piecewise linear components. The effectiveness of this approach\nheavily depends on how accurately u and l can be estimated. Unfortunately, Ehlers et al. [10] used\nnaive interval propagation to estimate u and l leading to large overestimation errors. Furthermore,\ntheir approach cannot ef\ufb01ciently re\ufb01ne the estimated bounds and thus cannot bene\ufb01t from increasing\ncomputing power.\n\n2 ], ..., Xd} and X\u2032\u2032 = {X1, ..., [ Xi+Xi\n\nFigure 1: Linear relaxation of a ReLU node.\n\n3 Approach\n\nIn this paper, we make two major contributions to scale formal safety analysis to networks signi\ufb01cantly\nlarger than those evaluated in prior works [17, 10, 42, 39]. First, we combine symbolic interval\nanalysis and linear relaxation (described in Section 2) in a novel way to create a signi\ufb01cantly\nmore ef\ufb01cient propagation method\u2013symbolic linear relaxation\u2013that can achieve substantially tighter\nestimations (evaluated in Section 4). Second, we present a technique for identifying the overestimated\nintermediate nodes, i.e., the nodes whose outputs are overestimated, during symbolic linear relaxation\nand propose directed constraint re\ufb01nement to iteratively re\ufb01ne the output ranges of these nodes. In\nSection 4, we also show that this method mitigates the limitations of input bisection [39] and scales\nto larger networks.\nFigure 2 illustrates the high-level work\ufb02ow of Neurify. Neurify takes in a range of inputs X and\nthen determines using linear solver whether the output estimation generated by symbolic linear\nrelaxation satis\ufb01es the safety proprieties. A property is proven to be safe if the solver \ufb01nd the\nrelaxed constraints unsatis\ufb01able. Otherwise, the solver returns potential counterexamples. Note that\nthe returned counterexamples found by the solver might be false positives due to the inaccuracies\n\n3\n\n\fintroduced by the relaxation process. Thus Neurify will check whether a counterexample is a false\npositive. If so, Neurify will use directed constraint re\ufb01nement guided by symbolic linear relaxation\nto obtain a tighter output bound and recheck the property with the solver.\n\n3.1 Symbolic Linear Relaxation\nThe symbolic linear relaxation of the output of each\nReLU z = Relu(z\u2032) leverages the bounds on z\u2032,\nEqlow and Equp (Eqlow \u2264 Eq\u2217(x) \u2264 Equp). Here\nEq\u2217 denotes the closed-form representation of z\u2032.\nSpeci\ufb01cally, Equation 1 shows the symbolic linear\nrelaxation where %\u2192 denotes \u201crelax to\u201d. In addition,\n[llow, ulow] and [lup, uup] denote the concrete lower\nand upper bounds for Eqlow and Equp, respectively.\nIn supplementary material Section 1.2, we give a de-\ntailed proof showing that this relaxation is the tightest\nachievable due to its least maximum distance from\nEq\u2217. In the following discussion, we simplify Eqlow\nand Equp as Eq and the corresponding lower and\nupper bounds as [l, u]. Figure 3 shows the differ-\nence between our symbolic relaxation process and\nthe naive concretizations used by Wang et al. [39].\nMore detailed discussions can be found in supple-\nmentary material Section 2.\n\nRelu(Eqlow) %\u2192\n\nulow\n\n(Eqlow)\n\nInput intervals\n\nSymbolic linear \n\nrelaxation\nSplit\ntarget\nnode\n\nSafe\n\nUnsat\n\nConstraints\n\nLinear solver\n\nDNN\n\nSafety\nproperty\n\nConcrete\nsample\n\nRefine \n\noverest. node\n\nFalse positive\n\nTimeout\n\nCheck for \nviolation\n\nViolated\n\nUnsafe\n\nFigure 2: Work\ufb02ow of Neurify to formally\nanalyze safety properties of neural networks.\n\nuup\n\nz \u2264 u \n\nl \n\nRelu(Equp) %\u2192\nz \n\n(Equp \u2212lup)\nz \n\n(1)\n\nz \u2264\n\nz\u2032\n\nz \u2265 0 \n\n0\n\nu\n\nz \u2265\n\nu\nEq \nu - l\nz\u2032 \n\nuup \u2212lup\n\nu\nu - l (Eq - l) \n\nulow \u2212llow\nIn practice, symbolic linear relaxation\ncan cut (on average) 59.64% more\noverestimation error than symbolic in-\nterval analysis (cf. Section 2) and\nsaves the time needed to prove a prop-\nerty by several orders of magnitude\n(cf. Section 4). There are three key\nreasons behind such signi\ufb01cant per-\nformance improvement. First, the\nmaximum possible error after intro-\nducing relaxations is \u2212lup\u2217uup\nfor up-\nuup\u2212lup\nper bound and \u2212llow\u2217ulow\nfor lower\nulow\u2212llow\nbound in Figure 3(b) (the proof is in\nsupplementary material Section 1.2).\nThese relaxations are considerably\ntighter than naive concretizations shown in Figure 3(a), which introduces a larger error uup. Second,\nsymbolic linear relaxation, unlike naive concretization, partially keeps the input dependencies during\ninterval propagation ([ u\nu\u2212l (Eq \u2212l)] by maintaining symbolic equations. Third, as the \ufb01nal\noutput error is exponential to the error introduced at each node (proved in supplementary 1.2), tighter\nbounds on earlier nodes produced by symbolic relaxation signi\ufb01cantly reduce the \ufb01nal output error.\n\nFigure 3: An illustration of symbolic linear relaxation for\nan intermediate node. (a) Original symbolic interval anal-\nysis [39] used naive concretization.\n(b) Symbolic linear\nrelaxation leverages the knowledge of concrete bounds for z\u2032\nand computes relaxed symbolic interval. Eq is the symbolic\nrepresentation of z\u2032.\n\n(b) Symbolic linear relaxation\n\nu\u2212l Eq, u\n\n(a) Naive concretizaion\n\nl\n\n0\n\nu \n\n3.2 Directed Constraint Re\ufb01nement\nBesides symbolic linear relaxation, we also develop another generic approach, directed constraint\nre\ufb01nement, to further improve the overall performance of property checking. Our empirical results\nin Section 4 shows the substantial improvement from using this approach combined with symbolic\nlinear relaxation. In the following, we \ufb01rst de\ufb01ne overestimated nodes before describing the directed\nconstraint re\ufb01nement process in detail.\nOverestimated nodes. We note that, for most networks, only a small proportion of intermediate\nReLU nodes operate in the non-linear region for a given input range X. These are the only nodes that\n\n4\n\n\fneed to be relaxed (cf. Section 2). We call these nodes overestimated as they introduce overestimation\nerror during relaxation. We include other useful properties and proofs regarding overestimated nodes\nin supplementary material Section 1.1.\nBased on the de\ufb01nition of overestimated nodes, we de\ufb01ne one step of directed constraint re\ufb01nement\nas computing the re\ufb01ned output range F \u2032(X):\n\nF \u2032(X) = F (x \u2208 X|Eq(x) \u2264 0) \u222a F (x \u2208 X|Eq(x) > 0)\n\n(2)\nwhere X denotes the input intervals to the network, F is the corresponding network, and Eq is the\ninput equation of an overestimated node. Note that here we are showing the input of a node as a\nsingle equation for simplicity instead of the upper and lower bounds shown in Section 3.1.\nWe iteratively re\ufb01ne the bounds by invoking a linear solver, allowing us to make Neurify more\nscalable for dif\ufb01cult safety properties. The convergence analysis is given in supplementary material\nSection 1.3.\nThe re\ufb01nement includes the following three steps:\nLocating overestimated nodes. From symbolic linear relaxations, we can get the set of overestimated\nnodes within the network. We then prioritize the overestimated nodes with larger output gradient\nand re\ufb01ne these in\ufb02uential overestimated nodes \ufb01rst. We borrow the idea from [39] of computing\nthe gradient of network output with respect to the input interval of the overestimated node. A larger\ngradient value of a node signi\ufb01es that the input of that node has a greater in\ufb02uence towards changing\nthe output than than the inputs of other nodes.\nSplitting. After locating the target overestimated node, we split its input ranges into two independent\ncases, Eqt > 0 and Eqt \u2264 0 where Eqt denotes the input of the target overestimated node. Now,\nunlike symbolic linear relaxation where Relu([Eqt, Eqt]) %\u2192 [ u\nu\u2212l (Eqt \u2212l)], neither of\nthe two split cases requires any relaxation (Section 2) as the input interval no longer includes 0.\nTherefore, splitting creates two tighter approximations of the output F (x \u2208 X|Eqt(x) > 0) and\nF (x \u2208 X|Eqt(x) \u2264 0).\nSolving. We solve the resulting linear constraints, along with the constraints de\ufb01ned in safety\nproperties, by instantiating an underlying linear solver. In particular, we de\ufb01ne safety properties that\ncheck that the con\ufb01dence value of a target output class F t is always greater than the outputs of other\nclasses F o (e.g., outputs other than 7 for an image of a hand-written 7). We thus de\ufb01ne the constraints\nfor safety properties as Eqt\nup are the lower bound equations\nfor F t and the upper bound equations for F o derived using symbolic linear relaxation. Each step\nof directed constraint re\ufb01nement of an overestimated node results in two independent problems as\nshown in Equation 3 that can be checked with a linear solver.\n\nu\u2212l Eqt, u\n\nup < 0. Here, Eqt\n\nlow and Eqo\n\nlow \u2212Eqo\n\nCheck Sati\ufb01ability: Eqt\nCheck Sati\ufb01ability: Eqt\n\nlow1\u2212Eqo\nlow2\u2212Eqo\n\nup1 < 0; Eqt \u2264 0; xi \u2212\u03f5 \u2264 xi \u2264 xi + \u03f5 (i = 1 . . . d)\nup2 < 0; Eqt > 0; xi \u2212\u03f5 \u2264 xi \u2264 xi + \u03f5 (i = 1 . . . d)\n\n(3)\n\nIn this process, we invoke the solver in two ways. (1) If the solver tells that both cases are unsatis\ufb01able,\nthen the property is formally proved to be safe. Otherwise, further iterative re\ufb01nement steps can be\napplied. (2) If either case is satis\ufb01able, we treat the solutions returned by the linear solver as potential\ncounterexamples violating the safety properties. Note that these solutions might be false positives\ndue to the inaccuracies introduced during the relaxation process. We thus resort to directly executing\nthe target network with the solutions returned from the solver as input. If the solution does not violate\nthe property, we repeat the above process for another overestimated node (cf. Figure 2).\n\n3.3 Safety Properties\nIn this work, we support checking diverse safety properties of networks including \ufb01ve different\nclasses of properties based on the input constraints. Particularly, we specify the safety properties of\nneural network based on de\ufb01ning constraints on its input-output. For example, as brie\ufb02y mentioned\nin Section 3.1, we specify that the output of the network on input x should not change (i.e., remain\ninvariant) when x is allowed to vary within a certain range X. For output constraints, taking an\narbitrary classi\ufb01er as an example, we de\ufb01ne the output invariance by specifying the difference\ngreater than 0 between lower and upper bound of con\ufb01dence value of the original class of the input\nand other classes. For specifying input constraints, we consider three popular bounds, i.e., L\u221e,\n\n5\n\n\fL1, and L2, which are widely used in the literature of adversarial machine learning [14]. These\nthree bounds allow for arbitrary perturbations of the input features as long as the corresponding\nnorms of the overall perturbation are within a certain threshold.\nIn addition to these arbitrary\nperturbations, we consider two speci\ufb01c perturbations that change brightness and contrast of the\ninput images as discussed in [30]. Properties speci\ufb01ed using L\u221e naturally \ufb01t into our symbolic\nlinear relaxation process where each input features are bounded by an interval. For properties\ni=1|xi|\u2264 \u03f5 for L1, or\n2\u2264 \u03f5 for L2, which are no longer linear. We handle such cases by using solvers that support\nquadratic constraints (see details in Section 4). The safety properties involving changes in brightness\nand contrast can be ef\ufb01ciently checked by iteratively bisecting the input nodes simultaneously as\nminx\u2208[x\u2212\u03f5,x+\u03f5](F (x)) = min(minx\u2208[x,x+\u03f5](F (x)), minx\u2208[x\u2212\u03f5,x](F (x))) where F represents the\ncomputation performed by the target network .\n\nspeci\ufb01ed in L1 \u2264 \u03f5 or L2 \u2264 \u03f5, we need to add more constraints, i.e.,\"d\n\"d\n\ni=1xi\n\n4 Experiments\n\nImplementation. We implement Neurify with about 26,000 lines of C code. We use the highly\noptimized OpenBLAS1 library for matrix multiplications and lp_solve 5.52 for solving the linear\nconstraints generated during the directed constraint re\ufb01nement process. We further use Gurobi 8.0.0\nsolver for L2-bounded safety properties. All our evaluations were performed on a Linux server\nrunning Ubuntu 16.04 with 8 CPU cores and 256GB memory. Besides, Neurify uses optimization\nlike thread rebalancing for parallelization and outward rounding to avoid incorrect results due to\n\ufb02oating point imprecision. Details of such techniques can be found in Section 3 of the supplementary\nmaterial.\nTable 1: Details of the evaluated networks and corresponding safety properties. The last three columns\nsummarize the number of safety properties that are satis\ufb01ed, violated, and timed out, respectively as\nfound by Neurify with a timeout threshold of 1 hour.\nArchitecture\n<5, 50, 50, 50,\n50, 50, 50, 5>#\n\nSafe Violated Timeout\n\nDataset\nACAS\nXu [16]\n\n# of\nReLUs\n\nModels\n\n300\nACAS Xu\n48\nMNIST_FC1\n100\nMNIST_FC2\nMNIST_FC3\n1024\nMNIST_CN 4804\nDrebin_FC1\n100\n210\nDrebin_FC2\nDrebin_FC3\n400\n\nMNIST [21]\n\nDrebin [5]\n\nSafety\nProperty\n\nC.P.\u2217\nin [39]\nL\u221e\nL\u221e\nL\u221e\nL\u221e\nC.P.\u2217\nin [29]\nL\u221e,L1,\nBrightness,\nContrast\n\n<784, 24, 24, 10>#\n<784, 50, 50, 10>#\n<784, 512, 512, 10>#\n<784, k:16*4*4 s:2,\n\nk:32*4*4 s:2, 100, 10>+\n\n<545334, 50, 50, 2>#\n<545334, 200, 10, 2>#\n<545334, 200, 200, 2>#\n<30000, k:24*5*5 s:5,\nk:36*5*5 s:5, 100, 10>+\n\n141\n267\n271\n322\n91\n458\n437\n297\n\n37\n233\n194\n41\n476\n21\n22\n27\n\n0\n0\n35\n137\n233\n21\n41\n176\n\nCar [2]\n\nDAVE\n\n10276\n\n80\n\n82\n\n58\n\n* Custom properties.\n# <x, y, ...> denotes hidden layers with x neurons in \ufb01rst layer, y neurons in second layer, etc.\n+ k:c*w*h s:stride denotes the output channel (c), kernel width (w), height (h) and stride (stride).\n\n4.1 Properties Checked by Neurify for Each Model\n\nSummary. To evaluate the performance of Neurify, we test it on nine models trained over \ufb01ve\ndatasets for different tasks where each type of model includes multiple architectures. Speci\ufb01cally, we\nevaluate on fully connected ACAS Xu models [16], three fully connected Drebin models [5], three\nfully connected MNIST models [21], one convolutional MNIST model [42], and one convolutional\nself-driving car model [2]. Table 1 summarizes the detailed structures of these models. We include\nmore detailed descriptions in supplementary material Section 4. All the networks closely follow\nthe publicly-known settings and are either pre-trained or trained of\ufb02ine to achieve comparable\nperformance to the real-world models on these datasets.\n\n1https://www.openblas.net/\n2http://lpsolve.sourceforge.net/5.5/\n\n6\n\n\fWe also summarize the safety properties checked by Neurify in Table 1 with timeout threshold set to\n3,600 seconds. Here we report the result of the self-driving care model (DAVE) to illustrate how we\nde\ufb01ne the safety properties and the numbers of safe and violated properties found by Neurify. We\nreport the other results in supplementary material Section 5.\n\nTable 2: Different safety properties checked by Neurify out of 10 random images on Dave within\n3600 seconds.\n\n(a) ||X\u2032 \u2212 X||\u221e \u2264 \u03f5\n\n\u03f5\nSafe(%)\nViolated(%)\nTimeout(%)\n\n1\n50\n0\n50\n\n2\n10\n20\n70\n\n5\n0\n70\n30\n\n8\n0\n100\n0\n\n10\n0\n100\n0\n\n\u03f5\nSafe(%)\nViolated(%)\nTimeout(%)\n\n(b) ||X\u2032 \u2212 X||1 \u2264 \u03f5\n300\n100\n100\n10\n40\n0\n0\n50\n\n200\n100\n0\n0\n\n500\n10\n50\n40\n\n700\n0\n60\n40\n\n(c) Brightness: X \u2212 \u03f5 \u2264 X\u2032 \u2264 X + \u03f5\n\n\u03f5\nSafe(%)\nViolated(%)\nTimeout(%)\n\n10\n100\n0\n0\n\n70\n30\n30\n40\n\n80\n20\n50\n30\n\n90\n10\n60\n30\n\n100\n10\n70\n20\n\n(d) Contrast: \u03f5X \u2264 X\u2032 \u2264 X or X \u2264 X\u2032 \u2264 \u03f5X\n2.5\n\u03f5\n0\nSafe(%)\n50\nViolated(%)\nTimeout(%)\n50\n\n1.01\n100\n0\n0\n\n0.99\n100\n0\n0\n\n0.5\n10\n20\n70\n\n0.2\n0\n70\n30\n\nDave. We show that Neurify is the \ufb01rst formal analysis tool that can systematically check different\nsafety properties for a large (over 10,000 ReLUs) convolutional self-driving car network, Dave [2, 6].\nWe use the dataset from Udacity self-driving car challenge containing 101,396 training and 5,614\ntesting samples [4]. Our model\u2019s architecture is similar to the DAVE-2 self-driving car architecture\nfrom NVIDIA [6, 2] and it achieves similar 1-MSE as models used in [29]. We formally analyze the\nnetwork with inputs bounded by L\u221e, L1, brightness, and contrast as described in Section 3.3. We\nde\ufb01ne the safe range of deviation of the output steering direction from the original steering angle to\nbe less than 30 degrees. The total number of cases Neurify can verify are shown in Table 2.\n\nTable 3: Total cases that can be veri\ufb01ed by Neurify on three Drebin models out of 100 random\nmalware apps. The timeout setting here is 3600 seconds.\n\nModels\n\nDrebin_FC1\n\nDrebin_FC2\n\nDrebin_FC3\n\nCases(%)\n\nSafe\n\nViolated\n\nViolated\n\nTotal\nSafe\n\nTotal\nSafe\n\nViolated\n\nTotal\n\n10\n0\n100\n100\n0\n100\n100\n0\n100\n100\n\n50\n1\n98\n99\n4\n96\n100\n4\n89\n93\n\n100\n3\n97\n100\n4\n90\n94\n4\n74\n78\n\n150\n5\n86\n91\n6\n81\n87\n4\n23\n33\n\n200\n12\n77\n89\n8\n70\n78\n15\n11\n26\n\nDREBIN. We also evaluate Neurify on three different Drebin models containing 545,334 input\nfeatures. The safety property we check is that simply adding app permissions without changing any\nfunctionality will not cause the models to misclassify malware apps as benign. Here we show in\nTable 3 that Neurify can formally verify safe and unsafe cases for most of the apps within 3,600\nseconds.\n\n4.2 Comparisons with Other Formal Checkers\n\nACAS Xu. Unmanned aircraft alert systems (ACAS Xu) [19] are networks advising steering decisions\nfor aircrafts, which is on schedule to be installed in over 30,000 passengers and cargo aircraft\nworldwide [26] and US Navy\u2019s \ufb02eets [1]. It is comparably small and only has \ufb01ve input features so\nthat ReluVal [39] can ef\ufb01ciently check different safety properties. However, its performance still\nsuffers from the over-approximation of output ranges due to the concretizations introduced during\nsymbolic interval analysis. Neurify leverages symbolic linear relaxation and achieves on average\n20\u00d7 better performance than ReluVal [39] and up to 5,000\u00d7 better performance than Reluplex [17].\nIn Table 4, we summarize the time and speedup of Neurify compared to ReluVal and Reluplex for all\nthe properties tested in [17, 39].\n\n7\n\n\f\f\f6 Acknowledgements\n\nWe thank the anonymous reviewers for their constructive and valuable feedback. This work is\nsponsored in part by NSF grants CNS-16-17670, CNS-15-63843, and CNS-15-64055; ONR grants\nN00014-17-1-2010, N00014-16-1- 2263, and N00014-17-1-2788; and a Google Faculty Fellowship.\nAny opinions, \ufb01ndings, conclusions, or recommendations expressed herein are those of the authors,\nand do not necessarily re\ufb02ect those of the US Government, ONR, or NSF.\n\nReferences\n[1] NAVAIR plans to install ACAS Xu on MQ-4C \ufb02eet.\n\narticles/navair-plans-to-install-acas-xu-on-mq-4c-fleet-444989/.\n\nhttps://www.flightglobal.com/news/\n\n[2] Nvidia-Autopilot-Keras. https://github.com/0bserver07/Nvidia-Autopilot-Keras.\n[3] Tesla\u2019s autopilot was involved in another deadly car crash. https://www.wired.com/story/tesla-\n\nautopilot-self-driving-crash-california/.\n\n[4] Using Deep Learning to Predict Steering Angles. https://github.com/udacity/self-driving-car.\n[5] D. Arp, M. Spreitzenbarth, M. Hubner, H. Gascon, K. Rieck, and C. Siemens. Drebin: Effective and\nexplainable detection of android malware in your pocket. In Proceedings of the Network and Distributed\nSystem Security Symposium, volume 14, pages 23\u201326, 2014.\n\n[6] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort,\nU. Muller, J. Zhang, et al. End to end learning for self-driving cars. IEEE Intelligent Vehicles Symposium,\n2017.\n\n[7] S. Dutta, S. Jha, S. Sankaranarayanan, and A. Tiwari. Output range analysis for deep feedforward neural\n\nnetworks. In NASA Formal Methods Symposium, pages 121\u2013138. Springer, 2018.\n\n[8] K. Dvijotham, S. Gowal, R. Stanforth, R. Arandjelovic, B. O\u2019Donoghue, J. Uesato, and P. Kohli. Training\n\nveri\ufb01ed learners with learned veri\ufb01ers. arXiv preprint arXiv:1805.10265, 2018.\n\n[9] K. Dvijotham, R. Stanforth, S. Gowal, T. Mann, and P. Kohli. A dual approach to scalable veri\ufb01cation of\n\ndeep networks. The Conference on Uncertainty in Arti\ufb01cial Intelligence, 2018.\n\n[10] R. Ehlers. Formal veri\ufb01cation of piece-wise linear feed-forward neural networks. 15th International\n\nSymposium on Automated Technology for Veri\ufb01cation and Analysis, 2017.\n\n[11] R. Eldan. A polynomial number of random points does not determine the volume of a convex body.\n\nDiscrete & Computational Geometry, 46(1):29\u201347, 2011.\n\n[12] M. Fischetti and J. Jo. Deep neural networks as 0-1 mixed integer linear programs: A feasibility study.\n\narXiv preprint arXiv:1712.06174, 2017.\n\n[13] T. Gehr, M. Mirman, D. Drachsler-Cohen, P. Tsankov, S. Chaudhuri, and M. Vechev. Ai 2: Safety and\nrobustness certi\ufb01cation of neural networks with abstract interpretation. In IEEE Symposium on Security\nand Privacy, 2018.\n\n[14] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. International\n\nConference on Learning Representations, 2015.\n\n[15] X. Huang, M. Kwiatkowska, S. Wang, and M. Wu. Safety veri\ufb01cation of deep neural networks. In\n\nInternational Conference on Computer Aided Veri\ufb01cation, pages 3\u201329. Springer, 2017.\n\n[16] K. D. Julian, J. Lopez, J. S. Brush, M. P. Owen, and M. J. Kochenderfer. Policy compression for aircraft\n\ncollision avoidance systems. In 35th Digital Avionics Systems Conference, pages 1\u201310. IEEE, 2016.\n\n[17] G. Katz, C. Barrett, D. Dill, K. Julian, and M. Kochenderfer. Reluplex: An ef\ufb01cient smt solver for verifying\n\ndeep neural networks. International Conference on Computer Aided Veri\ufb01cation, 2017.\n\n[18] G. Katz, C. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer. Towards proving the adversarial\nrobustness of deep neural networks. 1st Workshop on Formal Veri\ufb01cation of Autonomous Vehicles, 2017.\n\n[19] M. J. Kochenderfer, J. E. Holland, and J. P. Chryssanthacopoulos. Next-generation airborne collision\navoidance system. Technical report, Massachusetts Institute of Technology-Lincoln Laboratory Lexington\nUnited States, 2012.\n\n10\n\n\f[20] P. W. Koh and P. Liang. Understanding black-box predictions via in\ufb02uence functions. International\n\nConference on Machine Learning, 2017.\n\n[21] Y. LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998.\n[22] M. Lecuyer, V. Atlidakis, R. Geambasu, H. Daniel, and S. Jana. Certi\ufb01ed robustness to adversarial\n\nexamples with differential privacy. arXiv preprint arXiv:1802.03471, 2018.\n\n[23] J. Li, W. Monroe, and D. Jurafsky. Understanding neural networks through representation erasure. arXiv\n\npreprint arXiv:1612.08220, 2016.\n\n[24] M. Mirman, T. Gehr, and M. Vechev. Differentiable abstract interpretation for provably robust neural\n\nnetworks. In International Conference on Machine Learning, pages 3575\u20133583, 2018.\n\n[25] G. F. Montufar, R. Pascanu, K. Cho, and Y. Bengio. On the number of linear regions of deep neural\n\nnetworks. In Advances in neural information processing systems, pages 2924\u20132932, 2014.\n\n[26] M. T. Notes. Airborne collision avoidance system x. MIT Lincoln Laboratory, 2015.\n[27] R. Pascanu, G. Montufar, and Y. Bengio. On the number of response regions of deep feed forward networks\n\nwith piece-wise linear activations. Advances in neural information processing systems, 2013.\n\n[28] J. Peck, J. Roels, B. Goossens, and Y. Saeys. Lower bounds on the robustness to adversarial perturbations.\n\nIn Advances in Neural Information Processing Systems, pages 804\u2013813, 2017.\n\n[29] K. Pei, Y. Cao, J. Yang, and S. Jana. Deepxplore: Automated whitebox testing of deep learning systems.\n\nIn 26th Symposium on Operating Systems Principles, pages 1\u201318. ACM, 2017.\n\n[30] K. Pei, Y. Cao, J. Yang, and S. Jana. Towards practical veri\ufb01cation of machine learning: The case of\n\ncomputer vision systems. arXiv preprint arXiv:1712.01785, 2017.\n\n[31] L. Pulina and A. Tacchella. An abstraction-re\ufb01nement approach to veri\ufb01cation of arti\ufb01cial neural networks.\n\nIn International Conference on Computer Aided Veri\ufb01cation, pages 243\u2013257. Springer, 2010.\n\n[32] A. Raghunathan, J. Steinhardt, and P. Liang. Certi\ufb01ed defenses against adversarial examples. International\n\nConference on Learning Representations, 2018.\n\n[33] M. J. C. Ramon E. Moore, R. Baker Kearfott. Introduction to Interval Analysis. SIAM, 2009.\n[34] A. Shrikumar, P. Greenside, and A. Kundaje. Learning important features through propagating activation\n\ndifferences. International Conference on Machine Learning, 2017.\n\n[35] M. Spreitzenbarth, F. Freiling, F. Echtler, T. Schreck, and J. Hoffmann. Mobile-sandbox: having a deeper\nlook into android applications. In 28th Annual ACM Symposium on Applied Computing, pages 1808\u20131815.\nACM, 2013.\n\n[36] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing\n\nproperties of neural networks. International Conference on Learning Representations, 2013.\n\n[37] Y. Tian, K. Pei, S. Jana, and B. Ray. DeepTest: Automated testing of deep-neural-network-driven\n\nautonomous cars. In 40th International Conference on Software Engineering, 2018.\n\n[38] V. Tjeng, K. Xiao, and R. Tedrake. Evaluating robustness of neural networks with mixed integer program-\n\nming. arXiv preprint arXiv:1711.07356, 2017.\n\n[39] S. Wang, K. Pei, W. Justin, J. Yang, and S. Jana. Formal security analysis of neural networks using\n\nsymbolic intervals. 27th USENIX Security Symposium, 2018.\n\n[40] T.-W. Weng, H. Zhang, H. Chen, Z. Song, C.-J. Hsieh, D. Boning, I. S. Dhillon, and L. Daniel. Towards\n\nfast computation of certi\ufb01ed robustness for relu networks. arXiv preprint arXiv:1804.09699, 2018.\n\n[41] T.-W. Weng, H. Zhang, P.-Y. Chen, J. Yi, D. Su, Y. Gao, C.-J. Hsieh, and L. Daniel. Evaluating the\nrobustness of neural networks: An extreme value theory approach. International Conference on Learning\nRepresentations, 2018.\n\n[42] E. Wong and J. Z. Kolter. Provable defenses against adversarial examples via the convex outer adversarial\n\npolytope. International Conference on Machine Learning, 2018.\n\n[43] E. Wong, F. Schmidt, J. H. Metzen, and J. Z. Kolter. Scaling provable adversarial defenses. Advances in\n\nNeural Information Processing Systems, 2018.\n\n[44] H. Zhang, T.-W. Weng, P.-Y. Chen, C.-J. Hsieh, and L. Daniel. Ef\ufb01cient neural network robustness\ncerti\ufb01cation with general activation functions. Advances in Neural Information Processing Systems, 2018.\n\n11\n\n\f", "award": [], "sourceid": 3136, "authors": [{"given_name": "Shiqi", "family_name": "Wang", "institution": "Columbia University"}, {"given_name": "Kexin", "family_name": "Pei", "institution": "Columbia University"}, {"given_name": "Justin", "family_name": "Whitehouse", "institution": "Columbia University"}, {"given_name": "Junfeng", "family_name": "Yang", "institution": "Columbia University"}, {"given_name": "Suman", "family_name": "Jana", "institution": "Columbia University"}]}