{"title": "Timely Object Recognition", "book": "Advances in Neural Information Processing Systems", "page_first": 890, "page_last": 898, "abstract": "In a large visual multi-class detection framework, the timeliness of results can be crucial. Our method for timely multi-class detection aims to give the best possible performance at any single point after a start time; it is terminated at a deadline time. Toward this goal, we formulate a dynamic, closed-loop policy that infers the contents of the image in order to decide which detector to deploy next. In contrast to previous work, our method significantly diverges from the predominant greedy strategies, and is able to learn to take actions with deferred values. We evaluate our method with a novel timeliness measure, computed as the area under an Average Precision vs. Time curve. Experiments are conducted on the eminent PASCAL VOC object detection dataset. If execution is stopped when only half the detectors have been run, our method obtains $66\\%$ better AP than a random ordering, and $14\\%$ better performance than an intelligent baseline. On the timeliness measure, our method obtains at least $11\\%$ better performance. Our code, to be made available upon publication, is easily extensible as it treats detectors and classifiers as black boxes and learns from execution traces using reinforcement learning.", "full_text": "Timely Object Recognition\n\nSergey Karayev\nUC Berkeley\n\nTobias Baumgartner\n\nRWTH Aachen University\n\nMario Fritz\n\nMPI for Informatics\n\nTrevor Darrell\nUC Berkeley\n\nAbstract\n\nIn a large visual multi-class detection framework, the timeliness of results can be\ncrucial. Our method for timely multi-class detection aims to give the best possible\nperformance at any single point after a start time; it is terminated at a deadline\ntime. Toward this goal, we formulate a dynamic, closed-loop policy that infers the\ncontents of the image in order to decide which detector to deploy next. In contrast\nto previous work, our method signi\ufb01cantly diverges from the predominant greedy\nstrategies, and is able to learn to take actions with deferred values. We evaluate our\nmethod with a novel timeliness measure, computed as the area under an Average\nPrecision vs. Time curve. Experiments are conducted on the PASCAL VOC object\ndetection dataset. If execution is stopped when only half the detectors have been\nrun, our method obtains 66% better AP than a random ordering, and 14% better\nperformance than an intelligent baseline. On the timeliness measure, our method\nobtains at least 11% better performance. Our method is easily extensible, as it\ntreats detectors and classi\ufb01ers as black boxes and learns from execution traces\nusing reinforcement learning.\n\n1\n\nIntroduction\n\nIn real-world applications of visual object recognition, performance is time-sensitive. In robotics,\na small \ufb01nite amount of processing power per unit time is all that is available for robust object\ndetection, if the robot is to usefully interact with humans. In large-scale detection systems, such as\nimage search, results need to be obtained quickly per image as the number of items to process is\nconstantly growing. In such cases, an acceptable answer at a reasonable time may be more valuable\nthan the best answer given too late.\nA hypothetical system for vision-based advertising presents a case study: companies pay money to\nhave their products detected in images on the internet. The system has different values (in terms of\ncost per click) and accuracies for different classes of objects, and the queue of unprocessed images\nvaries in size. The detection strategy to maximize pro\ufb01t in such an environment has to exploit every\ninter-object context signal available to it, because there is not enough time to run detection for all\nclasses.\nWhat matters in the real world is timeliness, and either not all images can be processed or not all\nclasses can be evaluated in a detection task. Yet the conventional approach to evaluating visual\nrecognition does not consider ef\ufb01ciency, and evaluates performance independently across classes.\nWe argue that the key to tackling problems of dynamic recognition resource allocation is to start\nasking a new question: What is the best performance we can get on a budget?\nTaking the task of object detection, we propose a new timeliness measure of performance vs. time\n(shown in Figure 1). We present a method that treats different detectors and classi\ufb01ers as black\nboxes, and uses reinforcement learning to learn a dynamic policy for selecting actions to achieve the\nhighest performance under this evaluation.\nSpeci\ufb01cally, we run scene context and object class detectors over the whole image sequentially,\nusing the results of detection obtained so far to select the next actions. Evaluating on the PASCAL\n\n1\n\n\fFigure 1: A sample trace of our method. At each time step beginning at t = 0, potential actions\nare considered according to their predicted value, and the maximizing action is picked. The selected\naction is performed and returns observations. Different actions return different observations: a\ndetector returns a list of detections, while a scene context action simply returns its computed feature.\nThe belief model of our system is updated with the observations, which in\ufb02uences the selection of the\nnext action. The \ufb01nal evaluation of a detection episode is the area of the AP vs. Time curve between\ngiven start and end times. The value of an action is the expected result of \ufb01nal evaluation if the\naction is taken and the policy continues to be followed, which allows actions without an immediate\nbene\ufb01t to be scheduled.\n\nVOC dataset and evaluation regime, we are able to obtain better performance than all baselines when\nthere is less time available than is needed to exhaustively run all detectors.\n\n2 Recognition Problems and Related Work\n\nFormally, we deal with a dataset of images D, where each image I contains zero or more objects.\nEach object is labeled with exactly one category label k \u2208 {1, . . . , K}.\nThe multi-class, multi-label classi\ufb01cation problem asks whether I contains at least one object of\nclass k. We write the ground truth for an image as C = {C1, . . . , CK}, where Ck \u2208 {0, 1} is set to\n1 if an object of class k is present.\nThe detection problem is to output a list of bounding boxes (sub-images de\ufb01ned by four coordi-\nnates), each with a real-valued con\ufb01dence that it encloses a single instance of an object of class k,\nfor each k. The answer for a single class k is given by an algorithm detect(I, k), which outputs a\nlist of sub-image bounding boxes B and their associated con\ufb01dences.\nPerformance is evaluated by plotting precision vs. recall across dataset D (by progressively lowering\nthe con\ufb01dence threshold for a positive detection). The area under the curve yields the Average\nPrecision (AP) metric, which has become the standard evaluation for recognition performance on\nchallenging datasets in vision [1]. A common measure of a correct detection is the PASCAL overlap:\ntwo bounding boxes are considered to match if they have the same class label and the ratio of their\n2.\nintersection to their union is at least 1\nTo highlight the hierarchical structure of these problems, we note that the con\ufb01dences for each sub-\nimage b \u2208 B may be given by classify(b, k), and, more saliently for our setup, correct answer to the\ndetection problem also answers the classi\ufb01cation problem.\n\n2\n\nC3C2C1adet1adet2adet3agistC3C2C1adet1adet2adet3agistt=0.1t=0.3t=0C3C2C1adet1adet2adet3agistscenecontext2machinetranslationandinformationretrieval.Forex-ample,untilrecentlyspeechrecognitionandmachinetranslationsystemsbasedonn-gramlanguagemodelsoutperformedsystemsbasedongrammarsandphrasestructure.Inourexperiencemaintainingperformanceseemstorequiregradualenrichmentofthemodel.Onereasonwhysimplemodelscanperformbetterinpracticeisthatrichmodelsoftensufferfromdif\ufb01cultiesintraining.Forobjectdetection,rigidtemplatesandbag-of-featuresmodelscanbeeasilytrainedusingdiscrimi-nativemethodssuchassupportvectormachines(SVM).Richermodelsaremoredif\ufb01culttotrain,inparticularbecausetheyoftenmakeuseoflatentinformation.Considertheproblemoftrainingapart-basedmodelfromimageslabeledonlywithboundingboxesaroundtheobjectsofinterest.Sincethepartlocationsarenotlabeled,theymustbetreatedaslatent(hidden)variablesduringtraining.Whileitispossiblethatmorecompletelabelingwouldsupportbettertraining,itcouldalsoresultininferiortrainingifthelabelingusedsubop-timalparts.Automaticpartlabelinghasthepotentialtoachievebetterperformancebyautomatically\ufb01ndingeffectiveparts.Moreelaboratelabelingisalsotimecon-sumingandexpensive.TheDalal-Triggsdetector[10],whichwonthe2006PASCALobjectdetectionchallenge,usedasingle\ufb01lteronhistogramoforientedgradients(HOG)featurestorepresentanobjectcategory.TheDalal-Triggsdetectorusesaslidingwindowapproach,wherea\ufb01lterisappliedatallpositionsandscalesofanimage.Wecanthinkofthedetectorasaclassi\ufb01erwhichtakesasinputanimage,apositionwithinthatimage,andascale.Theclassi\ufb01erdetermineswhetherornotthereisaninstanceofthetargetcategoryatthegivenpositionandscale.Sincethemodelisasimple\ufb01lterwecancomputeascoreas\u00b7(x)whereisthe\ufb01lter,xisanimagewithaspeci\ufb01edpositionandscale,and(x)isafeaturevector.AmajorinnovationoftheDalal-Triggsdetectorwastheconstructionofparticularlyeffectivefeatures.Our\ufb01rstinnovationinvolvesenrichingtheDalal-Triggsmodelusingastar-structuredpart-basedmodelde\ufb01nedbya\u201croot\u201d\ufb01lter(analogoustotheDalal-Triggs\ufb01lter)plusacollectionofpart\ufb01ltersandassociateddeformationmodels.Thescoreofoneofourstarmodelsataparticularpositionandscalewithinanimageisthescoreoftheroot\ufb01lteratthegivenlocationplusthesumoverpartsofthemaximum,overplacementsofthatpart,ofthepart\ufb01lterscoreonitslocationminusadeformationcostmeasuringthedeviationofthepartfromitsideallocation.Bothrootandpart\ufb01lterscoresarede\ufb01nedbythedotproductbetweena\ufb01lter(asetofweights)andasubwindowofafeaturepyramidcomputedfromtheinputimage.Figure1showsastarmodelforthepersoncategory.Oneinterestingaspectofourmodelsisthatthefeaturesforthepart\ufb01ltersarecomputedattwicethespatialresolutionoftheroot\ufb01lter.TotrainmodelsusingpartiallylabeleddataweusealatentvariableformulationofMI-SVM[3]thatwecall(a)(b)(c)Fig.1.Detectionsobtainedwithasinglecomponentpersonmodel.Themodelisde\ufb01nedbyacoarseroot\ufb01lter(a),severalhigherresolutionpart\ufb01lters(b)andaspatialmodelforthelocationofeachpartrelativetotheroot(c).The\ufb01ltersspecifyweightsforhistogramoforientedgradientsfeatures.Theirvisualizationshowthepositiveweightsatdifferentorientations.Thevisualizationofthespatialmodelsre\ufb02ectsthe\u201ccost\u201dofplacingthecenterofapartatdifferentlocationsrelativetotheroot.latentSVM(LSVM).InalatentSVMeachexamplexisscoredbyafunctionofthefollowingform,f(x)=maxz2Z(x)\u00b7(x,z).(1)Hereisavectorofmodelparameters,zarelatentvalues,and(x,z)isafeaturevector.Inthecaseofoneofourstarmodelsistheconcatenationoftheroot\ufb01lter,thepart\ufb01lters,anddeformationcostweights,zisaspeci\ufb01cationoftheobjectcon\ufb01guration,and(x,z)isaconcatenationofsubwindowsfromafeaturepyramidandpartdeformationfeatures.Wenotethat(1)canhandleverygeneralformsoflatentinformation.Forexample,zcouldspecifyaderiva-tionunderarichvisualgrammar.Oursecondclassofmodelsrepresentseachobjectcategorybyamixtureofstarmodels.Thescoreofoneofourmixturemodelsatagivenpositionandscaleisthemaximumovercomponents,ofthescoreofthatcomponentmodelatthegivenlocation.Inthiscasethelatentinformation,z,speci\ufb01esacomponentlabelandacon\ufb01gurationforthatcomponent.Figure2showsamixturemodelforthebicyclecategory.Toobtainhighperformanceusingdiscriminativetrain-ingitisoftenimportanttouselargetrainingsets.Inthecaseofobjectdetectionthetrainingproblemishighlyun-balancedbecausethereisvastlymorebackgroundthanobjects.Thismotivatesaprocessofsearchingthrough3Fig.2.Detectionsobtainedwitha2componentbicyclemodel.Theseexamplesillustratetheimportanceofdeformationsmixturemodels.Inthismodelthe\ufb01rstcomponentcapturessidewaysviewsofbicycleswhilethesecondcomponentcapturesfrontalandnearfrontalviews.Thesidewayscomponentcandeformtomatcha\u201cwheelie\u201d.thebackgroundto\ufb01ndarelativelysmallnumberofpotentialfalsepositives.Amethodologyofdata-miningforhardnegativeex-ampleswasadoptedbyDalalandTriggs[10]butgoesbackatleasttothebootstrappingmethodsusedby[38]and[35].Hereweanalyzedata-miningalgorithmsforSVMandLSVMtraining.Weprovethatdata-miningmethodscanbemadetoconvergetotheoptimalmodelde\ufb01nedintermsoftheentiretrainingset.Ourobjectmodelsarede\ufb01nedusing\ufb01ltersthatscoresubwindowsofafeaturepyramid.WehaveinvestigatedfeaturesetssimilartoHOG[10]andfoundlowerdimen-sionalfeatureswhichperformaswellastheoriginalones.BydoingprincipalcomponentanalysisonHOGfeaturesthedimensionalityofthefeaturevectorcanbesigni\ufb01cantlyreducedwithnonoticeablelossofinforma-tion.Moreover,byexaminingtheprincipaleigenvectorswediscoverstructurethatleadsto\u201canalytic\u201dversionsoflow-dimensionalfeatureswhichareeasilyinterpretableandcanbecomputedef\ufb01ciently.Wehavealsoconsideredsomespeci\ufb01cproblemsthatariseinthePASCALobjectdetectionchallengeandsim-ilardatasets.Weshowhowthelocationsofpartsinanobjecthypothesiscanbeusedtopredictaboundingboxfortheobject.Thisisdonebytrainingamodelspeci\ufb01cpredictorusingleast-squaresregression.Wealsodemon-strateasimplemethodforaggregatingtheoutputofseveralobjectdetectors.Thebasicideaisthatobjectsofsomecategoriesprovideevidencefor,oragainst,objectsofothercategoriesinthesameimage.Weexploitthisideabytrainingacategoryspeci\ufb01cclassi\ufb01erthatrescoreseverydetectionofthatcategoryusingitsoriginalscoreandthehighestscoringdetectionfromeachoftheothercategories.2RELATEDWORKThereisasigni\ufb01cantbodyofworkondeformablemod-elsofvarioustypesforobjectdetection,includingseveralkindsofdeformabletemplatemodels(e.g.[7],[8],[21],[43]),andavarietyofpart-basedmodels(e.g.[2],[6],[9],[15],[18],[20],[28],[42]).Intheconstellationmodelsfrom[18],[42]partsareconstrainedtobeinasparsesetoflocationsdeterminedbyaninterestpointoperator,andtheirgeometricar-rangementiscapturedbyaGaussiandistribution.Incontrast,pictorialstructuremodels[15],[20]de\ufb01neamatchingproblemwherepartshaveanindividualmatchcostinadensesetoflocations,andtheirgeometricarrangementisconstrainedbyasetof\u201csprings\u201dconnect-ingpairsofparts.Thepatchworkofpartsmodelfrom[2]issimilar,butitexplicitlyconsidershowtheappearancemodelofoverlappingpartsinteracttode\ufb01neadenseappearancemodelforimages.Ourmodelsarelargelybasedonthepictorialstruc-turesframeworkfrom[15],[20].Weuseadensesetofpossiblepositionsandscalesinanimage,andde\ufb01neabicycledetectorpersondetectorTsTsTsTdTdTdtime\fMulti-class performance is evaluated by averaging the individual per-class AP values. In a special-\nized system such as the advertising case study from section 1, the metric generalizes to a weighted\naverage, with the weights set by the values of the classes.\n\n2.1 Related Work\nObject detection The best recent performance has come from detectors that use gradient-based\nfeatures to represent objects as either a collection of local patches or as object-sized windows [2, 3].\nClassi\ufb01ers are then used to distinguish between featurizations of a given class and all other possible\ncontents of an image window. Window proposal is most often done exhaustively over the image\nspace, as a \u201csliding window\u201d.\nFor state-of-the-art performance, the object-sized window models are augmented with parts [4],\nand the bag-of-visual-words models employ non-linear classi\ufb01ers [5]. We employ the widely used\nDeformable Part Model detector [4] in our evaluation.\nUsing context The most common source of context for detection is the scene or other non-detector\ncues; the most common scene-level feature is the GIST [6] of the image. We use this source of scene\ncontext in our evaluation.\nInter-object context has also been shown to improve detection [7]. In a standard evaluation setup,\ninter-object context plays a role only in post-\ufb01ltering, once all detectors have been run. In contrast,\nour work leverages inter-object context in the action-planning loop.\nA critical summary of the main approaches to using context for object and scene recognition is given\nin [8]. For the commonly used PASCAL VOC dataset [1], GIST and other sources of context are\nquantitatively explored in [9].\nEf\ufb01ciency through cascades An early success in ef\ufb01cient object detection of a single class uses\nsimple, fast features to build up a cascade of classi\ufb01ers, which then considers image regions in\na sliding window regime [10]. Most recently, cyclic optimization has been applied to optimize\ncascades with respect to feature computation cost as well as classi\ufb01er performance [11].\nCascades are not dynamic policies: they cannot change the order of execution based on observations\nobtained during execution, which is our goal.\nAnytime and active classi\ufb01cation This surprisingly little-explored line of work in vision is clos-\nest to our approach. A recent application to the problem of visual detection picks features with\nmaximum value of information in a Hough-voting framework [12]. There has also been work on\nactive classi\ufb01cation [13] and active sensing [14], in which intermediate results are considered in\norder to decide on the next classi\ufb01cation step. Most commonly, the scheduling in these approaches\nis greedy with respect to some manual quantity such as expected information gain. In contrast, we\nlearn policies that take actions without any immediate reward.\n\n3 Multi-class Recognition Policy\n\nOur goal is a multi-class recognition policy \u03c0 that takes an image I and outputs a list of multi-class\ndetection results by running detector and global scene actions sequentially.\nThe policy repeatedly selects an action ai \u2208 A, executes it, receiving observations oi, and then\nselects the next action. The set of actions A can include both classi\ufb01ers and detectors: anything that\nwould be useful for inferring the contents of the image.\nEach action ai has an expected cost c(ai) of execution. Depending on the setting, the cost can be\nde\ufb01ned in terms of algorithmic runtime analysis, an idealized property such as number of \ufb02ops, or\nsimply the empirical runtime on speci\ufb01c hardware. We take the empirical approach: every executed\naction advances t, the time into episode, by its runtime.\nAs shown in Figure 1, the system is given two times:\nthe setup time Ts and deadline Td. We\nwant to obtain the best possible answer if stopped at any given time between the setup time and\nthe deadline. A single-number metric that corresponds to this objective is the area captured under\nthe curve between the start and deadline bounds, normalized by the total area. We evaluate policies\nby this more robust metric and not simply by the \ufb01nal performance at deadline time for the same\n\n3\n\n\freason that Average Precision is used instead of a \ufb01xed Precision vs. Recall point in the conventional\nevaluations.\n\n3.1 Sequential Execution\n\nAn open-loop policy, such as the common classi\ufb01er cascade [10], takes actions in a sequence that\ndoes not depend on observations received from previous actions. In contrast, our goal is to learn a\ndynamic, or closed-loop, policy, which would exploit the signal in scene and inter-object context for\na maximally ef\ufb01cient path through the actions.\nWe refer to the information available to the decision process as the state s. The state includes the\ncurrent estimate of the distribution over class presence variables P (C) = {P (C0), . . . , P (CK)},\nwhere we write P (Ck) to mean P (Ck = 1) (class k is present in the image).\nAdditionally, the state records that an action ai has been taken by adding it to the initially empty\nset O and recording the resulting observations oi. We refer to the current set of observations as\no = {oi|ai \u2208 O}. The state also keeps track of the time into episode t, and the setup and deadline\ntimes Ts, Td.\nA recognition episode takes an image I and proceeds from the initial state s0 and action a0 to the\nnext pair (s1, a1), and so on until (sJ , aJ ), where J is the last step of the process with t \u2264 Td. At\nthat point, the policy is terminated, and a new episode can begin on a new image.\nThe speci\ufb01c actions we consider in the following exposition are detector actions adeti, where deti\nis a detector class Ci, and a scene-level context action agist, which updates the probabilities of\nall classes. Although we avoid this in the exposition, note that our system easily handles multiple\ndetector actions per class.\n\n3.2 Selecting actions\nAs our goal is to pick actions dynamically, we want a function Q(s, a) : S \u00d7A (cid:55)\u2192 R, where S is the\nspace of all possible states, to assign a value to a potential action a \u2208 A given the current state s of\nthe decision process. We can then de\ufb01ne the policy \u03c0 as simply taking the action with the maximum\nvalue:\n\n\u03c0(s) = argmax\nai\u2208A\\O\n\nQ(s, ai)\n\n(1)\n\nAlthough the action space A is manageable, the space of possible states S is intractable, and we must\nuse function approximation to represent Q(s, a): a common technique in reinforcement learning\n[15]. We featurize the state-action pair and assume linear structure:\n\nQ\u03c0(s, a) = \u03b8(cid:62)\u03c0 \u03c6(s, a)\n\n(2)\n\nThe policy\u2019s performance at time t is determined by all detections that are part of the set of observa-\ntions oj at the last state sj before t. Recall that detector actions returns lists of detection hypotheses.\nTherefore, the \ufb01nal AP vs. Time evaluation of an episode is a function eval(h, Ts, Td) of the history\nof execution h = s0, s1, . . . , sJ. It is precisely the normalized area under the AP vs. Time curve\nbetween Ts and Td, as determined by the detections in oj for all steps j in the episode.\nNote from Figure 3b that this evaluation function is additive per action, as each action a generates\nobservations that may raise or lower the mean AP of the results so far (\u2206ap) and takes a certain time\n(\u2206t). We can accordingly represent the \ufb01nal evaluation eval(h, Ts, Td) in terms of individual action\n\nrewards:(cid:80)J\n\nj=0 R(sj, aj).\n\nSpeci\ufb01cally, as shown in Figure 3b, we de\ufb01ne the reward of an action a as\n\nR(sj, a) = \u2206ap(tj\n\nT \u2212\n\n1\n2\n\n\u2206t)\n\n(3)\n\nwhere tj\nproduced by the action a. (We do not account for Ts here for clarity of exposition.)\n\nT is the time left until Td at state sj, and \u2206t and \u2206ap are the time taken and AP change\n\n4\n\n\f3.3 Learning the policy\n\nThe expected value of the \ufb01nal evaluation can be written recursively in terms of the value function:\n\n(4)\n\nsj+1[R(sj, a) + \u03b3Q\u03c0(sj+1, \u03c0(sj+1))]\n\nQ\u03c0(sj, a) = E\nwhere \u03b3 \u2208 [0, 1] is the discount value.\nWith \u03b3 = 0, the value function is determined entirely by the immediate reward, and so only com-\npletely greedy policies can be learned. With \u03b3 = 1, the value function is determined by the correct\nexpected rewards to the end of the episode. However, a lower value of \u03b3 mitigates the effects of\nincreasing uncertainty regarding the state transitions over long episodes. We set this meta-parameter\nof our approach through cross-validation, and \ufb01nd that a mid-level value (0.4) works best.\nWhile we can\u2019t directly compute the expectation in (4), we can sample it by running actual episodes\nto gather < s, a, r, s(cid:48) > samples, where r is the reward obtained by taking action a in state s, and s(cid:48)\nis the following state.\nWe then learn the optimal policy by repeatedly gathering samples with the current policy, minimizing\nthe error between the discounted reward to the end of the episode as predicted by our current Q(sj, a)\nand the actual values gathered, and updating the policy with the resulting weights.\nTo ensure suf\ufb01cient exploration of the state space, we implement \u0001-greedy action selection during\ntraining: with a probability that decreases with each training iteration, a random action is selected\ninstead of following the policy. During test time, \u0001 is set to 0.05.\nTo prevent over\ufb01tting to the training data, we use L2-regularized regression. We run 15 iterations\nof accumulating samples by running 350 episodes, starting with a baseline policy which will be\ndescribed in section 4, and cross-validating the regularization parameter at each iteration. Samples\nare not thrown away between iterations.\nWith pre-computed detections on the PASCAL VOC 2007 dataset, the training procedure takes\nabout 4 hours on an 8-core Xeon E5620 machine.\n\n3.4 Feature representation\n\nOur policy is at its base determined by a linear function of the features of the state:\n\n\u03c0(s) = argmax\nai\u2208A\\O\n\n\u03b8(cid:62)\u03c0 \u03c6(s, ai).\n\n(5)\n\nWe include the following quantities as features \u03c6(s, a):\n\nP (Ca)\n\nvations.\n\nThe prior probability of the class that corresponds to the detector of\naction a (omitted for the scene-context action).\nThe probabilities for all classes, conditioned on the current set of\nobservations.\n\nP (C0|o) . . . P (CK|o)\nH(C0|o) . . . H(CK|o) The entropies for all classes, conditioned on the current set of obser-\nAdditionally, we include the mean and maximum of [H(C0|o) . . . H(CK|o)], and 4 time features\nthat represent the times until start and deadline, for a total of F = 1 + 2K + 6 features.\nWe note that this setup is commonly used to solve Markov Decision Processes [15]. There are two\nrelated limitations of MDPs when it comes to most systems of interesting complexity, however: the\nstate has to be functionally approximated instead of exhaustively enumerated; and some aspects of\nthe state are not observed, making the problem a Partially Observed MDP (POMDP), for which\nexact solution methods are intractable for all but rather small problems [16]. Our initial solution to\nthe problem of partial observability is to include features corresponding to our level of uncertainty\ninto the feature representation, as in the technique of augmented MDPs [17].\nTo formulate learning the policy as a single regression problem, we represent the features in block\nform, where \u03c6(s, a) is a vector of size F|A|, with all values set to 0 except for the F -sized block\ncorresponding to a.\n\n5\n\n\fAs an illustration, we visualize the learned weights on these features in Figure 2, reshaped such that\neach row shows the weights learned for an action, with the top row representing the scene context\naction and then next 20 rows corresponding to the PASCAL VOC class detector actions.\n\n(a) Greedy\n\n(b) Reinforcement Learning\n\nFigure 2: Learned policy weights \u03b8\u03c0 (best viewed in color: red corresponds to positive, blue to\nnegative values). The \ufb01rst row corresponds to the scene-level action, which does not generate de-\ntections itself but only helps reduce uncertainty about the contents of the image. Note that in the\ngreedy learning case, this action is learned to never be taken, but it is shown to be useful in the\nreinforcement learning case.\n\n3.5 Updating with observations\n\nThe bulk of our feature representation is formed by probability of individual class occurrence, con-\nditioned on the observations so far: P (C0|o) . . . P (CK|o). This allows the action-value function\nto learn correlations between presence of different classes, and so the policy can look for the most\nprobable classes given the observations.\nHowever, higher-order co-occurrences are not well represented in this form. Additionally, updating\nP (Ci|o) presents choices regarding independence assumptions between the classes. We evaluate\ntwo approaches for updating probabilities: direct and MRF.\nIn the direct method, P (Ci|o) = score(Ci) if o includes the observations for class Ci and\nP (Ci|o) = P (Ci) otherwise. This means that an observation of class i does not directly in\ufb02u-\nence the estimated probability of any class but Ci.\nThe MRF approach employs a pairwise fully-connected Markov Random Field (MRF), as shown in\nFigure 1, with the observation nodes set to score(Ci) appropriately, or considered unobserved.\nThe graphical model structure is set as fully-connected, but some classes almost never co-occurr\nin our dataset. Accordingly, the edge weights are learned with L1 regularization, which obtains a\nsparse structure [18]. All parameters of the model are trained on fully-observed data, and Loopy\nBelief Propagation inference is implemented with an open-source graphical model package [19].\nAn implementation detail: score(Ci) for adeti is obtained by training a probabilistic classi\ufb01er on\nthe list of detections, featurized by the top few con\ufb01dence scores and the total number of detections.\nSimilarly, score(Ci) for agist is obtained by training probabilistic classi\ufb01ers on the GIST feature,\nfor all classes.\n\n4 Evaluation\n\nWe evaluate our system on the multi-class, multi-label detection task, as previously described. We\nevaluate on a popular detection challenge task: the PASCAL VOC 2007 dataset [1]. This datasets\nexhibits a rather modest amount of class co-occurrence: the \u201cperson\u201d class is highly likely to occur,\nand less than 10% of the images have more than two classes.\nWe learn weights on the training and validation sets, and run our policy on all images in the testing\nset. The \ufb01nal evaluation pools all detections up to a certain time, and computes their multi-class AP\nper image, averaging over all images. This is done for different times to plot the AP vs. Time curve\nover the whole dataset. Our method of averaging per-image performance follows [20].\n\n6\n\nP(C|o)H(C|o)GISTactionGISTactiontimeP(Ca)RLGreedyP(C|o)H(C|o)GISTactionGISTactiontimeP(Ca)RLGreedyP(C|o)H(C|o)GISTactionGISTactiontimeP(Ca)RLGreedy\fFor the detector actions, we use one-vs-all cascaded deformable part-model detectors on a HOG\nfeaturization of the image [21], with linear classi\ufb01cation of the list of detections as described in\nthe previous section. There are 20 classes in the PASCAL challenge task, so there are 20 detector\nactions. Running a detector on a PASCAL image takes about 1 second.\nWe test three different settings of the start and deadline times. In the \ufb01rst one, the start time is\nimmediate and execution is cut off at 20 seconds, which is enough time to run all actions. In the\nsecond one, execution is cut off after only 10 seconds. Lastly, we measure performance between 5\nseconds and 15 seconds. These operating points show how our method behaves when deployed in\ndifferent conditions. The results are given in rows of Table 1.\n\n(a)\n\n(b)\n\nFigure 3: (a) AP vs. Time curves for Random, Oracle, the Fixed Order baseline, and our best-\nperforming policy. (b) Graphically representing our reward function, as described in section 3.2.\n\nWe establish the \ufb01rst baseline for our system by selecting actions randomly at each step. As shown\nin Figure 3a, the Random policy results in a roughly linear gain of AP vs. time. This is expected:\nthe detectors are capable of obtaining a certain level of performance; if half the detectors are run,\nthe expected performance level is half of the maximum level.\nTo establish an upper bound on performance, we plot the Oracle policy, obtained by re-ordering the\nactions at the end of each detection episode in the order of AP gains they produced.\nWe consider another baseline: selecting actions in a \ufb01xed order based on the value they bring to the\nAP vs. Time evaluation, which is roughly proportional to their occurrence probability. We refer to\nthis as Fixed Order.\nThen there are instantiations of our method, as described in the previous section : RL w/ Direct\ninference and RL w/ MRF inference. As the MRF model consistently outperformed Direct by a\nsmall margin, we report results for that model only.\nIn Figure 3a, we can see that due to the dataset bias, the \ufb01xed-order policy performs well at \ufb01rst, as\nthe person class is disproportionately likely to be in the image, but is signi\ufb01cantly overtaken by our\nmodel as execution goes on and more rare classes have to be detected.\nLastly, we include an additional scene-level GIST feature that updates the posterior probabilities of\nall classes. This is considered one action, and takes about 0.3 seconds. This setting always uses the\nMRF model to properly update the class probabilities with GIST observations. This brings another\nsmall boost in performance. The results are shown in Table 1.\nVisualizing the learned weights in Figure 2, we note that the GIST action is learned to never be taken\nin the greedy (\u03b3 = 0) setting, but is learned to be taken with a higher value of \u03b3. It is additionally\ninformative to consider the action trajectories of different policies in Figure 4.\n\n7\n\nTstjTTdtapap(tjT12t)\fFigure 4: Visualizing the action trajectories of different policies. Action selection traces are plotted\nin orange over many episodes; the size of the blue circles correspond to the increase in AP obtained\nby the action. We see that the Random policy selects actions and obtains rewards randomly, while\nthe Oracle policy obtains all rewards in the \ufb01rst few actions. The Fixed Order policy selects actions\nin a static optimal order. Our policy does not stick a static order but selects actions dynamically to\nmaximize the rewards obtained early on.\n\nTable 1: The areas under the AP vs. Time curve for different experimental conditions.\n\nBounds Random Fixed Order\n(0,20)\n(0,10)\n(5,15)\n\n0.250\n0.119\n0.257\n\n0.342\n0.240\n0.362\n\nRL\n0.378\n0.266\n0.418\n\nRL w/ GIST Oracle\n0.488\n0.464\n0.530\n\n0.382\n0.267\n0.420\n\n5 Conclusion\nWe presented a method for learning \u201cclosed-loop\u201d policies for multi-class object recognition, given\nexisting object detectors and classi\ufb01ers and a metric to optimize. The method learns the optimal\npolicy using reinforcement learning, by observing execution traces in training. If detection on an\nimage is cut off after only half the detectors have been run, our method does 66% better than a\nrandom ordering, and 14% better than an intelligent baseline. In particular, our method learns to\ntake action with no intermediate reward in order to improve the overall performance of the system.\nAs always with reinforcement learning problems, de\ufb01ning the reward function requires some manual\nwork. Here, we derive it for the novel detection AP vs. Time evaluation that we suggest is useful\nfor evaluating ef\ufb01ciency in recognition. Although computation devoted to scheduling actions is less\nsigni\ufb01cant than the computation due to running the actions, the next research direction is to explicitly\nconsider this decision-making cost; the same goes for feature computation costs. Additionally, it is\ninteresting to consider actions de\ufb01ned not just by object category but also by spatial region. The\ncode for our method is available1.\n\nAcknowledgments\n\nThis research was made with Government support under and awarded by DoD, Air Force Of\ufb01ce of\nScienti\ufb01c Research, National Defense Science and Engineering Graduate (NDSEG) Fellowship, 32\nCFR 168a.\n\n1http://sergeykarayev.com/work/timely/\n\n8\n\n\fReferences\n[1] M Everingham, L Van Gool, C K I Williams, J Winn, and A Zisserman. The PASCAL VOC Challenge.\n\nhttp://www.pascal-network.org/challenges/VOC/, 2010. 2, 3, 6\n\n[2] N Dalal and B Triggs. Histograms of Oriented Gradients for Human Detection. In CVPR, pages 886\u2013893,\n\n2005. 3\n\n[3] David G Lowe. Distinctive Image Features from Scale-Invariant Keypoints. IJCV, 60(2):91\u2013110, Novem-\n\nber 2004. 3\n\n[4] Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with\n\ndiscriminatively trained part-based models. PAMI, 32(9):1627\u20131645, September 2010. 3\n\n[5] Andrea Vedaldi, Varun Gulshan, Manik Varma, and Andrew Zisserman. Multiple kernels for object\n\ndetection. ICCV, pages 606\u2013613, September 2009. 3\n\n[6] Aude Oliva and Antonio Torralba. Modeling the Shape of the Scene: A Holistic Representation of the\n\nSpatial Envelope. IJCV, 42(3):145\u2013175, 2001. 3\n\n[7] Antonio Torralba, Kevin P Murphy, and William T Freeman. Contextual Models for Object Detection\n\nUsing Boosted Random Fields. MIT CSAIL Technical Report, 2004. 3\n\n[8] Carolina Galleguillos and Serge Belongie. Context based object categorization: A critical survey. Com-\n\nputer Vision and Image Understanding, 114(6):712\u2013722, June 2010. 3\n\n[9] Santosh K Divvala, Derek Hoiem, James H Hays, Alexei A Efros, and Martial Hebert. An empirical study\n\nof context in object detection. In CVPR, pages 1271\u20131278, June 2009. 3\n\n[10] Paul Viola and Michael Jones. Rapid object detection using a boosted cascade of simple features. In\n\nCVPR, 2001. 3, 4\n\n[11] Minmin Chen, Zhixiang (Eddie) Xu, Kilian Q Weinberger, Olivier Chapelle, and Dor Kedem. Classi\ufb01er\n\nCascade for Minimizing Feature Evaluation Cost. In AISTATS, 2012. 3\n\n[12] Sudheendra Vijayanarasimhan and Ashish Kapoor. Visual Recognition and Detection Under Bounded\n\nComputational Resources. In CVPR, pages 1006\u20131013, 2010. 3\n\n[13] Tianshi Gao and Daphne Koller. Active Classi\ufb01cation based on Value of Classi\ufb01er. In NIPS, 2011. 3\n[14] Shipeng Yu, Balaji Krishnapuram, Romer Rosales, and R Bharat Rao. Active Sensing. In AISTATS, pages\n\n639\u2013646, 2009. 3\n\n[15] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 1998. 4, 5\n[16] Nicholas Roy and Geoffrey Gordon. Exponential Family PCA for Belief Compression in POMDPs. In\n\nNIPS, 2002. 5\n\n[17] Cody Kwok and Dieter Fox. Reinforcement Learning for Sensing Strategies. In IROS, 2004. 5\n[18] Su-In Lee, Varun Ganapathi, and Daphne Koller. Ef\ufb01cient Structure Learning of Markov Networks using\n\nL1-Regularization. In NIPS, 2006. 6\n\n[19] Ariel Jaimovich and Ian Mcgraw. FastInf: An Ef\ufb01cient Approximate Inference Library. Journal of\n\nMachine Learning Research, 11:1733\u20131736, 2010. 6\n\n[20] Chaitanya Desai, Deva Ramanan, and Charless Fowlkes. Discriminative models for multi-class object\n\nlayout. In ICCV, pages 229\u2013236, September 2009. 6\n\n[21] Pedro F Felzenszwalb, Ross B Girshick, and David McAllester. Cascade object detection with deformable\n\npart models. In CVPR, pages 2241\u20132248. IEEE, June 2010. 7\n\n9\n\n\f", "award": [], "sourceid": 413, "authors": [{"given_name": "Sergey", "family_name": "Karayev", "institution": null}, {"given_name": "Tobias", "family_name": "Baumgartner", "institution": null}, {"given_name": "Mario", "family_name": "Fritz", "institution": null}, {"given_name": "Trevor", "family_name": "Darrell", "institution": null}]}