{"title": "Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition", "book": "Advances in Neural Information Processing Systems", "page_first": 853, "page_last": 865, "abstract": "Human activities often occur in specific scene contexts, e.g., playing basketball on a basketball court. Training a model using existing video datasets thus inevitably captures and leverages such bias (instead of using the actual discriminative cues). The learned representation may not generalize well to new action classes or different tasks. In this paper, we propose to mitigate scene bias for video representation learning. Specifically, we augment the standard cross-entropy loss for action classification with 1) an adversarial loss for scene types and 2) a human mask confusion loss for videos where the human actors are masked out. These two losses encourage learning representations that are unable to predict the scene types and the correct actions when there is no evidence. We validate the effectiveness of our method by transferring our pre-trained model to three different tasks, including action classification, temporal localization, and spatio-temporal action detection. Our results show consistent improvement over the baseline model without debiasing.", "full_text": "Why Can\u2019t I Dance in the Mall?\nLearning to Mitigate Scene Bias\n\nin Action Recognition\n\nJinwoo Choi\nVirginia Tech\n\njinchoi@vt.edu\n\nChen Gao\nVirginia Tech\n\nchengao@vt.edu\n\nJoseph C. E. Messou\n\nVirginia Tech\n\nmejc2014@vt.edu\n\nJia-Bin Huang\nVirginia Tech\n\njbhuang@vt.edu\n\nFigure 1: Quiz time! Can you guess what action the (blocked) person is doing in the four videos?\nEven though we cannot see a human actor, we can easily predict the action by considering where the\nscene is. Training a CNN model from these examples may lead to a strong bias toward recognizing\nthe scene or the objects present in the video as opposed to paying attention to the actual action the\nperson is doing. In this work, we show that learning video representation with debiasing leads to\nimproved generalization to novel classes and tasks.\n\nAbstract\n\nHuman activities often occur in speci\ufb01c scene contexts, e.g., playing basketball on\na basketball court. Training a model using existing video datasets thus inevitably\ncaptures and leverages such bias (instead of using the actual discriminative cues).\nThe learned representation may not generalize well to new action classes or differ-\nent tasks. In this paper, we propose to mitigate scene bias for video representation\nlearning. Speci\ufb01cally, we augment the standard cross-entropy loss for action classi-\n\ufb01cation with 1) an adversarial loss for scene types and 2) a human mask confusion\nloss for videos where the human actors are masked out. These two losses encourage\nlearning representations that are unable to predict the scene types and the correct\nactions when there is no evidence. We validate the effectiveness of our method\nby transferring our pre-trained model to three different tasks, including action\nclassi\ufb01cation, temporal localization, and spatio-temporal action detection. Our\nresults show consistent improvement over the baseline model without debiasing.\n\n1\n\nIntroduction\n\nConvolutional neural networks (CNNs) [8, 64, 52, 60] have demonstrated impressive performance\non action recognition datasets such as the Kinetics [31], UCF-101 [50], HMDB-51 [34], and others.\nThese CNN models, however, may sometimes make correct predictions for wrong reasons, such as\nleveraging scene context or object information instead of focusing on actual human actions in videos.\nFor example, CNN models may recognize a classroom or a whiteboard in an input video and predict\nthe action in the video as giving a lecture, as opposed to paying attention to the actual activity in\nthe scene, which could be, for example, eating, jumping or even dancing. Such biases are known\nas representation bias [38]. In this paper, we focus on mitigating the effect of scene representation\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fFigure 2: Motivation of the proposed debiasing algorithm. (Left): A man is singing in a baseball\n\ufb01eld. However, representations with certain bias toward the scene may predict the incorrect action\ne.g., , playing baseball. (Right): A person (masked-out) is swimming in a swimming pool. A model\nis capable of predicting the correct action i.e., swimming even without looking at the evidence. Video\nrepresentations that make correct (or incorrect) predictions by leveraging the scene bias may not\ngeneralize well to unseen action classes and tasks.\n\nbias [38]. Following Li et al. [38], we de\ufb01ne the scene representation bias of a dataset D as\n\nBscene = log[M(D,\u03c6scene)/Mrand].\n\n(1)\n\nHere, \u03c6scene is a scene representation; M(D,\u03c6scene) is an action classi\ufb01cation accuracy with a scene\nrepresentation \u03c6scene on the dataset D; Mrand is a random chance action classi\ufb01cation accuracy on\nthe dataset D. With this de\ufb01nition, we can now measure the scene representation bias of the UCF-\n101 [50] dataset by computing the log ratio between two accuracies: 1) the action classi\ufb01cation\naccuracy of ResNet-50 backbone pre-trained on the Places365 dataset [73] on UCF-101: 59.7%. 2)\nthe random chance accuracy on UCF-101: 1.0%. To get the \ufb01rst accuracy, a linear classi\ufb01er is trained\non UCF-101 on top of the Places365 pre-trained ResNet-50 feature backbone. As an input to the\nlinear classi\ufb01er, we use a 2,048 dimensional feature vector extracted from the penultimate layer of\nthe ResNet-50 feature backbone. The scene representation bias of UCF-101 is log(59.7/1.0) = 4.09.\nA completely unbiased dataset would have log(1.0/1.0) = 0 scene representation bias. Thus, the\nUCF-101 dataset has quite a large scene representation bias.\nThe reason for a scene representation bias is that human activities often occur in speci\ufb01c scene\ncontexts (e.g., playing basketball in a basketball court). Figure 1 provides several examples. Even\nthough the actors in these videos are masked-out, we can still easily infer the actions of the masked-\nout actors by reasoning about where the scene is. As a result, training CNN models on these examples\nmay capture the biases towards recognizing scene contexts. Such strong scene bias could make CNN\nmodels unable to generalize to unseen action classes in the same scene context and novel tasks.\nIn this paper, we propose a debiasing algorithm to mitigate scene bias of CNNs for action understand-\ning tasks. Speci\ufb01cally, we pre-train a CNN on an action classi\ufb01cation dataset (Mini-Kinetics-200\ndataset [64] in our experiment) using the standard cross-entropy loss for the action labels. To miti-\ngate scene representation bias, we introduce two additional losses: (1) scene adversarial loss that\nencourages a network to learn scene-invariant feature representations and (2) human mask confusion\nloss that prevents a model from predicting an action if humans are not visible in the video. We\nvalidate the proposed debiasing method by showing transfer learning results on three different activity\nunderstanding tasks: action classi\ufb01cation, temporal action localization, and spatio-temporal action\ndetection. Our debiased model shows the consistent performance improvement of transfer learning\nover the baseline model without debiasing across various tasks.\nWe make the following three contributions in this paper.\n\ngeneralization to various action understanding tasks.\n\n\u2022 We tackle a relatively under-explored problem of mitigating scene biases of CNNs for better\n\u2022 We propose two novel losses for mitigating scene biases when pre-training a CNN. We use\na scene-adversarial loss to obtain scene-invariant feature representation. We use a human\nmask confusion loss to encourage a network to be unable to predict correct actions when\nthere is no visual evidence.\n\u2022 We demonstrate the effectiveness of our method by transferring our pre-trained model to\n\nthree action understanding tasks and show consistent improvements over the baseline.\n\n2\n\n\fFigure 3: Overview of the proposed approach for learning debiased video representation. Here,\nour goal is to learn the parameters \u03b8 f for the feature extractor \u03c6 by pre-training it on a large-scale\nvideo classi\ufb01cation task. Our training involves three types of losses. First, we use a standard cross-\nentropy loss LCE for training action classi\ufb01cation. Second, we impose a scene adversarial loss LAdv\nso that one cannot infer the scene types based on the learned representation. Third, we prepare\nadditional videos by detecting and masking out the humans using an off-the-shelf human detector.\nWe apply an entropy loss LEnt on the predicted action class distributions of the human-masked-out\nvideos. Our intuition here is that the model should not be able to infer the correct action without\nseeing the evidence.\n\n2 Related Work\n\nMitigating biases. Mitigating unintended biases is a critical challenge in machine learning. Examples\ninclude reducing gender or age biases for fairness [2, 5, 33, 58], easy-example biases [59, 49], and\ntexture bias of image CNNs for improved generalization [17]. The most closely related work to ours\nis on mitigating scene/object/people biases by resampling the original dataset [38] to generate less\nbiased datasets for action recognition. In contrast, we learn scene-invariant video representations\nfrom an original biased dataset without resampling.\nUsing scene context. Leveraging scene context is useful for object detection [40, 7, 11, 13], semantic\nsegmentation [35, 39, 40, 69], predicting invisible things [32], and action recognition without looking\nat the human [23, 54]. Some work have shown that explicitly factoring human action out of context\nleads to improved performance in action recognition [71, 61]. In contrast to prior work that uses\nscene contexts to facilitate recognition, our method aims to learn representations that are invariant to\nscene bias. We show that the debiased model generalizes better to new datasets and tasks.\nAction recognition in video. State-of-the-art action recognition models either use two-stream (RGB\nand \ufb02ow) networks [46, 14] or 3D CNNs [8, 52, 64, 60, 20]. Recent advances in this \ufb01eld focus on\ncapturing longer-range temporal dependencies [55, 56, 63]. Instead, our work focuses on mitigating\nscene bias when training these models on existing video datasets. Our debiasing approach is model-\nagnostic. We show the proposed debiasing losses improve the performance on different backbone\narchitectures: 3D-ResNet-18 [20] and VGG-16 network [47].\nVideo representation transfer for action understanding tasks. Many recent CNNs for action\nclassi\ufb01cation [8, 52, 64, 60, 20], temporal action localization [66, 45, 65, 12, 70], and spatio-temporal\naction detection [19, 42, 41, 48, 4, 44, 62, 30, 74, 21, 26, 43] rely on pre-training a model on a\nlarge-scale dataset such as ImageNet or Kinetics and \ufb01netuning the pre-trained model on the target\ndatasets for different tasks. While Kinetics is a large-scale video dataset, it still contains a signi\ufb01cant\nscene representation bias [38]. In light of this, we propose to mitigate scene bias for pre-training a\nmore generalizable video representation.\n\n3\n\nAction labelPseudo scene soft labelEq. (4)ActionclassifierGradient reversalEq. (3)SceneclassifierEq. (2)ActionclassifierHuman masked videoOriginal video-dim-dim-dimSharedFeature extractorFeature extractor\fAdversarial training. Domain adversarial training introduces a discriminator to determine where\nthe data is coming from. It has been successfully applied to unsupervised domain adaptation [16, 53]\nand later extended to pixel-wise adaptation [6, 24, 10] and multi-source adaptation [25]. Building\nupon adversarial training, recent work aims to mitigate unintended biases by using the adversary\nto predict the protected variables [68, 58] or quantify the statistical dependency between the inputs\nand the protected variables [1]. Our work uses a similar strategy for mitigating scene bias from\nvideo representation. Unlike existing work where the protected variables are given (demographic\ninformation such as gender, age, race), the ground truth scene labels of our training videos are not\navailable. To address this, we propose to use the output of a pre-trained scene classi\ufb01er as a proxy.\nArti\ufb01cial occlusion. Applying occlusion masks to input images (or features) and monitoring the\nchanges at the output of a model has been used to visualizing whether a classi\ufb01er can localize objects\nin an image [67], learning object detectors from weak image-level labels [3, 49, 36, 37], improving\nrobustness of object detectors [59], and regularizing model training [18]. We adopt a similar approach\nby masking out humans detected by an off-the-shelf object detector. Our focus, however, differs from\nexisting work in that we aim to train the model so that it is unable to predict the correct class label.\n\n3 Method\n3.1 Overview of the method\nPre-training. We show an overview of the proposed approach for learning debiased video repre-\nsentations in Figure 3. Our goal is to learn parameters \u03b8 f of a feature extractor G by pre-training\nit on a large-scale video classi\ufb01cation dataset. Given a video and the corresponding action label\n(x,y) \u2208 X\u00d7 Y, where X is a video dataset and Y is the action label set with N classes, we extract\nfeatures using a CNN denoted as G\u03b8 f with parameters \u03b8 f . Note that our method is model-agnostic in\nthat we can use any 3D CNN or 2D CNN as our feature extractor. We feed the extracted features\ninto an action classi\ufb01er f\u03b8A with parameters \u03b8A. We use a standard cross-entropy loss for the action\nclassi\ufb01er for penalizing the incorrect action predictions as follows.\n\nLCE = \u2212E(x,y)\u223c(X,Y)\n\nN\n\n\u2211\n\nk=1\n\nyk log f\u03b8A (G\u03b8 f (x)).\n\n(2)\n\nIn addition to the cross-entropy loss in (2), we propose two losses for debiasing purpose: 1) scene\nadversarial loss LAdv, and 2) human mask confusion loss LEnt. We impose a scene adversarial loss to\nencourage learning scene-invariant representations. The scene adversarial loss penalizes the model\nif it could infer the scene types based on the learned representation. The human mask confusion\nloss penalizes the model if it could predict the actions when the humans in the video are masked-out.\nFor this loss, we \ufb01rst detect humans in the input video and mask-out the detected humans. We\nthen extract features of the human-masked-out video. We feed the features into the same action\nclassi\ufb01er. As shown in Figure 3 (dotted line), the weights of the feature extractor \u03b8 f and action\nclassi\ufb01ers \u03b8A for the human mask confusion loss are shared with those for the action classi\ufb01cation\ncross-entropy loss. We maximize the entropy of the predicted action class distributions when the\ninput is a human-masked-out video. We provide more details on each of the two losses and the\noptimization process in the following subsections.\nTransfer learning. After pre-training a CNN with the proposed debiasing method, we initialize\nthe weights of the feature extractor, \u03b8 f for downstream target tasks: action classi\ufb01cation, temporal\naction localization, and spatio-temporal action detection. We remove the scene prediction and the\nhuman-masked action prediction heads of the network i.e., \u03b8A, \u03b8S. Then we \ufb01netune \u03b8 f and the\ntask-speci\ufb01c classi\ufb01cation and regression parameters on the target datasets.\n\n3.2 Scene adversarial loss\n\nThe motivation behind the scene adversarial loss is that we want to learn feature representations\nsuitable for action classi\ufb01cation but invariant to scene types. For example, on Figure 2 left, we aim to\nencourage a network to focus on a singer in the video and to predict the action as singing. We do\nnot want a network to classify the video as playing baseball because the network recognizes that the\nscene is a baseball \ufb01eld. To enforce the scene invariance criterion to the network, we learn a scene\nclassi\ufb01er f\u03b8S with parameters \u03b8S on top of the feature extractor G\u03b8 f in an adversarial fashion. We\n\n4\n\n\fde\ufb01ne the scene adversarial loss as,\n\nM\n\n\u2211\n\nLAdv = \u2212E(x,p)\u223c(X,P)\n\npm log f\u03b8S (G\u03b8 f (x)).\n\n(3)\nHere we denote a scene label as p \u2208 P, and the number of scene types by M. The loss (3) is adversarial\nin that the parameters of the feature extractor \u03b8 f maximize the loss while the parameters of the scene\nclassi\ufb01er \u03b8S minimize the loss.\nPseudo scene label. Most of the action recognition datasets such as Kinetics and UCF-101 do not\nhave scene annotations. In this work, we obtain a pseudo scene label \u02dcp \u2208 \u02dcP by running Places365\ndataset pre-trained ResNet-50 [73] on the Kinetics dataset.\n\nm=1\n\n3.3 Human mask confusion loss\n\nWe show the motivation for using the human mask confusion loss on the right of Figure 2. If every\nhuman, (who is swimming in this example), in the input video is masked out, we aim to make the\nnetwork unable to infer the true action label (swimming). We denote a human-masked-out video\nas xhm \u2208 Xhm. We de\ufb01ne the human mask confusion loss as an entropy function of the action label\ndistributions as follows.\n\nLEnt = \u2212Exhm\u223cXhm\n\nN\n\n\u2211\n\nk=1\n\nf\u03b8A(G\u03b8 f (xhm))log f\u03b8A(G\u03b8 f (xhm)).\n\n(4)\n\nBoth of the parameters of the feature extractor \u03b8 f and the action classi\ufb01er \u03b8A maximize (4) when\nan input is a human-masked-out video xhm. Our intuition here is that models should not be able to\npredict correct action without seeing the evidence.\nHuman mask. To mask-out humans in videos, we run an off-the-shelf human detector [22] on the\nKinetics dataset of\ufb02ine and store the detection results. During training, we load the cached human\ndetection results. For every human bounding box in a frame, we \ufb01ll in the human-mask regions with\nthe average pixel value of the video frame, following the setting from Anne Hendricks et al. [2].\n\n3.4 Optimization\n\nUsing all three losses, we de\ufb01ne the optimization problem as follows. When an input video is an\noriginal video x without human masking, the optimization is\n\nL(\u03b8 f ,\u03b8S,\u03b8A) = LCE (\u03b8 f ,\u03b8A)\u2212 \u03bb LAdv(\u03b8 f ,\u03b8S),\n\n(\u03b8\u2217\n\nf ,\u03b8\u2217\nA) = argmin\n\u03b8 f ,\u03b8A\n\u03b8\u2217\nS = argmax\n\nL(\u03b8 f ,\u03b8\u2217\nL(\u03b8\u2217\n\nS ,\u03b8A),\nf ,\u03b8S,\u03b8\u2217\nA).\n\n\u03b8S\n\n(5)\n\nHere \u03bb is a hyperparameter for controlling the strength of the scene adversarial loss. We use the\ngradient reversal layer for adversarial training [15]. When an input video is a human-masked-out\nvideo xhm, the optimization is\n\n(\u03b8\u2217\n\nf ,\u03b8\u2217\n\nA) = argmax\n\u03b8 f ,\u03b8A\n\nLEnt(\u03b8 f ,\u03b8A).\n\n(6)\n\nFor every iteration, we alternate between the optimization (5) and (6).\n\n4 Experimental Results\n\nWe start with describing the datasets used in our experiments (Section 4.1) and implementation details\n(Section 4.2). We then address the following questions: i) Does the proposed debiasing method\nmitigate scene representation bias? (Section 4.3) ii) Can debiasing improve generalization to other\ntasks? (Section 4.4, Section 4.5) iii) What is the effect of the two proposed losses designed for\nmitigating scene bias? What is the effect of using different types of pseudo scene labels? (Section 4.6)\n\n5\n\n\f4.1 Datasets\n\nPre-training. We pre-train our model on the Mini-Kinetics-200 dataset [64]. Mini-Kinetics-200 is\na subset of the full Kinetics-400 dataset [31]. Since the full Kinetics is very large and we do not\nhave suf\ufb01cient computational resources, we resort to using Mini-Kinetics-200 for pre-training a\nmodel. The training set consists of 80K videos and the validation set consists of 5K videos.1 To\nvalidate whether our proposed debiasing method improves generalization, we pre-train models with\ndebiasing and another model without debiasing. We then compare the transfer learning performances\nof the two models on three target tasks: 1) action classi\ufb01cation, 2) temporal action localization, 3)\nspatio-temporal action detection.\nAction classi\ufb01cation. For the action classi\ufb01cation task, we evaluate the transfer learning perfor-\nmance on the UCF-101 [50], HMDB-51 [34], and Diving48 [38] datasets. UCF-101 consists of\n13,320 videos with 101 action classes. HMDB-51 consists of 6,766 videos with 51 action classes.\nDiving48 [38] is an interesting dataset as it contains no signi\ufb01cant biases towards the scene, object,\nand human. Diving48 consists of 18K videos with 48 \ufb01ne-grained diving action classes. We use the\ntrain/test split provided by Li et al. [38]. Videos in all three datasets were temporally trimmed. We\nreport top-1 accuracy on all thee splits of the UCF-101 and the HMDB-51 datasets. The Diving48\ndataset provides only one split. Thus we report the top-1 accuracy on this split.\nTemporal action localization. Temporal action localization is a task to not only predict action labels\nbut also localize the start and end time of the actions. We use the THUMOS-14 [29] dataset as our\ntestbed. THUMOS-14 contains 20 action classes. We follow Xu et al. [65]\u2019s setting for training and\ntesting. We train our model on the temporally annotated validation set with 200 videos. We test our\nmodel on the test set consisting of 213 videos. We report the video mean average precision at various\nIoU threshold values.\nSpatio-temporal action detection. Given an untrimmed video, the task of spatio-temporal action\ndetection aims to predict action label(s) and also localize the person(s) performing the action in\nboth space and time (e.g., an action tube). We use the standard JHMDB [27] dataset for this task.\nJHMDB is a subset of HMDB-51. It consists of 928 videos with 21 action categories with frame-level\nbounding box annotations. We evaluate models on all three splits of JHMDB. We report the frame\nmean average precision at the IoU threshold 0.5 as our evaluation metric.\n\n4.2\n\nImplementation details\n\nWe implement our method with PyTorch (version 0.4.1). We choose 3D-ResNet-18 [20] as our\nfeature backbone for Mini-Kinetics-200 \u2192 UCF-101/HMDB51/Diving48, and Mini-Kinetics-200 \u2192\nTHUMOS-14 experiments because open-source implementations for action classi\ufb01cation [20] and\ntemporal action localization [57] are available online. We use a 3 channels \u00d7 16 frames \u00d7 112 pixels\n\u00d7 112 pixels clip as our input to the 3D-ResNet-18 model. We use the last activations of the Conv5\nblock of the 3D-ResNet-18 as our feature representation G\u03b8 f (x).\nFor the spatio-temporal action detection task, we adopt the frame-level action detection code [48]\nbased on the VGG-16 network [47]. We use a 3 channels \u00d7 1 frame \u00d7 300 pixels \u00d7 300 pixels frame\nas our input to the VGG-16 network. We use the fc7 activations of the VGG-16 network as our\nfeature representation G\u03b8 f (x).\nWe use a four-layer MLP as our scene classi\ufb01er, where the hidden fully connected layers have 512\nunits each. We choose \u03bb = 0.5 for the gradient reversal layer [15] using cross-validation. We set the\nbatch size as 32 with two P100 GPUs. We use SGD with a momentum of 0.9 as our optimizer. We set\nthe weight decay as 0.00001. The learning rate starts from 0.001, and we divide it by 10 whenever\nthe validation loss saturates. We train our network for 100 epochs. We use the validation loss on\nMini-Kinetics-200 for model selection and hyperparameter tuning. When we conduct the transfer\nlearning on the target datasets of the target tasks, we follow the same hyperparameter settings of\nHara et al. [20], Wang and Cheng [57] and Singh et al. [48] for action classi\ufb01cation, temporal action\nlocalization, and spatio-temporal action detection, respectively.\n\n1As the sources of Kinetics dataset are from YouTube videos, some of the videos are no longer available.\n\nThe exact number of training videos we used is 76,103, and the number of validation videos is 4,839.\n\n6\n\n\fTable 1: Transfer learning results on action classi\ufb01cation. The video representation trained using the\nproposed debiasing techniques consistently improves the accuracy on new datasets. For HMDB-51\nand UCF-101, we show the average accuracy of all three test splits. All methods but TSN use a clip\nlength of 16. In the \ufb01rst block, we list the accuracies of the other methods.\n\nMethod\nC3D [61]\nFactor-C3D [61]\nRESOUND-C3D [38]\nTSN [55]\n3D-ResNet-18 [20]\n3D-ResNet-18 [20] + debiased (ours)\n\nBackbone\nC3D [51]\nC3D [51]\nC3D [28]\n\nBN-Inception\n3D-ResNet-18\n3D-ResNet-18\n\nHMDB-51 UCF-101 Diving48\n\n-\n-\n-\n\n51.0\n53.6\n56.7\n\n82.3\n84.5\n\n-\n\n85.1\n83.5\n84.5\n\n-\n-\n\n16.4\n16.8\n18.0\n20.5\n\n4.3 Scene classi\ufb01cation accuracy\nOn the Mini-Kinetics-200 validation set, our scene classi\ufb01er achieves an accuracy of 29.7% when\ntraining the action classi\ufb01cation without debiasing (i.e., , with standard cross-entropy loss only).2\nWith debiasing, the scene classi\ufb01cation accuracy drops to 2.9% (the accuracy of random guess is\n0.3%.) The proposed debiasing method signi\ufb01cantly reduces scene-dependent features.\n\n)\ne\nv\ni\nt\na\nl\ne\nr\n(\n\nt\nn\ne\nm\ne\nv\no\nr\np\nm\n\nI\n\nScene representation bias\n\nFigure 4: A strong negative correlation between relative performance improvement and scene\nrepresentation bias. We measure the scene representation bias de\ufb01ned as (1) and the relative\nimprovement of each split of the HMDB-51, UCF-101, Diving48 datasets between models trained\nwithout and with the proposed debiasing method. The Pearson correlation is \u03c1 = \u22120.896 with a\np-value 0.006, highlighting a strong negative correlation.\n\n4.4 Transfer learning for action classi\ufb01cation\n\nTable 1 shows the results on transfer learning for action classi\ufb01cation on other datasets. As shown\nin the last two rows of Table 1, our debiased model consistently outperforms the baseline without\ndebiasing on all three datasets. The results validate that mitigating scene bias can improve generaliza-\ntion to the target action classi\ufb01cation datasets. As shown in the \ufb01rst block of Table 1, action-context\nfactorized C3D (referred to as Factor-C3D) [61] also improves the baseline C3D [51] on UCF-101.\nThe accuracy of Factor-C3D is on par with our debiased 3D-ResNet-18. Note that the model used\nin Factor-C3D is 2.4\u00d7 larger than ours. We also compare our method with RESOUND-C3D [38]\non the Diving48 dataset. All the videos in the Diving48 dataset share similar scenes. Our proposed\ndebiasing method shows a favorable result compared to [38]. We show a large relative improvement\n(14.1%) on the Diving48 dataset since it has a small scene representation bias of 1.26 [38].\n\n2Since there are no ground truth scene labels in the Mini-Kinetics-200 dataset, we use pseudo labels to\n\nmeasure the scene classi\ufb01cation accuracy.\n\n7\n\nDiving48UCF-101 split1UCF-101 split2UCF-101 split3HMDB-51 split1HMDB-51 split3HMDB-51 split2\u03c1 = -0.896, p-value=0.006\fTable 2: Transfer learning results on temporal action localization task: THUMOS-14 dataset. The\nevaluation metric is the video mAP at various IoU threshold values. The video representation trained\nusing the proposed debiasing techniques consistently improves the performance on the new task. In\nthe \ufb01rst block, we list the mAPs of the state-of-the-arts.\n\nMethod\nTAL-Net [9]\nSSN [70]\nR-C3D [65]\nCDC [45]\nR-C3D [57]\nR-C3D [57] + debiased (ours)\n\nInputs\n\nBackbone\n\nRGB+Flow\nRGB+Flow\n\nRGB\nRGB\nRGB\nRGB\n\nInceptionV3\n\nI3D\n\nC3D\nC3D\n\n3D-ResNet-18\n3D-ResNet-18\n\n0.1\n59.8\n66.0\n54.5\n\n-\n\n48.6\n50.2\n\n0.2\n57.1\n59.4\n51.5\n\n-\n\n48.6\n50.5\n\n0.3\n53.2\n51.9\n44.8\n40.1\n45.6\n47.9\n\nIoU threshold\n\n0.4\n48.5\n41.0\n35.6\n29.4\n40.8\n42.3\n\n0.5\n42.8\n29.8\n28.9\n23.3\n32.5\n33.4\n\n0.6\n33.8\n19.6\n\n-\n\n13.1\n25.5\n26.3\n\n0.7\n20.8\n10.7\n\n-\n7.9\n15.5\n16.8\n\navg.\n45.1\n39.8\n\n-\n-\n\n36.7\n38.2\n\nTable 3: Transfer learning results on spatio-temporal action detection. The evaluation metric is the\nframe mAP at the IoU threshold of 0.5. We list the frame mAP of the state-of-the-arts for reference.\n\nMethod\nACT [30]\nS3D-G [64]\nROAD [48]\nROAD [48] + debiased (ours)\n\nBackbone\n\nInputs\n\nPre-train\nImageNet\n\nVGG\n\nRGB+Flow\n\nInception (2D+1D) RGB+Flow ImageNet+FullKinetics\nImageNet+MiniKinetics\nImageNet+MiniKinetics\n\nVGG\nVGG\n\nRGB\nRGB\n\nJHMDB (all splits)\n\n65.7\n75.2\n32.5\n34.5\n\nRelative performance improvement vs. scene representation bias. Figure 4 illustrates the rela-\ntionship between relative performance improvement from the proposed debiasing method and the\nscene representation bias (de\ufb01ned in [38]). We measure the scene representation bias de\ufb01ned as\n(1) and the relative improvement of each split of the HMDB-51, UCF-101, Diving48 datasets. The\nPearson correlation is \u03c1 = \u22120.896 with a p-value 0.006, highlighting a strong negative correlation\nbetween the relative performance improvement and the scene representation bias. Our results show\nthat if a model is pre-trained with debiasing, the model generalizes better to the datasets with less\nscene bias, as the model pays attention to the actual action. In contrast, if a model is pre-trained on a\ndataset with a signi\ufb01cant scene bias e.g., Kinetics without any debiasing, the model would be biased\ntowards certain scene context. Such a model may still work well on target dataset with strong scene\nbiases (e.g., UCF-101), but does not generalize well to other less biased target datasets (e.g., Diving48\nand HMDB-51).\n4.5 Transfer learning for other activity understanding tasks.\n\nTemporal action localization Table 2 shows the results of temporal action localization on the\nTHUMOS-14 dataset. Using the pre-trained model with the proposed debiasing method consistently\noutperforms the baseline (pre-training without debiasing) on all the IoU threshold values. Our result\nsuggests that a model focusing on the actual discriminative cues from the actor(s) helps localize the\naction.\nSpatio-temporal action detection Table 3 shows spatio-temporal action detection results. With\ndebiasing, our model outperforms the baseline without debiasing on the JHMDB dataset. The\nresults in Table 2 and 3 validate that mitigating scene bias effectively improves the generalization of\npre-trained video representation to other activity understanding tasks.\n4.6 Ablation study\n\nWe conduct ablation studies to justify the design choices of the proposed debiasing technique. Here\nwe use the Mini-Kinetics-200 \u2192 HMDB-51 setting for the ablation study.\nEffect of the different pseudo scene labels. First, we study the effect of pseudo scene labels for\ndebiasing in Table 4. Compared to the model without using scene adversarial debiasing, using hard\npseudo labels improves transfer learning performance. Using soft pseudo labels for debiasing further\nenhances the performance. We attribute the performance improvement to many semantically similar\nscene categories in the Places365 dataset. Using soft scene labels alleviates the issues of committing\nto one particular scene class. In all the remaining experiments, we use pseudo scene soft labels for\nscene adversarial debiasing.\n\n8\n\n\fTable 4: Effect of debiasing using different\npseudo scene labels generated from the off-the-\nshelf classi\ufb01er.\n\nTable 5: Effect of using different losses for reduc-\ning scene bias. Both losses improve the perfor-\nmance in transfer learning.\n\nPseudo label used\n\nsplit-1\n\nHMDB-51\n\nsplit-2\n\nsplit-3\n\nNone (w/o debiasing)\nHard label\nSoft label (ours)\n\n52.9\n54.8\n56.4\n\n55.4\n54.2\n55.9\n\n52.6\n54.6\n56.4\n\navg.\n\n53.6\n54.5\n56.2\n\nLoss\n\nLAdv\n\u00d7\n\u00d7\n(cid:88)\n(cid:88)\n\nLEnt\n\u00d7\n(cid:88)\n\u00d7\n(cid:88)\n\nsplit-1\n\n52.9\n55.0\n56.4\n56.4\n\nHMDB-51\n\nsplit-2\n\nsplit-3\n\n55.4\n55.3\n55.9\n57.3\n\n52.6\n55.1\n56.4\n56.5\n\navg.\n\n53.6\n55.1\n56.2\n56.7\n\nbaseline\n\nours\n\nbaseline\n\nours\n\nbaseline\n\nours\n\nsword\n\nfencing\n\nsmoke\n\ndrink\n\nkick\n\nsword\n\nmixing\n\ncutting in kitchen high jump\n\nbalance beam\nFigure 5: Class activation maps (CAM) on the HMDB-51 (\ufb01rst row) and UCF-101 (second row)\ndatasets. The words underlined in blue are correct predictions, and those in red with no underline\nare incorrect predictions. The video representation using the proposed debiasing algorithm focuses\nmore on the direct visual cues (i.e., the main actors) rather than the surrounding scene contexts.\n\njavelin throw\n\nuneven bars\n\nEffect of the two different losses. Next, we ablate each of the two debiasing loss terms: scene\nadversarial loss (LAdv) and human mask confusion loss (LEnt) in Table 5. We observe that both losses\nimprove the performance individually. Using both debiasing losses gives the best results and suggests\nthat the two losses are regularizing the network in a complementary fashion.\n\n4.7 Class activation map visualization\n\nTo further demonstrate the ef\ufb01cacy of our debiasing algorithm, we show class activation maps\n(CAM) [72] from models with and without debiasing in Figure 5. We show the CAM overlayed over\nthe center frame (the eighth frame out of the sixteen frames) of each input clip. We present the results\non the HMDB-51 and UCF-101 datasets. We observe that without debiasing, a model predicts the\nincorrect classes because the model focuses on the scene instead of human actors. However, with\ndebiasing, a model focuses on human actions and predicts the correct action classes.\n\n5 Conclusions\n\nWe address the problem of learning video representation while mitigating scene bias. We augment\nthe standard cross-entropy loss for action classi\ufb01cation with two additional losses. The \ufb01rst one is\nan adversarial loss for scene class. The second one is an entropy loss for videos where humans are\nmasked out. Training with the two proposed losses encourages a network to focus on the actual\naction. We demonstrate the effectiveness of our method by transferring the pre-trained model to three\ntarget tasks.\nAs we build our model upon relatively weak baseline models, our model\u2019s \ufb01nal performance still\nfalls behind other state-of-the-art models. In this work, We only addressed one type of representation\nbias, i.e., scene bias. Extending the proposed debiasing method to mitigate other kinds of biases\ne.g., objects and persons for human action understanding is an interesting and important future work.\nAcknowledgment. This work was supported in part by NSF under Grant No. 1755785 and a Google\nFaculty Research Award. We thank the support of NVIDIA Corporation with the GPU donation.\n\n9\n\n\fReferences\n[1] Ehsan Adeli, Qingyu Zhao, Adolf Pfefferbaum, Edith V Sullivan, Li Fei-Fei, Juan Carlos Niebles, and\n\nKilian M Pohl. Bias-resilient neural network. arXiv preprint arXiv:1910.03676, 2019. 4\n\n[2] Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also\n\nsnowboard: Overcoming bias in captioning models. In ECCV, 2018. 3, 5\n\n[3] Loris Bazzani, Alessandra Bergamo, Dragomir Anguelov, and Lorenzo Torresani. Self-taught object\n\nlocalization with deep networks. 2016. 4\n\n[4] Harkirat S. Behl, Michael Sapienza, Gurkirt Singh, Suman Saha, Fabio Cuzzolin, and Philip H. S. Torr.\n\nIncremental tube construction for human action detection. In BMVC, 2018. 3\n\n[5] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to\n\ncomputer programmer as woman is to homemaker? debiasing word embeddings. In NeurIPS, 2016. 3\n\n[6] Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsuper-\n\nvised pixel-level domain adaptation with generative adversarial networks. In CVPR, 2017. 4\n\n[7] Peter Carbonetto, Nando De Freitas, and Kobus Barnard. A statistical model for general contextual object\n\nrecognition. In ECCV, 2004. 3\n\n[8] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset.\n\nIn CVPR, 2017. 1, 3\n\n[9] Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A. Ross, Jia Deng, and Rahul\nSukthankar. Rethinking the faster r-cnn architecture for temporal action localization. In CVPR, 2018. 8\n\n[10] Yun-Chun Chen, Yen-Yu Lin, Ming-Hsuan Yang, and Jia-Bin Huang. CrDoCo: Pixel-level domain transfer\n\nwith cross-domain consistency. In CVPR, 2019. 4\n\n[11] Myung Jin Choi, Joseph J Lim, Antonio Torralba, and Alan S Willsky. Exploiting hierarchical context on a\n\nlarge database of object categories. In CVPR, 2010. 3\n\n[12] Achal Dave, Olga Russakovsky, and Deva Ramanan. Predictivecorrective networks for action detection. In\n\nCVPR, 2017. 3\n\n[13] Santosh K Divvala, Derek Hoiem, James H Hays, Alexei A Efros, and Martial Hebert. An empirical study\n\nof context in object detection. In CVPR. IEEE, 2009. 3\n\n[14] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for\n\nvideo action recognition. In CVPR, 2016. 3\n\n[15] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML,\n\n2015. 5, 6\n\n[16] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Fran\u00e7ois Laviolette,\nMario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of\nMachine Learning Research, 17(1):2096\u20132030, 2016. 4\n\n[17] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Felix Wichmann, Wieland Brendel, and Matthias\nBethge. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and\nrobustness. In ICLR, 2019. 3\n\n[18] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le. Dropblock: A regularization method for convolutional\n\nnetworks. In NeurIPS, 2018. 4\n\n[19] Georgia Gkioxari and Jitendra Malik. Finding action tubes. In CVPR, 2015. 3\n\n[20] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d\n\ncnns and imagenet? In CVPR, 2018. 3, 6, 7\n\n[21] Jiawei He, Mostafa S Ibrahim, Zhiwei Deng, and Greg Mori. Generic tubelet proposals for action\n\nlocalization. 2018. 3\n\n[22] Kaiming He, Georgia Gkioxari, Piotr Doll\u00e1r, and Ross Girshick. Mask R-CNN. In ICCV, 2017. 5\n\n[23] Yun He, Soma Shirakabe, Yutaka Satoh, and Hirokatsu Kataoka. Human action recognition without human.\n\nIn ECCV Workshop, 2016. 3\n\n10\n\n\f[24] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and\n\nTrevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In ICML, 2017. 4\n\n[25] Judy Hoffman, Mehryar Mohri, and Ningshan Zhang. Algorithms and theory for multiple-source adaptation.\n\nIn NeurIPS, 2018. 4\n\n[26] Rui Hou, Chen Chen, and Mubarak Shah. Tube convolutional neural network (T-CNN) for action detection\n\nin videos. In ICCV, 2017. 3\n\n[27] Hueihan Jhuang, Juergen Gall, Silvia Zuf\ufb01, Cordelia Schmid, and Michael J Black. Towards understanding\n\naction recognition. In ICCV, 2013. 6\n\n[28] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action\n\nrecognition. TPAMI, 35(1):221\u2013231, 2013. 7\n\n[29] Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar. THUMOS\nchallenge: Action recognition with a large number of classes. http://crcv.ucf.edu/THUMOS14/,\n2014. 6\n\n[30] Vicky Kalogeiton, Philippe Weinzaepfel, Vittorio Ferrari, and Cordelia Schmid. Action tubelet detector for\n\nspatio-temporal action localization. In ICCV, 2017. 3, 8\n\n[31] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan,\nFabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv\npreprint arXiv:1705.06950, 2017. 1, 6\n\n[32] Aditya Khosla, Byoungkwon An An, Joseph J Lim, and Antonio Torralba. Looking beyond the visible\n\nscene. In CVPR, 2014. 3\n\n[33] Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn:\n\nTraining deep neural networks with biased data. In CVPR, 2019. 3\n\n[34] Hildegard Kuehne, Hueihan Jhuang, Est\u00edbaliz Garrote, Tomaso Poggio, and Thomas Serre. HMDB: A\n\nlarge video database for human motion recognition. In ICCV, 2011. 1, 6\n\n[35] Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. Beyond bags of features: Spatial pyramid matching\n\nfor recognizing natural scene categories. In CVPR, 2006. 3\n\n[36] Dong Li, Jia-Bin Huang, Yali Li, Shengjin Wang, and Ming-Hsuan Yang. Weakly supervised object\n\nlocalization with progressive domain adaptation. In CVPR, 2016. 4\n\n[37] Dong Li, Jia-Bin Huang, Yali Li, Shengjin Wang, and Ming-Hsuan Yang. Progressive representation\n\nadaptation for weakly supervised object localization. TPAMI, 2019. 4\n\n[38] Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representation\n\nbias. In ECCV, 2018. 1, 2, 3, 6, 7, 8\n\n[39] Aurelien Lucchi, Yunpeng Li, Xavier Boix, Kevin Smith, and Pascal Fua. Are spatial and global constraints\n\nreally necessary for segmentation? In ICCV, 2011. 3\n\n[40] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel\nUrtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild.\nIn CVPR, 2014. 3\n\n[41] Xiaojiang Peng and Cordelia Schmid. Multi-region two-stream r-cnn for action detection. In ECCV, 2016.\n\n3\n\n[42] Suman Saha, Gurkirt Singh, Michael Sapienza, Philip H.S. Torr, and Fabio Cuzzolin. Deep learning for\n\ndetecting multiple space-time action tubes in videos. In BMVC, 2016. 3\n\n[43] Suman Saha, Gurkirt Singh, and Fabio Cuzzolin. Amtnet: Action-micro-tube regression by end-to-end\n\ntrainable deep architecture. In ICCV, 2017. 3\n\n[44] Suman Saha, Gurkirt Singh, Michael Sapienza, Philip HS Torr, and Fabio Cuzzolin. Spatio-temporal\nhuman action localisation and instance segmentation in temporally untrimmed videos. arXiv preprint\narXiv:1707.07213, 2017. 3\n\n[45] Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih-Fu Chang. Cdc:\nConvolutional-de-convolutional networks for precise temporal action localization in untrimmed videos. In\nCVPR, 2017. 3, 8\n\n11\n\n\f[46] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in\n\nvideos. In NeurIPS, 2014. 3\n\n[47] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni-\n\ntion. In ICLR, 2015. 3, 6\n\n[48] Gurkirt Singh, Suman Saha, and Fabio Cuzzolin. Online real time multiple spatiotemporal action localisa-\n\ntion and prediction on a single platform. In ICCV, 2017. 3, 6, 8\n\n[49] Krishna Kumar Singh and Yong Jae Lee. Hide-and-seek: Forcing a network to be meticulous for weakly-\n\nsupervised object and action localization. In ICCV, 2017. 3, 4\n\n[50] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions\n\nclasses from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 1, 2, 6\n\n[51] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal\n\nfeatures with 3d convolutional networks. In ICCV, 2015. 7\n\n[52] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at\n\nspatiotemporal convolutions for action recognition. In CVPR, 2017. 1, 3\n\n[53] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation.\n\nIn CVPR, 2017. 4\n\n[54] Tuan-Hung Vu, Catherine Olsson, Ivan Laptev, Aude Oliva, and Josef Sivic. Predicting actions from static\n\nscenes. In ECCV, 2014. 3\n\n[55] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Val Gool. Temporal\n\nsegment networks: Towards good practices for deep action recognition. In ECCV, 2016. 3, 7\n\n[56] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal\n\nsegment networks for action recognition in videos. TPAMI, 2018. 3\n\n[57] Shiguang Wang\n\nand\n\nA faster\nhttps://github.com/sunnyxiaohu/R-C3D.pytorch.git, 2018. 6, 8\n\nJian Cheng.\n\npytorch\n\nimplementation\n\nof\n\nr-c3d.\n\n[58] Tianlu Wang, Jieyu Zhao, Kai-Wei Chang, Mark Yatskar, and Vicente Ordonez. Adversarial removal of\n\ngender from deep image representations. In ICCV, 2019. 3, 4\n\n[59] Xiaolong Wang, Abhinav Shrivastava, and Abhinav Gupta. A-fast-rcnn: Hard positive generation via\n\nadversary for object detection. In CVPR, 2017. 3, 4\n\n[60] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR,\n\n2018. 1, 3\n\n[61] Yang Wang and Minh Hoai. Pulling actions out of context: Explicit separation for effective combination.\n\nIn CVPR, 2018. 3, 7\n\n[62] Philippe Weinzaepfel, Zaid Harchaoui, and Cordelia Schmid. Learning to track for spatio-temporal action\n\nlocalization. In ICCV, 2015. 3\n\n[63] Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick.\n\nLong-term feature banks for detailed video understanding. In CVPR, 2019. 3\n\n[64] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal\n\nfeature learning for video understanding. In ECCV, 2018. 1, 2, 3, 6, 8\n\n[65] Huijuan Xu, Abir Das, and Kate Saenko. R-C3D: Region convolutional 3d network for temporal activity\n\ndetection. In ICCV, 2017. 3, 6, 8\n\n[66] Serena Yeung, Olga Russakovsky, Greg Mori, and Li Fei-Fei. End-to-end learning of action detection from\n\nframe glimpses in videos. In CVPR, 2016. 3\n\n[67] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.\n\n4\n\n[68] Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial\n\nlearning. In Proceedings of AAAI/ACM Conference on AI, Ethics, and Society, 2018. 4\n\n12\n\n\f[69] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing\n\nnetwork. In CVPR, 2017. 3\n\n[70] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action\n\ndetection with structured segment networks. In ICCV, 2017. 3, 8\n\n[71] Yue Zhao, Yuanjun Xiong, and Dahua Lin. Recognize actions by disentangling components of dynamics.\n\nIn CVPR, 2018. 3\n\n[72] B. Zhou, A. Khosla, Lapedriza. A., A. Oliva, and A. Torralba. Learning Deep Features for Discriminative\n\nLocalization. CVPR, 2016. 9\n\n[73] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million\n\nimage database for scene recognition. TPAMI, 2017. 2, 5\n\n[74] Mohammadreza Zolfaghari, Gabriel L. Oliveira, Nima Sedaghat, and Thomas Brox. Chained multi-stream\nnetworks exploiting pose, motion, and appearance for action classi\ufb01cation and detection. In ICCV, 2017. 3\n\n13\n\n\f", "award": [], "sourceid": 467, "authors": [{"given_name": "Jinwoo", "family_name": "Choi", "institution": "Virginia Tech"}, {"given_name": "Chen", "family_name": "Gao", "institution": "Virginia Tech"}, {"given_name": "Joseph C. E.", "family_name": "Messou", "institution": "Virginia Tech"}, {"given_name": "Jia-Bin", "family_name": "Huang", "institution": "Virginia Tech"}]}