直觉可能表明，运动和动态信息是基于视频的动作识别的关键。相比之下，有证据表明，最新的深入学习视频理解架构偏向单帧可用的静态信息。目前，缺少用于隔离视频中动态信息影响的方法和相应的数据集。他们的缺席使得很难理解当代体系结构如何利用动态和静态信息。我们以新颖的外观免费数据集（AFD）做出反应，以进行动作识别。 AFD缺乏与单个帧中的动作识别有关的静态信息。动力学的建模对于解决任务是必要的，因为仅通过考虑时间维度才能明显作用。我们评估了AFD上的11种当代行动识别体系结构及其相关的RGB视频。我们的结果表明，与RGB相比，AFD上所有体系结构的性能均显着下降。我们还对人类进行了免费研究，该研究表明他们在AFD和RGB上的识别准确性非常相似，并且比AFD评估的体系结构要好得多。我们的结果激发了一种新颖的体系结构，在当代设计中，在AFD和RGB上的最佳性能中恢复了光流的明确恢复。

Deep convolutional networks have achieved great success for visual recognition in still images. However, for action recognition in videos, the advantage over traditional methods is not so evident. This paper aims to discover the principles to design effective ConvNet architectures for action recognition in videos and learn these models given limited training samples. Our first contribution is temporal segment network (TSN), a novel framework for video-based action recognition. which is based on the idea of long-range temporal structure modeling. It combines a sparse temporal sampling strategy and video-level supervision to enable efficient and effective learning using the whole action video. The other contribution is our study on a series of good practices in learning ConvNets on video data with the help of temporal segment network. Our approach obtains the state-the-of-art performance on the datasets of HMDB51 (69.4%) and UCF101 (94.2%). We also visualize the learned ConvNet models, which qualitatively demonstrates the effectiveness of temporal segment network and the proposed good practices. 1

In this paper we discuss several forms of spatiotemporal convolutions for video analysis and study their effects on action recognition. Our motivation stems from the observation that 2D CNNs applied to individual frames of the video have remained solid performers in action recognition. In this work we empirically demonstrate the accuracy advantages of 3D CNNs over 2D CNNs within the framework of residual learning. Furthermore, we show that factorizing the 3D convolutional filters into separate spatial and temporal components yields significantly gains in accuracy. Our empirical study leads to the design of a new spatiotemporal convolutional block "R(2+1)D" which produces CNNs that achieve results comparable or superior to the state-of-theart on Sports-1M, Kinetics, UCF101, and HMDB51.

Recent applications of Convolutional Neural Networks (ConvNets) for human action recognition in videos have proposed different solutions for incorporating the appearance and motion information. We study a number of ways of fusing ConvNet towers both spatially and temporally in order to best take advantage of this spatio-temporal information. We make the following findings: (i) that rather than fusing at the softmax layer, a spatial and temporal network can be fused at a convolution layer without loss of performance, but with a substantial saving in parameters;(ii) that it is better to fuse such networks spatially at the last convolutional layer than earlier, and that additionally fusing at the class prediction layer can boost accuracy; finally (iii) that pooling of abstract convolutional features over spatiotemporal neighbourhoods further boosts performance. Based on these studies we propose a new ConvNet architecture for spatiotemporal fusion of video snippets, and evaluate its performance on standard benchmarks where this architecture achieves stateof-the-art results. Our code and models are available at http://www.robots.ox.ac.uk/ vgg/software/two stream action

事件摄像机是新型生物启发传感器，其异步捕获“事件”形式的像素级强度变化。由于它们的传感机制，事件相机几乎没有运动模糊，这是一个非常高的时间分辨率，并且需要比传统的基于帧的相机更小的电力和存储器。这些特性使它们成为一个完美的拟合若干现实世界应用，如在可穿戴设备上的专门动作识别，其中快速相机运动和有限的电力挑战传统视觉传感器。然而，迄今为止，基于事件的愿景的不断增长的愿景领域已经忽略了在此类应用中的活动摄像机的潜力。在本文中，我们表明事件数据是自我监测行动识别的非常有价值的模态。为此，我们介绍了N-EPIC-Kitchens，这是大型史诗厨房数据集的第一个基于事件的相机扩展。在此背景下，我们提出了两种策略：（i）使用传统的视频处理架构（E $ ^ 2 $（GO））和（ii）使用事件数据直接处理事件相机数据（E $ ^ 2 $（GO））和蒸馏光流信息（E $ ^ 2 $（go）mo）。在我们提出的基准测试中，我们表明事件数据为RGB和光流提供了可比性的性能，但在部署时没有任何额外的流量计算，以及相对于RGB的信息高达4％的性能。

We investigate architectures of discriminatively trained deep Convolutional Networks (ConvNets) for action recognition in video. The challenge is to capture the complementary information on appearance from still frames and motion between frames. We also aim to generalise the best performing hand-crafted features within a data-driven learning framework. Our contribution is three-fold. First, we propose a two-stream ConvNet architecture which incorporates spatial and temporal networks. Second, we demonstrate that a ConvNet trained on multi-frame dense optical flow is able to achieve very good performance in spite of limited training data. Finally, we show that multitask learning, applied to two different action classification datasets, can be used to increase the amount of training data and improve the performance on both. Our architecture is trained and evaluated on the standard video actions benchmarks of UCF-101 and HMDB-51, where it is competitive with the state of the art. It also exceeds by a large margin previous attempts to use deep nets for video classification.

The paucity of videos in current action classification datasets (UCF-101 and HMDB-51) has made it difficult to identify good video architectures, as most methods obtain similar performance on existing small-scale benchmarks. This paper re-evaluates state-of-the-art architectures in light of the new Kinetics Human Action Video dataset. Kinetics has two orders of magnitude more data, with 400 human action classes and over 400 clips per class, and is collected from realistic, challenging YouTube videos. We provide an analysis on how current architectures fare on the task of action classification on this dataset and how much performance improves on the smaller benchmark datasets after pre-training on Kinetics.We also introduce a new Two-Stream Inflated 3D Con-vNet (I3D) that is based on 2D ConvNet inflation: filters and pooling kernels of very deep image classification ConvNets are expanded into 3D, making it possible to learn seamless spatio-temporal feature extractors from video while leveraging successful ImageNet architecture designs and even their parameters. We show that, after pre-training on Kinetics, I3D models considerably improve upon the state-of-the-art in action classification, reaching 80.9% on HMDB-51 and 98.0% on UCF-101.

Spatiotemporal and motion features are two complementary and crucial information for video action recognition. Recent state-of-the-art methods adopt a 3D CNN stream to learn spatiotemporal features and another flow stream to learn motion features. In this work, we aim to efficiently encode these two features in a unified 2D framework. To this end, we first propose an STM block, which contains a Channel-wise SpatioTemporal Module (CSTM) to present the spatiotemporal features and a Channel-wise Motion Module (CMM) to efficiently encode motion features. We then replace original residual blocks in the ResNet architecture with STM blcoks to form a simple yet effective STM network by introducing very limited extra computation cost. Extensive experiments demonstrate that the proposed STM network outperforms the state-of-the-art methods on both temporal-related datasets (i.e., Something-Something v1 & v2 and Jester) and scene-related datasets (i.e., Kinetics-400, UCF-101, and HMDB-51) with the help of encoding spatiotemporal and motion features together. * The work was done during an internship at SenseTime.

Human activity recognition (HAR) using drone-mounted cameras has attracted considerable interest from the computer vision research community in recent years. A robust and efficient HAR system has a pivotal role in fields like video surveillance, crowd behavior analysis, sports analysis, and human-computer interaction. What makes it challenging are the complex poses, understanding different viewpoints, and the environmental scenarios where the action is taking place. To address such complexities, in this paper, we propose a novel Sparse Weighted Temporal Attention (SWTA) module to utilize sparsely sampled video frames for obtaining global weighted temporal attention. The proposed SWTA is comprised of two parts. First, temporal segment network that sparsely samples a given set of frames. Second, weighted temporal attention, which incorporates a fusion of attention maps derived from optical flow, with raw RGB images. This is followed by a basenet network, which comprises a convolutional neural network (CNN) module along with fully connected layers that provide us with activity recognition. The SWTA network can be used as a plug-in module to the existing deep CNN architectures, for optimizing them to learn temporal information by eliminating the need for a separate temporal stream. It has been evaluated on three publicly available benchmark datasets, namely Okutama, MOD20, and Drone-Action. The proposed model has received an accuracy of 72.76%, 92.56%, and 78.86% on the respective datasets thereby surpassing the previous state-of-the-art performances by a margin of 25.26%, 18.56%, and 2.94%, respectively.

本章旨在帮助开发网络 - 物理系统（CPS）在视频监控的各种应用中自动理解事件和活动。这些事件主要由无人机，中央电视台或新手和低端设备上的非熟板捕获。由于许多质量因素，这些视频是不受约束的，这些视频是非常挑战性的。我们在多年来提出了为解决问题的各种方法提供了广泛的陈述。这根据来自基于运动（SFM）的结构的方法，涉及涉及深神经网络的最近解决方案框架的方法。我们表明，长期运动模式在识别事件的任务中，单独发挥枢轴作用。因此，每个视频由使用基于图形的方法的固定数量的键帧显着表示。仅使用混合卷积神经网络（CNN）+经常性神经网络（RNN）架构利用时间特征。我们获得的结果是令人鼓舞的，因为它们优于标准的时间CNN，并且与使用空间信息以及运动提示的人员相提并论。进一步探索多际型号，我们构思了网络的空间和时间翼的多层融合策略。使用偏置的混合技术获得对视频和帧级别的各个预测载体的整合表示。与最先进的方法相比，融合策略在每个阶段的精度赋予我们更高的精度，因此在分类中实现了强大的共识。结果记录在动作识别域，即CCV，HMDB，UCF-101和KCV中广泛使用的四个基准数据集。可推动的是，专注于视频序列的更好分类肯定会导致强大的致动设计用于事件监视和对象暨活动跟踪的系统。

我们提出了MACLR，这是一种新颖的方法，可显式执行从视觉和运动方式中学习的跨模式自我监督的视频表示。与以前的视频表示学习方法相比，主要关注学习运动线索的研究方法是隐含的RGB输入，MACLR丰富了RGB视频片段的标准对比度学习目标，具有运动途径和视觉途径之间的跨模式学习目标。我们表明，使用我们的MACLR方法学到的表示形式更多地关注前景运动区域，因此可以更好地推广到下游任务。为了证明这一点，我们在五个数据集上评估了MACLR，以进行动作识别和动作检测，并在所有数据集上展示最先进的自我监督性能。此外，我们表明MACLR表示可以像在UCF101和HMDB51行动识别的全面监督下所学的表示一样有效，甚至超过了对Vidsitu和SSV2的行动识别的监督表示，以及对AVA的动作检测。

Temporal modeling is key for action recognition in videos. It normally considers both short-range motions and long-range aggregations. In this paper, we propose a Temporal Excitation and Aggregation (TEA) block, including a motion excitation (ME) module and a multiple temporal aggregation (MTA) module, specifically designed to capture both short-and long-range temporal evolution. In particular, for short-range motion modeling, the ME module calculates the feature-level temporal differences from spatiotemporal features. It then utilizes the differences to excite the motion-sensitive channels of the features. The long-range temporal aggregations in previous works are typically achieved by stacking a large number of local temporal convolutions. Each convolution processes a local temporal window at a time. In contrast, the MTA module proposes to deform the local convolution to a group of subconvolutions, forming a hierarchical residual architecture. Without introducing additional parameters, the features will be processed with a series of sub-convolutions, and each frame could complete multiple temporal aggregations with neighborhoods. The final equivalent receptive field of temporal dimension is accordingly enlarged, which is capable of modeling the long-range temporal relationship over distant frames. The two components of the TEA block are complementary in temporal modeling. Finally, our approach achieves impressive results at low FLOPs on several action recognition benchmarks, such as Kinetics, Something-Something, HMDB51, and UCF101, which confirms its effectiveness and efficiency.

人类相互作用的分析是人类运动分析的一个重要研究主题。它已经使用第一人称视觉（FPV）或第三人称视觉（TPV）进行了研究。但是，到目前为止，两种视野的联合学习几乎没有引起关注。原因之一是缺乏涵盖FPV和TPV的合适数据集。此外，FPV或TPV的现有基准数据集具有多个限制，包括样本数量有限，参与者，交互类别和模态。在这项工作中，我们贡献了一个大规模的人类交互数据集，即FT-HID数据集。 FT-HID包含第一人称和第三人称愿景的成对对齐的样本。该数据集是从109个不同受试者中收集的，并具有三种模式的90K样品。该数据集已通过使用几种现有的动作识别方法验证。此外，我们还引入了一种新型的骨骼序列的多视图交互机制，以及针对第一人称和第三人称视野的联合学习多流框架。两种方法都在FT-HID数据集上产生有希望的结果。可以预期，这一视力一致的大规模数据集的引入将促进FPV和TPV的发展，以及他们用于人类行动分析的联合学习技术。该数据集和代码可在\ href {https://github.com/endlichere/ft-hid} {here} {herefichub.com/endlichere.com/endlichere}中获得。

手术手术室（OR）为自动化和优化提供了许多机会。来自OR的各种来源的视频越来越多。医学界试图利用这些丰富的数据来开发自动化方法，以提高介入的护理，降低成本并改善整体患者的结果。因此，来自或房间摄像机的现有数据集的大小或方式限制了，因此尚不清楚哪些传感器方式最适合诸如识别视频外科手术的任务。这项研究表明，手术动作识别性能可能会根据所使用的图像方式而有所不同。我们对几种常用的传感器方式进行有条理的分析，并提出了两种改善分类性能的融合方法。这些分析是对18个腹腔镜程序的一组多视图RGB-D视频记录进行的。

Drone-camera based human activity recognition (HAR) has received significant attention from the computer vision research community in the past few years. A robust and efficient HAR system has a pivotal role in fields like video surveillance, crowd behavior analysis, sports analysis, and human-computer interaction. What makes it challenging are the complex poses, understanding different viewpoints, and the environmental scenarios where the action is taking place. To address such complexities, in this paper, we propose a novel Sparse Weighted Temporal Fusion (SWTF) module to utilize sparsely sampled video frames for obtaining global weighted temporal fusion outcome. The proposed SWTF is divided into two components. First, a temporal segment network that sparsely samples a given set of frames. Second, weighted temporal fusion, that incorporates a fusion of feature maps derived from optical flow, with raw RGB images. This is followed by base-network, which comprises a convolutional neural network module along with fully connected layers that provide us with activity recognition. The SWTF network can be used as a plug-in module to the existing deep CNN architectures, for optimizing them to learn temporal information by eliminating the need for a separate temporal stream. It has been evaluated on three publicly available benchmark datasets, namely Okutama, MOD20, and Drone-Action. The proposed model has received an accuracy of 72.76%, 92.56%, and 78.86% on the respective datasets thereby surpassing the previous state-of-the-art performances by a significant margin.

While large datasets have proven to be a key enabler for progress in computer vision, they can have biases that lead to erroneous conclusions. The notion of the representation bias of a dataset is proposed to combat this problem. It captures the fact that representations other than the ground-truth representation can achieve good performance on any given dataset. When this is the case, the dataset is said not to be well calibrated. Dataset calibration is shown to be a necessary condition for the standard state-of-the-art evaluation practice to converge to the ground-truth representation. A procedure, RESOUND, is proposed to quantify and minimize representation bias. Its application to the problem of action recognition shows that current datasets are biased towards static representations (objects, scenes and people). Two versions of RE-SOUND are studied. An Explicit RESOUND procedure is proposed to assemble new datasets by sampling existing datasets. An implicit RE-SOUND procedure is used to guide the creation of a new dataset, Div-ing48, of over 18,000 video clips of competitive diving actions, spanning 48 fine-grained dive classes. Experimental evaluation confirms the effectiveness of RESOUND to reduce the static biases of current datasets.

有效地对视频中的空间信息进行建模对于动作识别至关重要。为了实现这一目标，最先进的方法通常采用卷积操作员和密集的相互作用模块，例如非本地块。但是，这些方法无法准确地符合视频中的各种事件。一方面，采用的卷积是有固定尺度的，因此在各种尺度的事件中挣扎。另一方面，密集的相互作用建模范式仅在动作 - 欧元零件时实现次优性能，给最终预测带来了其他噪音。在本文中，我们提出了一个统一的动作识别框架，以通过引入以下设计来研究视频内容的动态性质。首先，在提取本地提示时，我们会生成动态尺度的时空内核，以适应各种事件。其次，为了将这些线索准确地汇总为全局视频表示形式，我们建议仅通过变压器在一些选定的前景对象之间进行交互，从而产生稀疏的范式。我们将提出的框架称为事件自适应网络（EAN），因为这两个关键设计都适应输入视频内容。为了利用本地细分市场内的短期运动，我们提出了一种新颖有效的潜在运动代码（LMC）模块，进一步改善了框架的性能。在几个大规模视频数据集上进行了广泛的实验，例如，某种东西，动力学和潜水48，验证了我们的模型是否在低拖鞋上实现了最先进或竞争性的表演。代码可在：https：//github.com/tianyuan168326/ean-pytorch中找到。

人类行动识别是计算机视觉中的重要应用领域。它的主要目的是准确地描述人类的行为及其相互作用，从传感器获得的先前看不见的数据序列中。识别，理解和预测复杂人类行动的能力能够构建许多重要的应用，例如智能监视系统，人力计算机界面，医疗保健，安全和军事应用。近年来，计算机视觉社区特别关注深度学习。本文使用深度学习技术的视频分析概述了当前的动作识别最新识别。我们提出了识别人类行为的最重要的深度学习模型，并分析它们，以提供用于解决人类行动识别问题的深度学习算法的当前进展，以突出其优势和缺点。基于文献中报道的识别精度的定量分析，我们的研究确定了动作识别中最新的深层体系结构，然后为该领域的未来工作提供当前的趋势和开放问题。

具有注释的缺乏大规模的真实数据集使转移学习视频活动的必要性。我们的目标是为少数行动分类开发几次拍摄转移学习的有效方法。我们利用独立培训的本地视觉提示来学习可以从源域传输的表示，该源域只能使用少数示例来从源域传送到不同的目标域。我们使用的视觉提示包括对象 - 对象交互，手掌和地区内的动作，这些地区是手工位置的函数。我们采用了一个基于元学习的框架，以提取部署的视觉提示的独特和域不变组件。这使得能够在使用不同的场景和动作配置捕获的公共数据集中传输动作分类模型。我们呈现了我们转让学习方法的比较结果，并报告了阶级阶级和数据间数据间际传输的最先进的行动分类方法。

The purpose of this study is to determine whether current video datasets have sufficient data for training very deep convolutional neural networks (CNNs) with spatio-temporal three-dimensional (3D) kernels. Recently, the performance levels of 3D CNNs in the field of action recognition have improved significantly. However, to date, conventional research has only explored relatively shallow 3D architectures. We examine the architectures of various 3D CNNs from relatively shallow to very deep ones on current video datasets. Based on the results of those experiments, the following conclusions could be obtained: (i) training resulted in significant overfitting for UCF-101, HMDB-51, and Ac-tivityNet but not for Kinetics. (ii) The Kinetics dataset has sufficient data for training of deep 3D CNNs, and enables training of up to 152 ResNets layers, interestingly similar to 2D ResNets on ImageNet. ResNeXt-101 achieved 78.4% average accuracy on the Kinetics test set. (iii) Kinetics pretrained simple 3D architectures outperforms complex 2D architectures, and the pretrained ResNeXt-101 achieved 94.5% and 70.2% on respectively. The use of 2D CNNs trained on ImageNet has produced significant progress in various tasks in image. We believe that using deep 3D CNNs together with Kinetics will retrace the successful history of 2D CNNs and ImageNet, and stimulate advances in computer vision for videos. The codes and pretrained models used in this study are publicly available1.