我们介绍了遮阳板，一个新的像素注释的新数据集和一个基准套件，用于在以自我为中心的视频中分割手和活动对象。遮阳板注释Epic-kitchens的视频，其中带有当前视频分割数据集中未遇到的新挑战。具体而言，我们需要确保像素级注释作为对象经历变革性相互作用的短期和长期一致性，例如洋葱被剥皮，切成丁和煮熟 - 我们旨在获得果皮，洋葱块，斩波板，刀，锅以及表演手的准确像素级注释。遮阳板引入了一条注释管道，以零件为ai驱动，以进行可伸缩性和质量。总共，我们公开发布257个对象类的272K手册语义面具，990万个插值密集口罩，67K手动关系，涵盖36小时的179个未修剪视频。除了注释外，我们还引入了视频对象细分，互动理解和长期推理方面的三个挑战。有关数据，代码和排行榜：http：//epic-kitchens.github.io/visor

The appearance of an object can be fleeting when it transforms. As eggs are broken or paper is torn, their color, shape and texture can change dramatically, preserving virtually nothing of the original except for the identity itself. Yet, this important phenomenon is largely absent from existing video object segmentation (VOS) benchmarks. In this work, we close the gap by collecting a new dataset for Video Object Segmentation under Transformations (VOST). It consists of more than 700 high-resolution videos, captured in diverse environments, which are 20 seconds long on average and densely labeled with instance masks. A careful, multi-step approach is adopted to ensure that these videos focus on complex object transformations, capturing their full temporal extent. We then extensively evaluate state-of-the-art VOS methods and make a number of important discoveries. In particular, we show that existing methods struggle when applied to this novel task and that their main limitation lies in over-reliance on static appearance cues. This motivates us to propose a few modifications for the top-performing baseline that improve its capabilities by better modeling spatio-temporal information. But more broadly, the hope is to stimulate discussion on learning more robust video object representations.

First-person vision is gaining interest as it offers a unique viewpoint on people's interaction with objects, their attention, and even intention. However, progress in this challenging domain has been relatively slow due to the lack of sufficiently large datasets. In this paper, we introduce EPIC-KITCHENS, a large-scale egocentric video benchmark recorded by 32 participants in their native kitchen environments. Our videos depict non-scripted daily activities: we simply asked each participant to start recording every time they entered their kitchen. Recording took place in 4 cities (in North America and Europe) by participants belonging to 10 different nationalities, resulting in highly diverse cooking styles. Our dataset features 55 hours of video consisting of 11.5M frames, which we densely labelled for a total of 39.6K action segments and 454.3K object bounding boxes. Our annotation is unique in that we had the participants narrate their own videos (after recording), thus reflecting true intention, and we crowd-sourced ground-truths based on these. We describe our object, action and anticipation challenges, and evaluate several baselines over two test splits, seen and unseen kitchens.

为了使AI安全地在医院，学校和工作场所等现实世界中安全部署，它必须能够坚定地理解物理世界。这种推理的基础是物理常识：了解可用对象的物理特性和提供的能力，如何被操纵以及它们如何与其他对象进行交互。物理常识性推理从根本上是一项多感官任务，因为物理特性是通过多种模式表现出来的，其中两个是视觉和声学。我们的论文通过贡献PACS来朝着现实世界中的物理常识推理：第一个用于物理常识属性注释的视听基准。 PACS包含13,400对答案对，涉及1,377个独特的物理常识性问题和1,526个视频。我们的数据集提供了新的机会来通过将音频作为此多模式问题的核心组成部分来推进物理推理的研究领域。使用PACS，我们在我们的新挑战性任务上评估了多种最先进的模型。尽管某些模型显示出令人鼓舞的结果（精度为70％），但它们都没有人类的绩效（精度为95％）。我们通过证明多模式推理的重要性并为未来的研究提供了可能的途径来结束本文。

The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the chal-

Progress on object detection is enabled by datasets that focus the research community's attention on open challenges. This process led us from simple images to complex scenes and from bounding boxes to segmentation masks. In this work, we introduce LVIS (pronounced 'el-vis'): a new dataset for Large Vocabulary Instance Segmentation. We plan to collect ∼2 million high-quality instance segmentation masks for over 1000 entry-level object categories in 164k images. Due to the Zipfian distribution of categories in natural images, LVIS naturally has a long tail of categories with few training samples. Given that state-of-the-art deep learning methods for object detection perform poorly in the low-sample regime, we believe that our dataset poses an important and exciting new scientific challenge. LVIS is available at http://www.lvisdataset.org.

为视频中的每个像素分配语义类和跟踪身份的任务称为视频Panoptic分段。我们的工作是第一个在真实世界中瞄准这项任务，需要在空间和时间域中的密集解释。由于此任务的地面真理难以获得，但是，现有数据集是合成构造的或仅在短视频剪辑中稀疏地注释。为了克服这一点，我们介绍了一个包含两个数据集，Kitti-Step和Motchallenge步骤的新基准。数据集包含长视频序列，提供具有挑战性的示例和用于研究长期像素精确分割和在真实条件下跟踪的测试床。我们进一步提出了一种新的评估度量分割和跟踪质量（STQ），其相当余额平衡该任务的语义和跟踪方面，并且更适合评估任意长度的序列。最后，我们提供了几个基线来评估此新具有挑战性数据集的现有方法的状态。我们已将我们的数据集，公制，基准服务器和基准公开提供，并希望这将激发未来的研究。

Generic motion understanding from video involves not only tracking objects, but also perceiving how their surfaces deform and move. This information is useful to make inferences about 3D shape, physical properties and object interactions. While the problem of tracking arbitrary physical points on surfaces over longer video clips has received some attention, no dataset or benchmark for evaluation existed, until now. In this paper, we first formalize the problem, naming it tracking any point (TAP). We introduce a companion benchmark, TAP-Vid, which is composed of both real-world videos with accurate human annotations of point tracks, and synthetic videos with perfect ground-truth point tracks. Central to the construction of our benchmark is a novel semi-automatic crowdsourced pipeline which uses optical flow estimates to compensate for easier, short-term motion like camera shake, allowing annotators to focus on harder sections of video. We validate our pipeline on synthetic data and propose a simple end-to-end point tracking model TAP-Net, showing that it outperforms all prior methods on our benchmark when trained on synthetic data.

多代理行为建模旨在了解代理之间发生的交互。我们从行为神经科学，Caltech鼠标社交交互（CALMS21）数据集中提供了一个多代理数据集。我们的数据集由社交交互的轨迹数据组成，从标准居民入侵者测定中自由行为小鼠的视频记录。为了帮助加速行为研究，CALMS21数据集提供基准，以评估三种设置中自动行为分类方法的性能：（1）用于培训由单个注释器的所有注释，（2）用于风格转移以进行学习互动在特定有限培训数据的新行为学习的行为定义和（3）的注释差异。 DataSet由600万个未标记的追踪姿势的交互小鼠组成，以及超过100万帧，具有跟踪的姿势和相应的帧级行为注释。我们的数据集的挑战是能够使用标记和未标记的跟踪数据准确地对行为进行分类，以及能够概括新设置。

Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in

Semantic understanding of visual scenes is one of the holy grails of computer vision. Despite efforts of the community in data collection, there are still few image datasets covering a wide range of scenes and object categories with pixel-wise annotations for scene understanding. In this work, we present a densely annotated dataset ADE20K, which spans diverse annotations of scenes, objects, parts of objects, and in some cases even parts of parts. Totally there are 25k images of the complex everyday scenes containing a variety of objects in their natural spatial context. On average there are 19.5 instances and 10.5 object classes per image. Based on ADE20K, we construct benchmarks for scene parsing and instance segmentation. We provide baseline performances on both of the benchmarks and re-implement the state-ofthe-art models for open source. We further evaluate the effect of synchronized batch normalization and find that a reasonably large batch size is crucial for the semantic segmentation performance. We show that the networks trained on ADE20K are able to segment a wide variety of scenes and objects 1 .

可穿戴摄像机可以从用户的角度获取图像和视频。可以处理这些数据以了解人类的行为。尽管人类的行为分析已在第三人称视野中进行了彻底的研究，但仍在以自我为中心的环境中，尤其是在工业场景中进行了研究。为了鼓励在该领域的研究，我们介绍了Meccano，这是一个以自我为中心视频的多式模式数据集来研究类似工业的环境中的人类行为理解。多模式的特征是凝视信号，深度图和RGB视频同时使用自定义耳机获得。该数据集已在从第一人称视角的人类行为理解的背景下明确标记为基本任务，例如识别和预测人类对象的相互作用。使用MECCANO数据集，我们探索了五个不同的任务，包括1）动作识别，2）活动对象检测和识别，3）以自我为中心的人类对象互动检测，4）动作预期和5）下一步活动对象检测。我们提出了一个旨在研究人类行为的基准，该基准在被考虑的类似工业的情况下，表明所研究的任务和所考虑的方案对于最先进的算法具有挑战性。为了支持该领域的研究，我们在https://iplab.dmi.unict.it/meccano/上公开发布数据集。

由于几个因素之间的微妙权衡：参与者的隐私，生态有效性，数据保真度和后勤开销，记录野外未脚本人类互动的动态是具有挑战性的。为了解决这些问题，在社区精神上为社区的“数据集”之后，我们提出了会议生活实验室（Conflab）：一个新的概念，用于多模式多模式数据收集，野生野外社交对话。对于此处描述的Conflab的首次实例化，我们在一次大型国际会议上组织了现实生活中的专业网络活动。该数据集涉及48个会议参与者，捕捉了地位，熟人和网络动机的各种组合。我们的捕获设置改善了先前野外数据集的数据保真度，同时保留隐私敏感性：从非侵入性的架空视图中获得8个视频（1920x1080，60 fps），并具有定制的可穿戴传感器，并带有车载记录（完整9） - 轴IMU），具有隐私性的低频音频（1250 Hz）和基于蓝牙的接近度。此外，我们开发了用于采集时分布式硬件同步的自定义解决方案，并以高采样速率对身体关键点和动作进行了及时的连续注释。我们的基准测试展示了与野外隐私保护社交数据分析有关的一些开放研究任务：从高架摄像头视图，基于骨架的No-Audio扬声器检测和F-Formation检测中的关键点检测。

对人类对象相互作用的理解在第一人称愿景（FPV）中至关重要。遵循相机佩戴者操纵的对象的视觉跟踪算法可以提供有效的信息，以有效地建模此类相互作用。在过去的几年中，计算机视觉社区已大大提高了各种目标对象和场景的跟踪算法的性能。尽管以前有几次尝试在FPV域中利用跟踪器，但仍缺少对最先进跟踪器的性能的有条理分析。这项研究差距提出了一个问题，即应使用当前的解决方案``现成''还是应进行更多特定领域的研究。本文旨在为此类问题提供答案。我们介绍了FPV中单个对象跟踪的首次系统研究。我们的研究广泛分析了42个算法的性能，包括通用对象跟踪器和基线FPV特定跟踪器。分析是通过关注FPV设置的不同方面，引入新的绩效指标以及与FPV特定任务有关的。这项研究是通过引入Trek-150（由150个密集注释的视频序列组成的新型基准数据集）来实现的。我们的结果表明，FPV中的对象跟踪对当前的视觉跟踪器构成了新的挑战。我们强调了导致这种行为的因素，并指出了可能的研究方向。尽管遇到了困难，但我们证明了跟踪器为需要短期对象跟踪的FPV下游任务带来好处。我们预计，随着新的和FPV特定的方法学会得到研究，通用对象跟踪将在FPV中受欢迎。

互动对象理解，或者我们可以对对象做些什么以及计算机愿景的长期目标。在本文中，我们通过观察野外的自我高端视频的人类手来解决这个问题。我们展示了观察人类的手与之交互以及如何提供相关数据和必要的监督。参加双手，容易定位并稳定积极的物体以进行学习，并揭示发生与对象的交互的地方。分析手显示我们可以对物体做些什么以及如何做些。我们在史诗厨房数据集上应用这些基本原则，并成功地学习了国家敏感的特征，以及互动区域和提供了麦克拉斯的地区），纯粹是通过观察在EGoCentric视频中的手。

多模型对现实世界应用的承诺激发了可视化和理解其内部力学的研究，其最终目标是使利益相关者能够可视化模型行为，执行模型调试并促进对机器学习模型的信任。但是，现代的多模型模型通常是黑盒神经网络，这使得了解其内部力学变得具有挑战性。我们如何能在这些模型中可视化多模式相互作用的内部建模？我们的论文旨在通过提出Multiviz来填补这一空白，这是一种通过将可解释性问题分为4个阶段来分析多模型模型行为的方法：（1）单峰的重要性：每种模式如何有助于下游建模和预测，（2）交叉交叉。 - 模式相互作用：不同模态如何相互关系，（3）多模式表示：如何在决策级特征中表示单峰和跨模式的交互作用，以及（4）多模式预测：决策级特征如何组成以制造一个预言。 Multiviz旨在在不同的模式，模型，任务和研究领域进行操作。通过对6个现实世界任务的8个训练模型的实验，我们表明，Multiviz中的互补阶段共同使用户能够（1）模拟模型预测，（2）将可解释的概念分配给功能，（3）对模型错误分析执行错误分析，（4）使用错误分析到调试模型的见解。 Multiviz公开可用，将定期使用新的解释工具和指标进行更新，并欢迎社区的意见。

我们介绍了一些源自摄影师的本地化数据集，他们实际上试图了解他们拍摄的图像中的视觉内容。它包括有4,500多个视觉障碍者拍摄的超过4,500张图像中的100个类别的近10,000个细分。与现有的少数弹射对象检测和实例分段数据集相比，我们的数据集是第一个在对象中找到孔（例如，在我们的分段的12.3 \％中找到），它显示的对象相对于占据相对于尺寸的范围较大。图像和文本在我们的对象中的常见五倍以上（例如，在我们的分割的22.4％中找到）。对三种现代少量定位算法的分析表明，它们概括为我们的新数据集。这些算法通常很难找到带有孔，非常小且非常大的物体以及缺乏文本的物体的对象。为了鼓励更大的社区致力于这些尚未解决的挑战，我们在https://vizwiz.org上公开分享了带注释的少数数据集。

人类具有出色的能力来推理绑架并假设超出图像的字面内容的内容。通过识别散布在整个场景中的具体视觉线索，我们几乎不禁根据我们的日常经验和对世界的知识来提出可能的推论。例如，如果我们在道路旁边看到一个“ 20英里 /小时”的标志，我们可能会假设街道位于居民区（而不是在高速公路上），即使没有房屋。机器可以执行类似的视觉推理吗？我们提出了Sherlock，这是一个带注释的103K图像的语料库，用于测试机器能力，以超出字面图像内容的绑架推理。我们采用免费观看范式：参与者首先观察并识别图像中的显着线索（例如，对象，动作），然后给定线索，然后提供有关场景的合理推论。我们总共收集了363K（线索，推理）对，该对形成了首个绑架的视觉推理数据集。使用我们的语料库，我们测试了三个互补的绑架推理轴。我们评估模型的能力：i）从大型候选人语料库中检索相关推论； ii）通过边界框来定位推论的证据，iii）比较合理的推论，以匹配人类在新收集的19k李克特级判断的诊断语料库上的判断。尽管我们发现具有多任务目标的微调夹RN50x64优于强大的基准，但模型性能与人类一致之间存在着重要的净空。可在http://visualabduction.com/上获得数据，模型和排行榜

This paper introduces a video dataset of spatiotemporally localized Atomic Visual Actions (AVA). The AVA dataset densely annotates 80 atomic visual actions in 430 15-minute video clips, where actions are localized in space and time, resulting in 1.58M action labels with multiple labels per person occurring frequently. The key characteristics of our dataset are: (1) the definition of atomic visual actions, rather than composite actions; (2) precise spatio-temporal annotations with possibly multiple annotations for each person; (3) exhaustive annotation of these atomic actions over 15-minute video clips; (4) people temporally linked across consecutive segments; and (5) using movies to gather a varied set of action representations. This departs from existing datasets for spatio-temporal action recognition, which typically provide sparse annotations for composite actions in short video clips.AVA, with its realistic scene and action complexity, exposes the intrinsic difficulty of action recognition. To benchmark this, we present a novel approach for action localization that builds upon the current state-of-the-art methods, and demonstrates better performance on JHMDB and UCF101-24 categories. While setting a new state of the art on existing datasets, the overall results on AVA are low at 15.6% mAP, underscoring the need for developing new approaches for video understanding.

We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.