Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks. However, due to data limitations, there has been much less work on video-based QA. In this paper, we present TVQA, a largescale video QA dataset based on 6 popular TV shows. TVQA consists of 152,545 QA pairs from 21,793 clips, spanning over 460 hours of video. Questions are designed to be compositional in nature, requiring systems to jointly localize relevant moments within a clip, comprehend subtitle-based dialogue, and recognize relevant visual concepts. We provide analyses of this new dataset as well as several baselines and a multi-stream end-to-end trainable neural network framework for the TVQA task. The dataset is publicly available at http://tvqa.cs.unc.edu.

它仍然是一个管道梦想，电话和AR眼镜的AI助手可以帮助我们的日常生活来解决我们的问题，如“如何调整这款手表日期？”和“如何设置加热持续时间？（指向烤箱的同时）”。传统任务中使用的查询（即视频问题应答，视频检索，时刻定位）通常是有关的，并基于纯文本。相比之下，我们提出了一项名为Cometdancy的问题驱动视频段检索（AQVSR）的新任务。我们每个问题都是一个图像框文本查询，专注于我们日常生活中的物品，并期望从教学视频转录程序段的语料库中检索相关的答案段。为了支持对此AQVSR任务的研究，我们构建一个名为AssionSR的新数据集。我们设计新颖的准则来创造高质量样本。此数据集包含有关1K视频片段的1.4K多模态问题，来自各种日用物品的教学视频。为了解决AQVSR，我们开发了一个称为双重多模式编码器（DME）的简单但有效的模型，显着优于几种基线方法，同时仍然有大型未来改善空间。此外，我们提供了详细的消融分析。我们的代码和数据可以在https://github.com/stanlei52/aqvsr中获得。

Visual understanding goes well beyond object recognition. With one glance at an image, we can effortlessly imagine the world beyond the pixels: for instance, we can infer people's actions, goals, and mental states. While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world. We formalize this task as Visual Commonsense Reasoning. Given a challenging question about an image, a machine must answer correctly and then provide a rationale justifying its answer.Next, we introduce a new dataset, VCR, consisting of 290k multiple choice QA problems derived from 110k movie scenes. The key recipe for generating non-trivial and highquality problems at scale is Adversarial Matching, a new approach to transform rich annotations into multiple choice questions with minimal bias. Experimental results show that while humans find VCR easy (over 90% accuracy), state-of-the-art vision models struggle (∼45%).To move towards cognition-level understanding, we present a new reasoning engine, Recognition to Cognition Networks (R2C), that models the necessary layered inferences for grounding, contextualization, and reasoning. R2C helps narrow the gap between humans and machines (∼65%); still, the challenge is far from solved, and we provide analysis that suggests avenues for future work.

Video Question Answering methods focus on commonsense reasoning and visual cognition of objects or persons and their interactions over time. Current VideoQA approaches ignore the textual information present in the video. Instead, we argue that textual information is complementary to the action and provides essential contextualisation cues to the reasoning process. To this end, we propose a novel VideoQA task that requires reading and understanding the text in the video. To explore this direction, we focus on news videos and require QA systems to comprehend and answer questions about the topics presented by combining visual and textual cues in the video. We introduce the ``NewsVideoQA'' dataset that comprises more than $8,600$ QA pairs on $3,000+$ news videos obtained from diverse news channels from around the world. We demonstrate the limitations of current Scene Text VQA and VideoQA methods and propose ways to incorporate scene text information into VideoQA methods.

Figure 1: Our MovieQA dataset contains 14,944 questions about 408 movies. It contains multiple sources of information: plots, subtitles, video clips, scripts, and DVS transcriptions. In this figure we show example QAs from The Matrix and localize them in the timeline.

We have seen great progress in basic perceptual tasks such as object recognition and detection. However, AI models still fail to match humans in high-level vision tasks due to the lack of capacities for deeper reasoning. Recently the new task of visual question answering (QA) has been proposed to evaluate a model's capacity for deep image understanding. Previous works have established a loose, global association between QA sentences and images. However, many questions and answers, in practice, relate to local regions in the images. We establish a semantic link between textual descriptions and image regions by object-level grounding. It enables a new type of QA with visual answers, in addition to textual answers used in previous work. We study the visual QA tasks in a grounded setting with a large collection of 7W multiple-choice QA pairs. Furthermore, we evaluate human performance and several baseline models on the QA tasks. Finally, we propose a novel LSTM model with spatial attention to tackle the 7W QA tasks.

AR眼镜/机器人等智能助手的长期目标是帮助用户以负担得起的现实世界情景，例如“我如何运行微波炉1分钟？”。但是，仍然没有明确的任务定义和合适的基准。在本文中，我们定义了一项名为“负担中心问题驱动的任务完成”的新任务，AI助手应从教学视频和脚本中学习，以指导用户逐步指导用户。为了支持该任务，我们构建了AssistQ，这是一个新的数据集，其中包括531个问答样本，该样本来自100个新电影的第一人称视频。每个问题都应通过从视觉细节（例如按钮的位置）和纹理细节（例如，按/转弯之类的操作）推断出多步导完成。为了解决这一独特的任务，我们开发了一个问题对行为（Q2A）模型，该模型极大地超过了几种基线方法，同时仍然有大量改进的空间。我们希望我们的任务和数据集能够推进Egentric AI助手的发展。我们的项目页面可在以下网址找到：https：//showlab.github.io/assistq

Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in

We propose the task of free-form and open-ended Visual Question Answering (VQA). Given an image and a natural language question about the image, the task is to provide an accurate natural language answer. Mirroring real-world scenarios, such as helping the visually impaired, both the questions and answers are open-ended. Visual questions selectively target different areas of an image, including background details and underlying context. As a result, a system that succeeds at VQA typically needs a more detailed understanding of the image and complex reasoning than a system producing generic image captions. Moreover, VQA is amenable to automatic evaluation, since many open-ended answers contain only a few words or a closed set of answers that can be provided in a multiple-choice format. We provide a dataset containing ∼0.25M images, ∼0.76M questions, and ∼10M answers (www.visualqa.org), and discuss the information it provides. Numerous baselines and methods for VQA are provided and compared with human performance. Our VQA demo is available on CloudCV (http://cloudcv.org/vqa).

问题回答（QA）是最重要的自然语言处理（NLP）任务之一。它旨在使用NLP技术根据大规模的非结构化语料库生成对给定问题的相应答案。随着深度学习的发展，正在提出越来越具有挑战性的质量检查数据集，并且许多用于解决它们的新方法也正在出现。在本文中，我们研究了在深度学习时代发布的有影响力的质量检查数据集。具体来说，我们首先引入两个最常见的质量检查任务 - 文本问题答案和视觉问题 - 分别涵盖最具代表性的数据集，然后给出质量检查研究的一些当前挑战。

现有的视频理解数据集主要集中在人类的互动上，几乎没有关注“在野外”设置，在户外录制了视频。我们提出了Wildqa，这是一个视频理解外部设置中录制的视频的数据集。除了视频问答（视频质量质量检查）外，我们还介绍了确定给定问答（视频证据选择）视觉支持的新任务。通过使用各种基线模型的评估，我们表明Wildqa对愿景和语言研究社区构成了新的挑战。该数据集可在https://lit.eecs.umich.edu/wildqa/上找到。

Artificial Intelligence (AI) and its applications have sparked extraordinary interest in recent years. This achievement can be ascribed in part to advances in AI subfields including Machine Learning (ML), Computer Vision (CV), and Natural Language Processing (NLP). Deep learning, a sub-field of machine learning that employs artificial neural network concepts, has enabled the most rapid growth in these domains. The integration of vision and language has sparked a lot of attention as a result of this. The tasks have been created in such a way that they properly exemplify the concepts of deep learning. In this review paper, we provide a thorough and an extensive review of the state of the arts approaches, key models design principles and discuss existing datasets, methods, their problem formulation and evaluation measures for VQA and Visual reasoning tasks to understand vision and language representation learning. We also present some potential future paths in this field of research, with the hope that our study may generate new ideas and novel approaches to handle existing difficulties and develop new applications.

The little girl jumps back up after falling. Figure 1: We consider localizing moments in video with natural language and demonstrate that incorporating local and global video features is important for this task. To train and evaluate our model, we collect the Distinct Describable Moments (DiDeMo) dataset which consists of over 40,000 pairs of localized video moments and corresponding natural language.

We present an approach named JSFusion (Joint Sequence Fusion) that can measure semantic similarity between any pairs of multimodal sequence data (e.g. a video clip and a language sentence). Our multimodal matching network consists of two key components. First, the Joint Semantic Tensor composes a dense pairwise representation of two sequence data into a 3D tensor. Then, the Convolutional Hierarchical Decoder computes their similarity score by discovering hidden hierarchical matches between the two sequence modalities. Both modules leverage hierarchical attention mechanisms that learn to promote well-matched representation patterns while prune out misaligned ones in a bottom-up manner. Although the JSFusion is a universal model to be applicable to any multimodal sequence data, this work focuses on video-language tasks including multimodal retrieval and video QA. We evaluate the JS-Fusion model in three retrieval and VQA tasks in LSMDC, for which our model achieves the best performance reported so far. We also perform multiple-choice and movie retrieval tasks for the MSR-VTT dataset, on which our approach outperforms many state-of-the-art methods.

We introduce a new dataset for joint reasoning about natural language and images, with a focus on semantic diversity, compositionality, and visual reasoning challenges. The data contains 107,292 examples of English sentences paired with web photographs. The task is to determine whether a natural language caption is true about a pair of photographs. We crowdsource the data using sets of visually rich images and a compare-and-contrast task to elicit linguistically diverse language. Qualitative analysis shows the data requires compositional joint reasoning, including about quantities, comparisons, and relations. Evaluation using state-of-the-art visual reasoning methods shows the data presents a strong challenge. * Contributed equally. † Work done as an undergraduate at Cornell University. 1 In parts of this paper, we use the term compositional differently than it is commonly used in linguistics to refer to reasoning that requires composition. This type of reasoning often manifests itself in highly compositional language.2 Appendix G contains license information for all photographs used in this paper. 3 The top example is True, while the bottom is False.

我们提出了一项新的3D问题答案的3D空间理解任务（3D-QA）。在3D-QA任务中，模型从丰富的RGB-D室内扫描的整个3D场景接收视觉信息，并回答关于3D场景的给定文本问题。与VQA的2D答案不同，传统的2D-QA模型遭受了对对象对齐和方向的空间理解的问题，并且从3D-QA中的文本问题中失败了对象本地化。我们为3D-QA提出了一个名为ScanQA模型的3D-QA基线模型，其中模型从3D对象提案和编码的句子嵌入中获取融合描述符。该学习描述符将语言表达式与3D扫描的底层几何特征相关联，并促进3D边界框的回归以确定文本问题中的描述对象。我们收集了人类编辑的问题答案对，自由表格答案将接地为3D场景中的3D对象。我们的新ScanQA数据集包含来自Scannet DataSet的800个室内场景的超过41K问答对。据我们所知，ScanQA是第一个在3D环境中执行对象接地的问答的大规模工作。

近年来，统一的视觉语言框架已经大大提高，其中大多数采用编码器架构将图像文本任务统一为序列到序列的生成。但是，现有的视频语言（VIDL）模型仍需要在每个任务的模型体系结构和培训目标中进行特定于任务的设计。在这项工作中，我们探索了一个统一的VIDL框架薰衣草，其中蒙版语言建模（MLM）用作所有前训练和下游任务的常见接口。这样的统一导致了简化的模型体系结构，在多模式编码器之上，只需要一个轻巧的MLM头，而不是具有更多参数的解码器。令人惊讶的是，实验结果表明，这个统一的框架在14个VIDL基准测试中实现了竞争性能，涵盖了视频问答，文本到视频检索和视频字幕。广泛的分析进一步证明了薰衣草比现有VIDL方法的优势：（i）在多任务列出时仅使用一组参数值支持所有下游任务； （ii）对各种下游任务的几乎没有概括； （iii）在视频问题回答任务上启用零射门评估。代码可从https://github.com/microsoft/lavender获得。

为了使AI安全地在医院，学校和工作场所等现实世界中安全部署，它必须能够坚定地理解物理世界。这种推理的基础是物理常识：了解可用对象的物理特性和提供的能力，如何被操纵以及它们如何与其他对象进行交互。物理常识性推理从根本上是一项多感官任务，因为物理特性是通过多种模式表现出来的，其中两个是视觉和声学。我们的论文通过贡献PACS来朝着现实世界中的物理常识推理：第一个用于物理常识属性注释的视听基准。 PACS包含13,400对答案对，涉及1,377个独特的物理常识性问题和1,526个视频。我们的数据集提供了新的机会来通过将音频作为此多模式问题的核心组成部分来推进物理推理的研究领域。使用PACS，我们在我们的新挑战性任务上评估了多种最先进的模型。尽管某些模型显示出令人鼓舞的结果（精度为70％），但它们都没有人类的绩效（精度为95％）。我们通过证明多模式推理的重要性并为未来的研究提供了可能的途径来结束本文。

基于知识的视觉问题的问题涉及除了图像内容之外还涉及需要外部知识的问题。这些知识通常有各种形式，包括视觉，文本和致辞知识。使用更多知识来源，增加了检索更无关紧要或嘈杂的事实的可能性，使其充实并找到答案的挑战。为了解决这一挑战，我们使用外部知识（MAVEX）提出了多模态答案验证，其中该想法是根据答案特定知识检索验证一组有希望的答案候选者。而不是在大多数现有方法中搜索大量不相关的事实中的答案，Mavex旨在学习如何从嘈杂来源中提取相关知识，这是对每个答复候选者的信任，以及如何使用候选者那个来源。除了以维基百科句子和概念概念的形式之外，我们的多模态设置是第一个利用外部视觉知识（使用谷歌搜索的图像）。我们的实验与OK-VQA是一个具有挑战性的知识VQA数据集，证明了MAVEX实现了新的最先进的结果。我们的代码可在https://github.com/jialinwu17/mavex提供

最近，3D视觉和语言任务吸引了不断增长的研究兴趣。与其他视觉和语言任务相比，3D视觉问题回答（VQA）任务的利用较小，并且更容易受到语言先验和共同参考的歧义。同时，由于规模和注释方法有限，最近提出的几个3D VQA数据集并不能很好地支持3D VQA任务。在这项工作中，我们通过收集一个新的3D VQA数据集（称为FE-3DGQA），正式定义和解决3D接地的VQA任务，并具有多样化且相对自由形式的提问，以及密集和完全接地的边界框注释。为了获得更多可解释的答案，我们标记了出现在复杂的质量检查对中的对象，该对象具有不同的语义类型，包括答案接地的对象（均出现并未出现在问题中），以及用于答案的对象的上下文对象。我们还提出了一个新的3D VQA框架，以有效地预测完全视觉扎根和可解释的答案。广泛的实验证明，我们新收集的基准数据集可有效地用于评估不同方面的各种3D VQA方法，而我们新提出的框架也可以在新的基准数据集中实现最新的性能。新收集的数据集和我们的代码都将在http://github.com/zlccccc/3dgqa上公开获得。