作为人类，我们可以通过想象我们的思想中的替代对象或概念来修改对场景的假设。例如，我们可以轻松地预见到雨云（例如，街道会被弄湿），并为此做准备。在本文中，我们介绍了一个新任务/数据集，称为反事实场景（COSIM），旨在评估AI系统对场景变化想象的推论的能力。在此任务/数据集中，为图像提供了模型和一个初始的问题响应对。接下来，应用了反事实想象的场景更改（以文本形式），该模型必须根据此场景更改预测对初始问题的新回答。我们收集3.5k高质量和具有挑战性的数据实例，每个实例都包含图像，一个常识性问题，响应，对反事实变化的描述，对问题的新回答以及三个干扰器响应。我们的数据集包含各种复杂的场景更改类型（例如对象添加/删除/状态更改，事件描述，环境更改等），这些更改需要模型来想象许多不同的场景和原因。我们提出了基于视觉变压器（即LXMERT）和消融研究的基线模型。通过人类的评估，我们证明了较大的人类模型性能差距，这为有望在这项具有挑战性的反事实，场景想象任务上做出有希望的未来工作。我们的代码和数据集可公开可用：https：//github.com/hyounghk/cosim

Visual understanding goes well beyond object recognition. With one glance at an image, we can effortlessly imagine the world beyond the pixels: for instance, we can infer people's actions, goals, and mental states. While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world. We formalize this task as Visual Commonsense Reasoning. Given a challenging question about an image, a machine must answer correctly and then provide a rationale justifying its answer.Next, we introduce a new dataset, VCR, consisting of 290k multiple choice QA problems derived from 110k movie scenes. The key recipe for generating non-trivial and highquality problems at scale is Adversarial Matching, a new approach to transform rich annotations into multiple choice questions with minimal bias. Experimental results show that while humans find VCR easy (over 90% accuracy), state-of-the-art vision models struggle (∼45%).To move towards cognition-level understanding, we present a new reasoning engine, Recognition to Cognition Networks (R2C), that models the necessary layered inferences for grounding, contextualization, and reasoning. R2C helps narrow the gap between humans and machines (∼65%); still, the challenge is far from solved, and we provide analysis that suggests avenues for future work.

We propose the task of free-form and open-ended Visual Question Answering (VQA). Given an image and a natural language question about the image, the task is to provide an accurate natural language answer. Mirroring real-world scenarios, such as helping the visually impaired, both the questions and answers are open-ended. Visual questions selectively target different areas of an image, including background details and underlying context. As a result, a system that succeeds at VQA typically needs a more detailed understanding of the image and complex reasoning than a system producing generic image captions. Moreover, VQA is amenable to automatic evaluation, since many open-ended answers contain only a few words or a closed set of answers that can be provided in a multiple-choice format. We provide a dataset containing ∼0.25M images, ∼0.76M questions, and ∼10M answers (www.visualqa.org), and discuss the information it provides. Numerous baselines and methods for VQA are provided and compared with human performance. Our VQA demo is available on CloudCV (http://cloudcv.org/vqa).

为了使AI安全地在医院，学校和工作场所等现实世界中安全部署，它必须能够坚定地理解物理世界。这种推理的基础是物理常识：了解可用对象的物理特性和提供的能力，如何被操纵以及它们如何与其他对象进行交互。物理常识性推理从根本上是一项多感官任务，因为物理特性是通过多种模式表现出来的，其中两个是视觉和声学。我们的论文通过贡献PACS来朝着现实世界中的物理常识推理：第一个用于物理常识属性注释的视听基准。 PACS包含13,400对答案对，涉及1,377个独特的物理常识性问题和1,526个视频。我们的数据集提供了新的机会来通过将音频作为此多模式问题的核心组成部分来推进物理推理的研究领域。使用PACS，我们在我们的新挑战性任务上评估了多种最先进的模型。尽管某些模型显示出令人鼓舞的结果（精度为70％），但它们都没有人类的绩效（精度为95％）。我们通过证明多模式推理的重要性并为未来的研究提供了可能的途径来结束本文。

When answering a question, people often draw upon their rich world knowledge in addition to the particular context. Recent work has focused primarily on answering questions given some relevant document or context, and required very little general background. To investigate question answering with prior knowledge, we present COMMONSENSEQA: a challenging new dataset for commonsense question answering. To capture common sense beyond associations, we extract from CON-CEPTNET (Speer et al., 2017) multiple target concepts that have the same semantic relation to a single source concept. Crowd-workers are asked to author multiple-choice questions that mention the source concept and discriminate in turn between each of the target concepts. This encourages workers to create questions with complex semantics that often require prior knowledge. We create 12,247 questions through this procedure and demonstrate the difficulty of our task with a large number of strong baselines. Our best baseline is based on BERT-large (Devlin et al., 2018) and obtains 56% accuracy, well below human performance, which is 89%.

From a visual scene containing multiple people, human is able to distinguish each individual given the context descriptions about what happened before, their mental/physical states or intentions, etc. Above ability heavily relies on human-centric commonsense knowledge and reasoning. For example, if asked to identify the "person who needs healing" in an image, we need to first know that they usually have injuries or suffering expressions, then find the corresponding visual clues before finally grounding the person. We present a new commonsense task, Human-centric Commonsense Grounding, that tests the models' ability to ground individuals given the context descriptions about what happened before, and their mental/physical states or intentions. We further create a benchmark, HumanCog, a dataset with 130k grounded commonsensical descriptions annotated on 67k images, covering diverse types of commonsense and visual scenes. We set up a context-object-aware method as a strong baseline that outperforms previous pre-trained and non-pretrained models. Further analysis demonstrates that rich visual commonsense and powerful integration of multi-modal commonsense are essential, which sheds light on future works. Data and code will be available https://github.com/Hxyou/HumanCog.

人类具有出色的能力来推理绑架并假设超出图像的字面内容的内容。通过识别散布在整个场景中的具体视觉线索，我们几乎不禁根据我们的日常经验和对世界的知识来提出可能的推论。例如，如果我们在道路旁边看到一个“ 20英里 /小时”的标志，我们可能会假设街道位于居民区（而不是在高速公路上），即使没有房屋。机器可以执行类似的视觉推理吗？我们提出了Sherlock，这是一个带注释的103K图像的语料库，用于测试机器能力，以超出字面图像内容的绑架推理。我们采用免费观看范式：参与者首先观察并识别图像中的显着线索（例如，对象，动作），然后给定线索，然后提供有关场景的合理推论。我们总共收集了363K（线索，推理）对，该对形成了首个绑架的视觉推理数据集。使用我们的语料库，我们测试了三个互补的绑架推理轴。我们评估模型的能力：i）从大型候选人语料库中检索相关推论； ii）通过边界框来定位推论的证据，iii）比较合理的推论，以匹配人类在新收集的19k李克特级判断的诊断语料库上的判断。尽管我们发现具有多任务目标的微调夹RN50x64优于强大的基准，但模型性能与人类一致之间存在着重要的净空。可在http://visualabduction.com/上获得数据，模型和排行榜

相同上下文的可能后果可能会因我们所指的情况而异。但是，当前在自然语言处理中的研究并不集中于多种可能情况下的常识性推理。本研究通过短篇小说文字提出与候选人答案相同的结尾的多个问题来构成这项任务。我们由此产生的数据集，可能的故事，包括超过1.3k的故事文本超过4.5k的问题。我们发现，即使是目前的强训练性语言模型也很难始终如一地回答问题，这强调了无监督环境中最高的准确性（60.2％）远远落后于人类准确性（92.5％）。通过与现有数据集进行比较，我们观察到数据集中的问题包含答案选项中的最小注释伪像。此外，我们的数据集还包括需要反事实推理的示例，以及需要读者的反应和虚构信息的示例，这表明我们的数据集可以作为对未来常识性推理的未来研究的挑战性测试。

目前的视觉问题应答（VQA）任务主要考虑回答自然图像的人为注释问题。然而，除了自然图像之外，在视觉理解和推理研究中仍然可以解读具有语义丰富性的抽象图。在这项工作中，我们介绍了ICON问题的新挑战（ICONQA），其目标是在图标图像上下文中回答问题。我们发布了ICONQA，这是一个由107,439个问题和三个子任务组成的大型数据集：多图像选择，多文本选择和填充空白。 ICONQA数据集是由真实世界图中的启发，突出了抽象图理解和综合认知推理的重要性。因此，ICONQA不仅需要对象识别和文本理解等感知技能，而且还需要多种认知推理技能，例如几何推理，致辞推理和算术推理。为了促进潜在的iconqa模型来学习图标图像的语义表示，我们进一步发布了一个图标数据集图标645，其中包含377级上的645,687个彩色图标。我们进行广泛的用户研究和盲目实验，并重现各种先进的VQA方法来基准iconQA任务。此外，我们开发了一个强大的ICONQA基线Patch-TRM，它应用金字塔跨模型变压器，其中包含在图标数据集上预先培训的输入图嵌入式。 iconqa和图标645可在https://iconqa.github.io提供。

虽然视觉和语言模型在视觉问题回答等任务上表现良好，但在基本的人类常识性推理技能方面，它们会挣扎。在这项工作中，我们介绍了Winogavil：在线游戏，以收集视觉和语言协会（例如，狼人到满月），用作评估最先进模型的动态基准。受欢迎的纸牌游戏代号的启发，Spymaster提供了与几个视觉候选者相关的文本提示，另一个玩家必须识别它们。人类玩家因创建对竞争对手AI模型而具有挑战性的联想而获得了回报，但仍然可以由其他人类玩家解决。我们使用游戏来收集3.5k实例，发现它们对人类的直观（> 90％的Jaccard索引），但对最先进的AI模型充满挑战，其中最佳模型（Vilt）的得分为52％ ，成功的位置在视觉上是显着的。我们的分析以及我们从玩家那里收集的反馈表明，收集的关联需要多种推理技能，包括一般知识，常识，抽象等。我们发布数据集，代码和交互式游戏，旨在允许未来的数据收集，可用于开发具有更好关联能力的模型。

视觉问题应答（VQA）任务利用视觉图像和语言分析来回回答图像的文本问题。它是一个流行的研究课题，在过去十年中越来越多的现实应用。本文介绍了我们最近对AliceMind-MMU的研究（阿里巴巴的编码器 - 解码器来自Damo Academy - 多媒体理解的机器智能实验室），其比人类在VQA上获得相似甚至略微更好的结果。这是通过系统地改善VQA流水线来实现的，包括：（1）具有全面的视觉和文本特征表示的预培训; （2）与学习参加的有效跨模型互动; （3）一个新颖的知识挖掘框架，具有专门的专业专家模块，适用于复杂的VQA任务。处理不同类型的视觉问题，需要具有相应的专业知识在提高我们的VQA架构的表现方面发挥着重要作用，这取决于人力水平。进行了广泛的实验和分析，以证明新的研究工作的有效性。

叙事中的事件可以通过其参与者的基本状态理解为一致的整体。通常，这些参与者在叙述中没有明确提及，而是通过常识性或推论填写。理解叙述的模型应该能够推断出这些隐性参与者状态，以及有关这些状态对叙事的影响的原因。为了促进这一目标，我们介绍了一个新的众包参与者指出的数据集意大利面。该数据集包含有效的，可推断的参与者状态；对国家的反事实扰动；如果反事实是真实的，那么故事的变化将是必要的。我们介绍了三项基于州的推理任务，这些任务测试了一个故事何时由故事启用，修改一个反事实状态的故事，并解释给定经过修订的故事的最有可能的状态变化。我们的基准测试实验表明，尽管当今的LLM能够在某种程度上推理有关州的推理，但仍有很大的改进空间，这表明了未来研究的潜在途径。

视觉问题回答是自然语言和愿景理解的重要任务。但是，在大多数公众视觉问题上回答了诸如VQA，CLEVR之类的数据集，这些问题是针对给定图像的特定于“她的眼睛是什么颜色？”的人类产生的。人类产生的众包问题相对简单，有时对某些实体或属性有偏见。在本文中，我们介绍了一个基于Image-Chiqa的新问题回答数据集。它包含Internet用户发布的现实查询，并结合了几个相关的开放域图像。系统应确定图像是否可以回答问题。与以前的VQA数据集不同，这些问题是现实世界中独立的查询，这些查询更加各种和无偏见。与先前的图像回程或图像捕获数据集相比，Chiqa不仅衡量了相关性，而且还可以衡量答案性，这需要更细粒度的视力和语言推理。 Chiqa包含超过40k的问题和超过200k的问题图像对。将三级2/1/0标签分配给每个对，指示完美的答案，部分答案和无关紧要。数据分析表明，Chiqa需要对语言和视觉有深入的了解，包括接地，比较和阅读。我们评估了几种最先进的视觉语言模型，例如ALBEF，表明仍然有一个很大的改进奇卡的空间。

Video Question Answering methods focus on commonsense reasoning and visual cognition of objects or persons and their interactions over time. Current VideoQA approaches ignore the textual information present in the video. Instead, we argue that textual information is complementary to the action and provides essential contextualisation cues to the reasoning process. To this end, we propose a novel VideoQA task that requires reading and understanding the text in the video. To explore this direction, we focus on news videos and require QA systems to comprehend and answer questions about the topics presented by combining visual and textual cues in the video. We introduce the ``NewsVideoQA'' dataset that comprises more than $8,600$ QA pairs on $3,000+$ news videos obtained from diverse news channels from around the world. We demonstrate the limitations of current Scene Text VQA and VideoQA methods and propose ways to incorporate scene text information into VideoQA methods.

Videos often capture objects, their visible properties, their motion, and the interactions between different objects. Objects also have physical properties such as mass, which the imaging pipeline is unable to directly capture. However, these properties can be estimated by utilizing cues from relative object motion and the dynamics introduced by collisions. In this paper, we introduce CRIPP-VQA, a new video question answering dataset for reasoning about the implicit physical properties of objects in a scene. CRIPP-VQA contains videos of objects in motion, annotated with questions that involve counterfactual reasoning about the effect of actions, questions about planning in order to reach a goal, and descriptive questions about visible properties of objects. The CRIPP-VQA test set enables evaluation under several out-of-distribution settings -- videos with objects with masses, coefficients of friction, and initial velocities that are not observed in the training distribution. Our experiments reveal a surprising and significant performance gap in terms of answering questions about implicit properties (the focus of this paper) and explicit properties of objects (the focus of prior work).

We have seen great progress in basic perceptual tasks such as object recognition and detection. However, AI models still fail to match humans in high-level vision tasks due to the lack of capacities for deeper reasoning. Recently the new task of visual question answering (QA) has been proposed to evaluate a model's capacity for deep image understanding. Previous works have established a loose, global association between QA sentences and images. However, many questions and answers, in practice, relate to local regions in the images. We establish a semantic link between textual descriptions and image regions by object-level grounding. It enables a new type of QA with visual answers, in addition to textual answers used in previous work. We study the visual QA tasks in a grounded setting with a large collection of 7W multiple-choice QA pairs. Furthermore, we evaluate human performance and several baseline models on the QA tasks. Finally, we propose a novel LSTM model with spatial attention to tackle the 7W QA tasks.

Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in

'Actions' play a vital role in how humans interact with the world. Thus, autonomous agents that would assist us in everyday tasks also require the capability to perform 'Reasoning about Actions & Change' (RAC). This has been an important research direction in Artificial Intelligence (AI) in general, but the study of RAC with visual and linguistic inputs is relatively recent. The CLEVR_HYP (Sampat et. al., 2021) is one such testbed for hypothetical vision-language reasoning with actions as the key focus. In this work, we propose a novel learning strategy that can improve reasoning about the effects of actions. We implement an encoder-decoder architecture to learn the representation of actions as vectors. We combine the aforementioned encoder-decoder architecture with existing modality parsers and a scene graph question answering model to evaluate our proposed system on the CLEVR_HYP dataset. We conduct thorough experiments to demonstrate the effectiveness of our proposed approach and discuss its advantages over previous baselines in terms of performance, data efficiency, and generalization capability.

为了实现长文档理解的构建和测试模型，我们引入质量，具有中文段的多项选择QA DataSet，具有约5,000个令牌的平均长度，比典型的当前模型更长。与经过段落的事先工作不同，我们的问题是由阅读整个段落的贡献者编写和验证的，而不是依赖摘要或摘录。此外，只有一半的问题是通过在紧缩时间限制下工作的注释器来应答，表明略读和简单的搜索不足以一直表现良好。目前的模型在此任务上表现不佳（55.4％），并且落后于人类性能（93.5％）。

Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. We introduce HOTPOTQA, a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) we provide sentence-level supporting facts required for reasoning, allowing QA systems to reason with strong supervision and explain the predictions; (4) we offer a new type of factoid comparison questions to test QA systems' ability to extract relevant facts and perform necessary comparison. We show that HOTPOTQA is challenging for the latest QA systems, and the supporting facts enable models to improve performance and make explainable predictions.