我们介绍了CLEVR-MATH，这是一个多模式数学单词问题数据集，该数据集由涉及加法/减法的简单数学单词问题组成，部分地表示文本描述，部分地是由图像说明了场景。文本描述了图像中描述的场景上执行的动作。由于提出的问题可能与图像中的场景有关，而是针对采用动作之前或之后的场景状态，因此求解器设想或想象由于这些动作而导致的状态发生了变化。解决这些单词问题需要语言，视觉和数学推理的结合。我们将最新的神经和神经符号模型应用于CLEVR-MATH的视觉问题，并经验评估其表现。我们的结果表明，两种方法如何推广到操作链。我们讨论了两者在解决多模式单词问题解决的任务时的局限性。

Artificial Intelligence (AI) and its applications have sparked extraordinary interest in recent years. This achievement can be ascribed in part to advances in AI subfields including Machine Learning (ML), Computer Vision (CV), and Natural Language Processing (NLP). Deep learning, a sub-field of machine learning that employs artificial neural network concepts, has enabled the most rapid growth in these domains. The integration of vision and language has sparked a lot of attention as a result of this. The tasks have been created in such a way that they properly exemplify the concepts of deep learning. In this review paper, we provide a thorough and an extensive review of the state of the arts approaches, key models design principles and discuss existing datasets, methods, their problem formulation and evaluation measures for VQA and Visual reasoning tasks to understand vision and language representation learning. We also present some potential future paths in this field of research, with the hope that our study may generate new ideas and novel approaches to handle existing difficulties and develop new applications.

When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover shortcomings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.

人类视觉感知的关键方面是能够将视觉场景分解为单个对象并进一步进入对象部分，形成部分整个层次结构。这种复合结构可以诱导丰富的语义概念和关系，从而在视觉信号的解释和组织中发挥着重要作用，以及视觉感知和推理的概括。但是，现有的视觉推理基准主要专注于物体而不是零件。基于完整的部分整个层次结构的视觉推理比以前粒度概念，更丰富的几何关系和更复杂的物理学所致的对象的推理更具挑战性。因此，为了更好地为基于部分的概念，关系和物理推理服务，我们介绍了一个名为PTR的新型大规模诊断视觉推理数据集。 PTR包含大约70k RGBD合成图像，具有地面真理对象和有关语义实例分段，颜色属性，空间和几何关系的部分级别注释，以及诸如稳定性的某些物理性质。这些图像与700K机生成的问题配对，涵盖各种类型的推理类型，使其成为视觉推理模型的良好测试平台。我们在这个数据集上检查了几种最先进的视觉推理模型，并观察到他们在人类可以容易地推断正确答案的情况下仍然存在许多令人惊讶的错误。我们认为，此数据集将开辟基于零件推理的新机会。

We introduce GQA, a new dataset for real-world visual reasoning and compositional question answering, seeking to address key shortcomings of previous VQA datasets. We have developed a strong and robust question engine that leverages Visual Genome scene graph structures to create 22M diverse reasoning questions, which all come with functional programs that represent their semantics. We use the programs to gain tight control over the answer distribution and present a new tunable smoothing technique to mitigate question biases. Accompanying the dataset is a suite of new metrics that evaluate essential qualities such as consistency, grounding and plausibility. A careful analysis is performed for baselines as well as state-of-the-art models, providing fine-grained results for different question types and topologies. Whereas a blind LSTM obtains a mere 42.1%, and strong VQA models achieve 54.1%, human performance tops at 89.3%, offering ample opportunity for new research to explore. We hope GQA will provide an enabling resource for the next generation of models with enhanced robustness, improved consistency, and deeper semantic understanding of vision and language.

Videos often capture objects, their visible properties, their motion, and the interactions between different objects. Objects also have physical properties such as mass, which the imaging pipeline is unable to directly capture. However, these properties can be estimated by utilizing cues from relative object motion and the dynamics introduced by collisions. In this paper, we introduce CRIPP-VQA, a new video question answering dataset for reasoning about the implicit physical properties of objects in a scene. CRIPP-VQA contains videos of objects in motion, annotated with questions that involve counterfactual reasoning about the effect of actions, questions about planning in order to reach a goal, and descriptive questions about visible properties of objects. The CRIPP-VQA test set enables evaluation under several out-of-distribution settings -- videos with objects with masses, coefficients of friction, and initial velocities that are not observed in the training distribution. Our experiments reveal a surprising and significant performance gap in terms of answering questions about implicit properties (the focus of this paper) and explicit properties of objects (the focus of prior work).

近年来，视觉问题应答（VQA）在近年来，由于了解来自多种方式的信息（即图像，语言），近年来近年来在近年来的机器学习社区中获得了很多牵引力。在VQA中，基于一组图像提出了一系列问题，并且手头的任务是到达答案。为实现这一目标，我们采用了一种基于象征的推理方法，使用正式逻辑框架。图像和问题被转换为执行显式推理的符号表示。我们提出了一种正式的逻辑框架，其中（i）图像在场景图的帮助下将图像转换为逻辑背景事实，（ii）问题被基于变压器的深度学习模型转换为一阶谓词逻辑条款，（iii）通过使用背景知识和谓词条款的接地来执行可靠性检查，以获得答案。我们所提出的方法是高度解释的，并且可以通过人容易地分析管道中的每个步骤。我们验证了我们在CLEVR和GQA数据集上的方法。我们在Clevr DataSet上实现了99.6％的近似完美的准确性，可与艺术模式相当，展示正式逻辑是一个可行的工具来解决视觉问题的回答。我们的模型也是数据高效，在仅在培训数据的10％培训时，在缩放数据集中实现99.1％的准确性。

我们提出了神经符号视觉对话框（NSVD），这是将深度学习和符号程序执行的第一种方法进行多轮视觉上的推理。 NSVD对视觉对话固有的两个关键挑战：长距离共同参考分辨率以及消失的问题效能绩效大大优于现有的纯粹连接主义方法。我们通过提出一个更现实，更严格的评估方案来演示后者，在计算准确性时，我们在其中为完整对话记录使用预测的答案。我们描述了我们模型的两个变体，并表明使用这种新方案，我们的最佳模型在CLEVR -DIALOG上的准确度达到99.72％ - 相对提高了10％以上的技术，而仅需培训的一部分即可数据。此外，我们证明了我们的神经符号模型具有更高的平均第一场失败回合，对不完整的对话历史更强大，并且不仅概括了对话的最多三倍的对话，而且还比在训练中看到的对话更长，而且还要看不见问题。类型和场景。

目前的视觉问题应答（VQA）任务主要考虑回答自然图像的人为注释问题。然而，除了自然图像之外，在视觉理解和推理研究中仍然可以解读具有语义丰富性的抽象图。在这项工作中，我们介绍了ICON问题的新挑战（ICONQA），其目标是在图标图像上下文中回答问题。我们发布了ICONQA，这是一个由107,439个问题和三个子任务组成的大型数据集：多图像选择，多文本选择和填充空白。 ICONQA数据集是由真实世界图中的启发，突出了抽象图理解和综合认知推理的重要性。因此，ICONQA不仅需要对象识别和文本理解等感知技能，而且还需要多种认知推理技能，例如几何推理，致辞推理和算术推理。为了促进潜在的iconqa模型来学习图标图像的语义表示，我们进一步发布了一个图标数据集图标645，其中包含377级上的645,687个彩色图标。我们进行广泛的用户研究和盲目实验，并重现各种先进的VQA方法来基准iconQA任务。此外，我们开发了一个强大的ICONQA基线Patch-TRM，它应用金字塔跨模型变压器，其中包含在图标数据集上预先培训的输入图嵌入式。 iconqa和图标645可在https://iconqa.github.io提供。

Visual Question Answering (VQA) models often perform poorly on out-of-distribution data and struggle on domain generalization. Due to the multi-modal nature of this task, multiple factors of variation are intertwined, making generalization difficult to analyze. This motivates us to introduce a virtual benchmark, Super-CLEVR, where different factors in VQA domain shifts can be isolated in order that their effects can be studied independently. Four factors are considered: visual complexity, question redundancy, concept distribution and concept compositionality. With controllably generated data, Super-CLEVR enables us to test VQA methods in situations where the test data differs from the training data along each of these axes. We study four existing methods, including two neural symbolic methods NSCL and NSVQA, and two non-symbolic methods FiLM and mDETR; and our proposed method, probabilistic NSVQA (P-NSVQA), which extends NSVQA with uncertainty reasoning. P-NSVQA outperforms other methods on three of the four domain shift factors. Our results suggest that disentangling reasoning and perception, combined with probabilistic uncertainty, form a strong VQA model that is more robust to domain shifts. The dataset and code are released at https://github.com/Lizw14/Super-CLEVR.

Visual question answering is fundamentally compositional in nature-a question like where is the dog? shares substructure with questions like what color is the dog? and where is the cat? This paper seeks to simultaneously exploit the representational capacity of deep networks and the compositional linguistic structure of questions. We describe a procedure for constructing and learning neural module networks, which compose collections of jointly-trained neural "modules" into deep networks for question answering. Our approach decomposes questions into their linguistic substructures, and uses these structures to dynamically instantiate modular networks (with reusable components for recognizing dogs, classifying colors, etc.). The resulting compound networks are jointly trained. We evaluate our approach on two challenging datasets for visual question answering, achieving state-of-the-art results on both the VQA natural image dataset and a new dataset of complex questions about abstract shapes.

在本报告中，我们展示了ICDAR 2021版文档视觉问题挑战的结果。此版本在单个文档VQA和Document Collection VQA上补充了以前的任务，并在Infographics VQA上进行了新引入的。信息图表VQA基于超过5,000个信息图表图像和30,000个问题答案对的新数据集。获胜者方法在Infographics VQA任务中获得了0.6120个ANL，0.7743 anlsl在文档集中的VQA任务和单个文档VQA中的0.8705 ANL中。我们展示了用于每个任务的数据集的摘要，每个提交的方法的描述以及它们的性能的结果和分析。由于还提出了自从第一版DocVQA 2020挑战以来在单个文档VQA上取得的摘要。

Video Question Answering methods focus on commonsense reasoning and visual cognition of objects or persons and their interactions over time. Current VideoQA approaches ignore the textual information present in the video. Instead, we argue that textual information is complementary to the action and provides essential contextualisation cues to the reasoning process. To this end, we propose a novel VideoQA task that requires reading and understanding the text in the video. To explore this direction, we focus on news videos and require QA systems to comprehend and answer questions about the topics presented by combining visual and textual cues in the video. We introduce the ``NewsVideoQA'' dataset that comprises more than $8,600$ QA pairs on $3,000+$ news videos obtained from diverse news channels from around the world. We demonstrate the limitations of current Scene Text VQA and VideoQA methods and propose ways to incorporate scene text information into VideoQA methods.

Visual understanding goes well beyond object recognition. With one glance at an image, we can effortlessly imagine the world beyond the pixels: for instance, we can infer people's actions, goals, and mental states. While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world. We formalize this task as Visual Commonsense Reasoning. Given a challenging question about an image, a machine must answer correctly and then provide a rationale justifying its answer.Next, we introduce a new dataset, VCR, consisting of 290k multiple choice QA problems derived from 110k movie scenes. The key recipe for generating non-trivial and highquality problems at scale is Adversarial Matching, a new approach to transform rich annotations into multiple choice questions with minimal bias. Experimental results show that while humans find VCR easy (over 90% accuracy), state-of-the-art vision models struggle (∼45%).To move towards cognition-level understanding, we present a new reasoning engine, Recognition to Cognition Networks (R2C), that models the necessary layered inferences for grounding, contextualization, and reasoning. R2C helps narrow the gap between humans and machines (∼65%); still, the challenge is far from solved, and we provide analysis that suggests avenues for future work.

视觉问题回答（VQA）近年来见证了巨大进展。但是，大多数努力只关注2D图像问题应答任务。在本文中，我们介绍了将VQA扩展到3D域的第一次尝试，这可以促进人工智能对3D现实世界情景的看法。与基于图像的VQA不同，3D问题应答（3DQA）将颜色点云作为输入，需要外观和3D几何理解能力来回答3D相关问题。为此，我们提出了一种基于新颖的基于变换器的3DQA框架\ TextBF {“3DQA-TR”}，其包括两个编码器，分别用于利用外观和几何信息。外观，几何和的多模码信息语言问题最终可以通过3D语言伯特互相参加，以预测目标答案。要验证我们提出的3DQA框架的有效性，我们还开发了第一个建立的3DQA DataSet \ TextBF {“scanqa”} SCANNet DataSet并包含$ \ SIM $ 6K问题，$ \ SIM $ 30k答案，可满足806美元的场景。在此数据集上的广泛实验展示了我们提出的3DQA框架在现有的VQA框架上的明显优势，以及我们主要设计的有效性。我们的代码和数据集将公开可用于促进此方向的研究。

在回答问题时，人类会利用跨不同模式可用的信息来综合一致，完整的思想链（COT）。在深度学习模型（例如大规模语言模型）的情况下，这个过程通常是黑匣子。最近，科学问题基准已用于诊断AI系统的多跳推理能力和解释性。但是，现有数据集无法为答案提供注释，或仅限于仅文本模式，小尺度和有限的域多样性。为此，我们介绍了科学问题答案（SQA），这是一个新的基准，由〜21k的多模式多种选择问题组成，其中包含各种科学主题和答案的注释，并提供相应的讲座和解释。我们进一步设计语言模型，以学习将讲座和解释作为思想链（COT），以模仿回答SQA问题时的多跳上推理过程。 SQA在语言模型中展示了COT的实用性，因为COT将问题的答案绩效提高了1.20％的GPT-3和3.99％的unifiedqa。我们还探索了模型的上限，以通过喂食输入中的那些来利用解释；我们观察到它将GPT-3的少量性能提高了18.96％。我们的分析进一步表明，与人类类似的语言模型受益于解释，从较少的数据中学习并仅使用40％的数据实现相同的性能。

We propose the task of free-form and open-ended Visual Question Answering (VQA). Given an image and a natural language question about the image, the task is to provide an accurate natural language answer. Mirroring real-world scenarios, such as helping the visually impaired, both the questions and answers are open-ended. Visual questions selectively target different areas of an image, including background details and underlying context. As a result, a system that succeeds at VQA typically needs a more detailed understanding of the image and complex reasoning than a system producing generic image captions. Moreover, VQA is amenable to automatic evaluation, since many open-ended answers contain only a few words or a closed set of answers that can be provided in a multiple-choice format. We provide a dataset containing ∼0.25M images, ∼0.76M questions, and ∼10M answers (www.visualqa.org), and discuss the information it provides. Numerous baselines and methods for VQA are provided and compared with human performance. Our VQA demo is available on CloudCV (http://cloudcv.org/vqa).

视觉问题回答是自然语言和愿景理解的重要任务。但是，在大多数公众视觉问题上回答了诸如VQA，CLEVR之类的数据集，这些问题是针对给定图像的特定于“她的眼睛是什么颜色？”的人类产生的。人类产生的众包问题相对简单，有时对某些实体或属性有偏见。在本文中，我们介绍了一个基于Image-Chiqa的新问题回答数据集。它包含Internet用户发布的现实查询，并结合了几个相关的开放域图像。系统应确定图像是否可以回答问题。与以前的VQA数据集不同，这些问题是现实世界中独立的查询，这些查询更加各种和无偏见。与先前的图像回程或图像捕获数据集相比，Chiqa不仅衡量了相关性，而且还可以衡量答案性，这需要更细粒度的视力和语言推理。 Chiqa包含超过40k的问题和超过200k的问题图像对。将三级2/1/0标签分配给每个对，指示完美的答案，部分答案和无关紧要。数据分析表明，Chiqa需要对语言和视觉有深入的了解，包括接地，比较和阅读。我们评估了几种最先进的视觉语言模型，例如ALBEF，表明仍然有一个很大的改进奇卡的空间。

Mathematical reasoning is a fundamental aspect of human intelligence and is applicable in various fields, including science, engineering, finance, and everyday life. The development of artificial intelligence (AI) systems capable of solving math problems and proving theorems has garnered significant interest in the fields of machine learning and natural language processing. For example, mathematics serves as a testbed for aspects of reasoning that are challenging for powerful deep learning models, driving new algorithmic and modeling advances. On the other hand, recent advances in large-scale neural language models have opened up new benchmarks and opportunities to use deep learning for mathematical reasoning. In this survey paper, we review the key tasks, datasets, and methods at the intersection of mathematical reasoning and deep learning over the past decade. We also evaluate existing benchmarks and methods, and discuss future research directions in this domain.

'Actions' play a vital role in how humans interact with the world. Thus, autonomous agents that would assist us in everyday tasks also require the capability to perform 'Reasoning about Actions & Change' (RAC). This has been an important research direction in Artificial Intelligence (AI) in general, but the study of RAC with visual and linguistic inputs is relatively recent. The CLEVR_HYP (Sampat et. al., 2021) is one such testbed for hypothetical vision-language reasoning with actions as the key focus. In this work, we propose a novel learning strategy that can improve reasoning about the effects of actions. We implement an encoder-decoder architecture to learn the representation of actions as vectors. We combine the aforementioned encoder-decoder architecture with existing modality parsers and a scene graph question answering model to evaluate our proposed system on the CLEVR_HYP dataset. We conduct thorough experiments to demonstrate the effectiveness of our proposed approach and discuss its advantages over previous baselines in terms of performance, data efficiency, and generalization capability.