近年来，视觉问题应答（VQA）在近年来，由于了解来自多种方式的信息（即图像，语言），近年来近年来在近年来的机器学习社区中获得了很多牵引力。在VQA中，基于一组图像提出了一系列问题，并且手头的任务是到达答案。为实现这一目标，我们采用了一种基于象征的推理方法，使用正式逻辑框架。图像和问题被转换为执行显式推理的符号表示。我们提出了一种正式的逻辑框架，其中（i）图像在场景图的帮助下将图像转换为逻辑背景事实，（ii）问题被基于变压器的深度学习模型转换为一阶谓词逻辑条款，（iii）通过使用背景知识和谓词条款的接地来执行可靠性检查，以获得答案。我们所提出的方法是高度解释的，并且可以通过人容易地分析管道中的每个步骤。我们验证了我们在CLEVR和GQA数据集上的方法。我们在Clevr DataSet上实现了99.6％的近似完美的准确性，可与艺术模式相当，展示正式逻辑是一个可行的工具来解决视觉问题的回答。我们的模型也是数据高效，在仅在培训数据的10％培训时，在缩放数据集中实现99.1％的准确性。

When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover shortcomings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.

Relational reasoning is a central component of generally intelligent behavior, but has proven difficult for neural networks to learn. In this paper we describe how to use Relation Networks (RNs) as a simple plug-and-play module to solve problems that fundamentally hinge on relational reasoning. We tested RN-augmented networks on three tasks: visual question answering using a challenging dataset called CLEVR, on which we achieve state-of-the-art, super-human performance; text-based question answering using the bAbI suite of tasks; and complex reasoning about dynamic physical systems. Then, using a curated dataset called Sort-of-CLEVR we show that powerful convolutional networks do not have a general capacity to solve relational questions, but can gain this capacity when augmented with RNs. Our work shows how a deep learning architecture equipped with an RN module can implicitly discover and learn to reason about entities and their relations.

我们介绍了CLEVR-MATH，这是一个多模式数学单词问题数据集，该数据集由涉及加法/减法的简单数学单词问题组成，部分地表示文本描述，部分地是由图像说明了场景。文本描述了图像中描述的场景上执行的动作。由于提出的问题可能与图像中的场景有关，而是针对采用动作之前或之后的场景状态，因此求解器设想或想象由于这些动作而导致的状态发生了变化。解决这些单词问题需要语言，视觉和数学推理的结合。我们将最新的神经和神经符号模型应用于CLEVR-MATH的视觉问题，并经验评估其表现。我们的结果表明，两种方法如何推广到操作链。我们讨论了两者在解决多模式单词问题解决的任务时的局限性。

尽管在现代的机器学习算法的最新进展，其内在机制的不透明仍是采用的障碍。在人工智能系统灌输信心和信任，解释的人工智能已成为提高现代机器学习算法explainability的响应。归纳逻辑程序（ILP），符号人工智能的子场中，起着产生，因为它的直观的逻辑驱动框架的可解释的解释有希望的作用。 ILP有效利用绎推理产生从实例和背景知识解释的一阶分句理论。然而，在发展中通过ILP需要启发方法的几个挑战，在实践中他们的成功应用来解决。例如，现有的ILP系统通常拥有广阔的解空间，以及感应解决方案是对噪声和干扰非常敏感。本次调查总结在ILP的最新进展和统计关系学习和神经象征算法的讨论，其中提供给ILP协同意见。继最新进展的严格审查，我们划定观察的挑战，突出对发展不言自明的人工智能系统进一步ILP动机研究的潜在途径。

We introduce GQA, a new dataset for real-world visual reasoning and compositional question answering, seeking to address key shortcomings of previous VQA datasets. We have developed a strong and robust question engine that leverages Visual Genome scene graph structures to create 22M diverse reasoning questions, which all come with functional programs that represent their semantics. We use the programs to gain tight control over the answer distribution and present a new tunable smoothing technique to mitigate question biases. Accompanying the dataset is a suite of new metrics that evaluate essential qualities such as consistency, grounding and plausibility. A careful analysis is performed for baselines as well as state-of-the-art models, providing fine-grained results for different question types and topologies. Whereas a blind LSTM obtains a mere 42.1%, and strong VQA models achieve 54.1%, human performance tops at 89.3%, offering ample opportunity for new research to explore. We hope GQA will provide an enabling resource for the next generation of models with enhanced robustness, improved consistency, and deeper semantic understanding of vision and language.

Neural-symbolic computing (NeSy), which pursues the integration of the symbolic and statistical paradigms of cognition, has been an active research area of Artificial Intelligence (AI) for many years. As NeSy shows promise of reconciling the advantages of reasoning and interpretability of symbolic representation and robust learning in neural networks, it may serve as a catalyst for the next generation of AI. In the present paper, we provide a systematic overview of the important and recent developments of research on NeSy AI. Firstly, we introduce study history of this area, covering early work and foundations. We further discuss background concepts and identify key driving factors behind the development of NeSy. Afterward, we categorize recent landmark approaches along several main characteristics that underline this research paradigm, including neural-symbolic integration, knowledge representation, knowledge embedding, and functionality. Then, we briefly discuss the successful application of modern NeSy approaches in several domains. Finally, we identify the open problems together with potential future research directions. This survey is expected to help new researchers enter this rapidly-developing field and accelerate progress towards data-and knowledge-driven AI.

我们提出了一种有效的可解释的神经象征模型来解决感应逻辑编程（ILP）问题。在该模型中，该模型是由在分层结构中组织的一组元规则构建的，通过学习嵌入来匹配元规则的事实和身体谓词来发明一阶规则。为了实例化它，我们专门设计了一种表现型通用元规则集，并证明了它们产生的喇叭条件的片段。在培训期间，我们注入了控制的\ PW {gumbel}噪声以避免本地最佳，并采用可解释性 - 正则化术语来进一步指导融合到可解释规则。我们在针对几种最先进的方法上证明我们对各种任务（ILP，视觉基因组，强化学习）的模型进行了验证。

Artificial Intelligence (AI) and its applications have sparked extraordinary interest in recent years. This achievement can be ascribed in part to advances in AI subfields including Machine Learning (ML), Computer Vision (CV), and Natural Language Processing (NLP). Deep learning, a sub-field of machine learning that employs artificial neural network concepts, has enabled the most rapid growth in these domains. The integration of vision and language has sparked a lot of attention as a result of this. The tasks have been created in such a way that they properly exemplify the concepts of deep learning. In this review paper, we provide a thorough and an extensive review of the state of the arts approaches, key models design principles and discuss existing datasets, methods, their problem formulation and evaluation measures for VQA and Visual reasoning tasks to understand vision and language representation learning. We also present some potential future paths in this field of research, with the hope that our study may generate new ideas and novel approaches to handle existing difficulties and develop new applications.

本文提出了一种基于答案设置编程（ASP）的方法，用于代表自然语言文本生成的知识。文本中的知识是使用Neo Davidsonian的形式主义建模的，然后将其表示为答案集计划。相关的致辞知识另外导入Wordnet等资源，并在ASP中表示。然后可以使用所产生的知识库来在ASP系统的帮助下执行推理。这种方法可以促进许多自然语言任务，如自动问题应答，文本摘要和自动化问题。基于ASP的技术表示，例如默认推理，分层知识组织，默认值等的首选项，用于模拟完成这些任务所需的致辞推理方法。在本文中，我们描述了我们开发的CaspR系统，以自动解决在给出英语文本时回答自然语言问题的任务。 CASPR可以被视为一个系统，通过“了解”文本并已在队列数据集上进行了测试，具有有希望的结果。

人类视觉感知的关键方面是能够将视觉场景分解为单个对象并进一步进入对象部分，形成部分整个层次结构。这种复合结构可以诱导丰富的语义概念和关系，从而在视觉信号的解释和组织中发挥着重要作用，以及视觉感知和推理的概括。但是，现有的视觉推理基准主要专注于物体而不是零件。基于完整的部分整个层次结构的视觉推理比以前粒度概念，更丰富的几何关系和更复杂的物理学所致的对象的推理更具挑战性。因此，为了更好地为基于部分的概念，关系和物理推理服务，我们介绍了一个名为PTR的新型大规模诊断视觉推理数据集。 PTR包含大约70k RGBD合成图像，具有地面真理对象和有关语义实例分段，颜色属性，空间和几何关系的部分级别注释，以及诸如稳定性的某些物理性质。这些图像与700K机生成的问题配对，涵盖各种类型的推理类型，使其成为视觉推理模型的良好测试平台。我们在这个数据集上检查了几种最先进的视觉推理模型，并观察到他们在人类可以容易地推断正确答案的情况下仍然存在许多令人惊讶的错误。我们认为，此数据集将开辟基于零件推理的新机会。

Visual question answering is fundamentally compositional in nature-a question like where is the dog? shares substructure with questions like what color is the dog? and where is the cat? This paper seeks to simultaneously exploit the representational capacity of deep networks and the compositional linguistic structure of questions. We describe a procedure for constructing and learning neural module networks, which compose collections of jointly-trained neural "modules" into deep networks for question answering. Our approach decomposes questions into their linguistic substructures, and uses these structures to dynamically instantiate modular networks (with reusable components for recognizing dogs, classifying colors, etc.). The resulting compound networks are jointly trained. We evaluate our approach on two challenging datasets for visual question answering, achieving state-of-the-art results on both the VQA natural image dataset and a new dataset of complex questions about abstract shapes.

归纳逻辑编程（ILP）是一种机器学习的形式。ILP的目标是诱导推广培训示例的假设（一组逻辑规则）。随着ILP转30，我们提供了对该领域的新介绍。我们介绍了必要的逻辑符号和主要学习环境;描述ILP系统的构建块;比较几个维度的几个系统;描述四个系统（Aleph，Tilde，Aspal和Metagol）;突出关键应用领域;最后，总结了未来研究的当前限制和方向。

我们考虑从示例中学习复合代数表达式语义的问题。结果是一个多功能框架，用于研究可以放入以下抽象形式中的学习任务：输入是部分代数$ \ alg $和一组有限的示例$（\ varphi_1，o_1），（\ varphi_2，o_2，o_2），\ ldots $，每个由代数项$ \ varphi_i $和一组对象〜$ o_i $组成。目的是在$ \ alg $中同时填写缺失的代数操作，并将每个$ \ varphi_i $的变量填充$ o_i $，以便优化条款的合并价值。我们通过案例研究在语法推理，图像学习和逻辑场景描述的基础中证明了该框架的适用性。

推理，学习和决策的整合是构建更多普通AI系统的关键。作为朝这个方向的一步，我们提出了一种新颖的神经逻辑架构，可以解决电感逻辑编程（ILP）和深增强学习（RL）问题。我们的体系结构通过分配权重来谓词而不是规则来定义一阶逻辑程序的受限但呈现的连续空间。因此，它是完全可分的，可以用梯度下降有效地培训。此外，在与演员批评算法的深度RL设置中，我们提出了一种新颖的高效评论家建筑。与ILP和RL问题的最先进方法相比，我们的命题实现了出色的性能，同时能够提供完全可解释的解决方案和更好地缩放，特别是在测试阶段。

我们考虑将机器学习模型相结合以执行更高级别的认知任务和明确规格的问题。我们提出了新的视觉歧视难题（VDP）的新问题，该问题需要找到可解释的歧视因子，这些歧视因子根据逻辑规范对图像进行分类。人类可以轻松解决这些难题，并提供强大，可验证和可解释的歧视者作为答案。我们提出了一个组成神经肌符号框架，该框架结合了一个神经网络，以检测对象和与符号学习者的关系，以发现可解释的歧视者。我们创建了涉及自然图像的大量VDP数据集，并表明与几种纯粹的神经方法相比，我们的神经成像框架表现出色。

Knowledge about space and time is necessary to solve problems in the physical world: An AI agent situated in the physical world and interacting with objects often needs to reason about positions of and relations between objects; and as soon as the agent plans its actions to solve a task, it needs to consider the temporal aspect (e.g., what actions to perform over time). Spatio-temporal knowledge, however, is required beyond interacting with the physical world, and is also often transferred to the abstract world of concepts through analogies and metaphors (e.g., "a threat that is hanging over our heads"). As spatial and temporal reasoning is ubiquitous, different attempts have been made to integrate this into AI systems. In the area of knowledge representation, spatial and temporal reasoning has been largely limited to modeling objects and relations and developing reasoning methods to verify statements about objects and relations. On the other hand, neural network researchers have tried to teach models to learn spatial relations from data with limited reasoning capabilities. Bridging the gap between these two approaches in a mutually beneficial way could allow us to tackle many complex real-world problems, such as natural language processing, visual question answering, and semantic image segmentation. In this chapter, we view this integration problem from the perspective of Neuro-Symbolic AI. Specifically, we propose a synergy between logical reasoning and machine learning that will be grounded on spatial and temporal knowledge. Describing some successful applications, remaining challenges, and evaluation datasets pertaining to this direction is the main topic of this contribution.

主张神经符号人工智能（NESY）断言，将深度学习与象征性推理相结合将导致AI更强大，而不是本身。像深度学习一样成功，人们普遍认为，即使我们最好的深度学习系统也不是很擅长抽象推理。而且，由于推理与语言密不可分，因此具有直觉的意义，即自然语言处理（NLP）将成为NESY特别适合的候选人。我们对实施NLP实施NESY的研究进行了结构化审查，目的是回答Nesy是否确实符合其承诺的问题：推理，分布概括，解释性，学习和从小数据的可转让性以及新的推理到新的域。我们研究了知识表示的影响，例如规则和语义网络，语言结构和关系结构，以及隐式或明确的推理是否有助于更高的承诺分数。我们发现，将逻辑编译到神经网络中的系统会导致满足最NESY的目标，而其他因素（例如知识表示或神经体系结构的类型）与实现目标没有明显的相关性。我们发现在推理的定义方式上，特别是与人类级别的推理有关的许多差异，这会影响有关模型架构的决策并推动结论，这些结论在整个研究中并不总是一致的。因此，我们倡导采取更加有条不紊的方法来应用人类推理的理论以及适当的基准的发展，我们希望这可以更好地理解该领域的进步。我们在GitHub上提供数据和代码以进行进一步分析。

Two approaches to AI, neural networks and symbolic systems, have been proven very successful for an array of AI problems. However, neither has been able to achieve the general reasoning ability required for human-like intelligence. It has been argued that this is due to inherent weaknesses in each approach. Luckily, these weaknesses appear to be complementary, with symbolic systems being adept at the kinds of things neural networks have trouble with and vice-versa. The field of neural-symbolic AI attempts to exploit this asymmetry by combining neural networks and symbolic AI into integrated systems. Often this has been done by encoding symbolic knowledge into neural networks. Unfortunately, although many different methods for this have been proposed, there is no common definition of an encoding to compare them. We seek to rectify this problem by introducing a semantic framework for neural-symbolic AI, which is then shown to be general enough to account for a large family of neural-symbolic systems. We provide a number of examples and proofs of the application of the framework to the neural encoding of various forms of knowledge representation and neural network. These, at first sight disparate approaches, are all shown to fall within the framework's formal definition of what we call semantic encoding for neural-symbolic AI.

通过归纳逻辑编程（ILP）综合大型逻辑程序通常需要中间定义。但是，用强化谓词混乱假设空间通常会降低性能。相比之下，梯度下降提供了一种有效的方法来在此类高维空间中找到溶液。到目前为止，神经符号ILP方法尚未完全利用这一点。我们提出了一种基于ILP的合成方法，该方法受益于大规模谓词发明，利用了高维梯度下降的功效。我们发现包含十个辅助定义以上的符号解决方案。这超出了现有的神经符号ILP系统的成就，因此构成了该领域的里程碑。