最近，几种技术旨在通过合并背景知识来提高场景图生成（SGG）的深度学习模型的性能。最先进的技术可以分为两个家庭：一个以潜在的方式将背景知识纳入模型，而另一种则以象征性形式保持背景知识。尽管有希望的结果，但两个技术家族都面临着几个缺点：第一个需要临时，更复杂的神经体系结构来增加培训或推理成本；第二个遭受有限的可伸缩性W.R.T.背景知识的大小。我们的工作引入了一种正则化技术，将符号背景知识注入神经SGG模型，以克服先前的艺术局限性。我们的技术是模型不合时宜的，在推理时间不会产生任何成本，并缩放到以前难以管理的背景知识规模。我们证明我们的技术可以提高最新SGG模型的准确性，最多可提高33％。

人工智能代理必须从周围环境中学到学习，并了解所学习的知识，以便做出决定。虽然从数据的最先进的学习通常使用子符号分布式表示，但是使用用于知识表示的一阶逻辑语言，推理通常在更高的抽象级别中有用。结果，将符号AI和神经计算结合成神经符号系统的尝试已经增加。在本文中，我们呈现了逻辑张量网络（LTN），一种神经组织形式和计算模型，通过引入许多值的端到端可分别的一阶逻辑来支持学习和推理，称为真实逻辑作为表示语言深入学习。我们表明LTN为规范提供了统一的语言，以及多个AI任务的计算，如数据聚类，多标签分类，关系学习，查询应答，半监督学习，回归和嵌入学习。我们使用TensorFlow2的许多简单的解释例实施和说明上述每个任务。关键词：神经组音恐怖症，深度学习和推理，许多值逻辑。

由Hong和Pavlic（2021）引入的单隐式层随机加权特征网络（RWFN）被开发为关系学习任务的神经张量网络方法的替代方案。其相对较小的占地面积结合使用了两个随机输入投影 - 一种昆虫 - 脑激发的输入表示和随机傅里叶特征 - 允许它以相对较低的培训成本实现有关关系的丰富表现力。特别是，当红和帕德奇比较RWFN到逻辑张量网络（LTNS）进行语义图像解释（SII）任务以提取图像的结构化语义描述，他们表明，两个隐藏的RWFN集成更好地捕获输入之间的关系具有更快的培训过程，即使它使用了更少的学习参数。在本文中，我们使用RWFN来执行视觉关系检测（VRD）任务，这些任务是更具挑战性的SII任务。零拍摄学习方法与RWFN一起使用，可以利用与其他所见关系和背景知识的相似性 - 以对象，关系和对象之间的逻辑约束表示 - 实现能够预测未出现在培训中的三维群体放。在视觉关系数据集上的实验，用于比较RWFN和LTNS之间的性能，其中一个领先的统计关系学习框架之一，显示RWFNS以谓词检测任务的销售胜过LTNS，同时使用较少数量的适应性参数（1:56比率）。此外，即使RWFNS的空间复杂性远小于LTNS（1:27比率），RWFN表示的背景技术也可用于减轻训练集的不完整性。

深度学习技术导致了通用对象检测领域的显着突破，近年来产生了很多场景理解的任务。由于其强大的语义表示和应用于场景理解，场景图一直是研究的焦点。场景图生成（SGG）是指自动将图像映射到语义结构场景图中的任务，这需要正确标记检测到的对象及其关系。虽然这是一项具有挑战性的任务，但社区已经提出了许多SGG方法并取得了良好的效果。在本文中，我们对深度学习技术带来了近期成就的全面调查。我们审查了138个代表作品，涵盖了不同的输入方式，并系统地将现有的基于图像的SGG方法从特征提取和融合的角度进行了综述。我们试图通过全面的方式对现有的视觉关系检测方法进行连接和系统化现有的视觉关系检测方法，概述和解释SGG的机制和策略。最后，我们通过深入讨论当前存在的问题和未来的研究方向来完成这项调查。本调查将帮助读者更好地了解当前的研究状况和想法。

Scene graph generation from images is a task of great interest to applications such as robotics, because graphs are the main way to represent knowledge about the world and regulate human-robot interactions in tasks such as Visual Question Answering (VQA). Unfortunately, its corresponding area of machine learning is still relatively in its infancy, and the solutions currently offered do not specialize well in concrete usage scenarios. Specifically, they do not take existing "expert" knowledge about the domain world into account; and that might indeed be necessary in order to provide the level of reliability demanded by the use case scenarios. In this paper, we propose an initial approximation to a framework called Ontology-Guided Scene Graph Generation (OG-SGG), that can improve the performance of an existing machine learning based scene graph generator using prior knowledge supplied in the form of an ontology (specifically, using the axioms defined within); and we present results evaluated on a specific scenario founded in telepresence robotics. These results show quantitative and qualitative improvements in the generated scene graphs.

我们提出了神经概率软逻辑（NEUPSL），这是一种新型的神经符号（NESY）框架，将最新的象征性推理与对深神经网络的低水平感知结合在一起。为了明确建模神经和符号表示之间的边界，我们引入了基于NESY Energy模型，这是一个结合神经和符号推理的基于能量的一般模型。使用此框架，我们展示了如何无缝整合神经和符号参数学习和推理。我们进行广泛的经验评估，并表明NEUPSL优于关节推断的现有方法，并且在几乎所有设置中的差异都显着降低。

Two approaches to AI, neural networks and symbolic systems, have been proven very successful for an array of AI problems. However, neither has been able to achieve the general reasoning ability required for human-like intelligence. It has been argued that this is due to inherent weaknesses in each approach. Luckily, these weaknesses appear to be complementary, with symbolic systems being adept at the kinds of things neural networks have trouble with and vice-versa. The field of neural-symbolic AI attempts to exploit this asymmetry by combining neural networks and symbolic AI into integrated systems. Often this has been done by encoding symbolic knowledge into neural networks. Unfortunately, although many different methods for this have been proposed, there is no common definition of an encoding to compare them. We seek to rectify this problem by introducing a semantic framework for neural-symbolic AI, which is then shown to be general enough to account for a large family of neural-symbolic systems. We provide a number of examples and proofs of the application of the framework to the neural encoding of various forms of knowledge representation and neural network. These, at first sight disparate approaches, are all shown to fall within the framework's formal definition of what we call semantic encoding for neural-symbolic AI.

场景图生成（SGG）旨在捕获对物体对之间的各种相互作用，这对于完整的场景了解至关重要。在整个关系集上培训的现有SGG方法未能由于培训数据中的各种偏差而导致视觉和文本相关性的复杂原理。学习表明像“ON”这样的通用空间配置的琐碎关系，而不是“停放”，例如“停放”，不执行这种复杂的推理，伤害泛化。为了解决这个问题，我们提出了一种新颖的SGG培训框架，以利用基于其信息的关系标签。我们的模型 - 不可知论培训程序对培训数据中的较少信息样本造成缺失的信息关系，并在算标签上培训算法的SGG模型以及现有的注释。我们表明，这种方法可以成功地与最先进的SGG方法结合使用，并在标准视觉基因组基准测试中显着提高它们的性能。此外，我们在更具挑战性的零射击设置中获得了看不见的三胞胎的相当大的改进。

尽管在现代的机器学习算法的最新进展，其内在机制的不透明仍是采用的障碍。在人工智能系统灌输信心和信任，解释的人工智能已成为提高现代机器学习算法explainability的响应。归纳逻辑程序（ILP），符号人工智能的子场中，起着产生，因为它的直观的逻辑驱动框架的可解释的解释有希望的作用。 ILP有效利用绎推理产生从实例和背景知识解释的一阶分句理论。然而，在发展中通过ILP需要启发方法的几个挑战，在实践中他们的成功应用来解决。例如，现有的ILP系统通常拥有广阔的解空间，以及感应解决方案是对噪声和干扰非常敏感。本次调查总结在ILP的最新进展和统计关系学习和神经象征算法的讨论，其中提供给ILP协同意见。继最新进展的严格审查，我们划定观察的挑战，突出对发展不言自明的人工智能系统进一步ILP动机研究的潜在途径。

The deployment of Deep Learning (DL) models is still precluded in those contexts where the amount of supervised data is limited. To answer this issue, active learning strategies aim at minimizing the amount of labelled data required to train a DL model. Most active strategies are based on uncertain sample selection, and even often restricted to samples lying close to the decision boundary. These techniques are theoretically sound, but an understanding of the selected samples based on their content is not straightforward, further driving non-experts to consider DL as a black-box. For the first time, here we propose a different approach, taking into consideration common domain-knowledge and enabling non-expert users to train a model with fewer samples. In our Knowledge-driven Active Learning (KAL) framework, rule-based knowledge is converted into logic constraints and their violation is checked as a natural guide for sample selection. We show that even simple relationships among data and output classes offer a way to spot predictions for which the model need supervision. The proposed approach (i) outperforms many active learning strategies in terms of average F1 score, particularly in those contexts where domain knowledge is rich. Furthermore, we empirically demonstrate that (ii) KAL discovers data distribution lying far from the initial training data unlike uncertainty-based strategies, (iii) it ensures domain experts that the provided knowledge is respected by the model on test data, and (iv) it can be employed even when domain-knowledge is not available by coupling it with a XAI technique. Finally, we also show that KAL is also suitable for object recognition tasks and, its computational demand is low, unlike many recent active learning strategies.

我们提出了一种有效的可解释的神经象征模型来解决感应逻辑编程（ILP）问题。在该模型中，该模型是由在分层结构中组织的一组元规则构建的，通过学习嵌入来匹配元规则的事实和身体谓词来发明一阶规则。为了实例化它，我们专门设计了一种表现型通用元规则集，并证明了它们产生的喇叭条件的片段。在培训期间，我们注入了控制的\ PW {gumbel}噪声以避免本地最佳，并采用可解释性 - 正则化术语来进一步指导融合到可解释规则。我们在针对几种最先进的方法上证明我们对各种任务（ILP，视觉基因组，强化学习）的模型进行了验证。

Recent scene graph generation (SGG) frameworks have focused on learning complex relationships among multiple objects in an image. Thanks to the nature of the message passing neural network (MPNN) that models high-order interactions between objects and their neighboring objects, they are dominant representation learning modules for SGG. However, existing MPNN-based frameworks assume the scene graph as a homogeneous graph, which restricts the context-awareness of visual relations between objects. That is, they overlook the fact that the relations tend to be highly dependent on the objects with which the relations are associated. In this paper, we propose an unbiased heterogeneous scene graph generation (HetSGG) framework that captures relation-aware context using message passing neural networks. We devise a novel message passing layer, called relation-aware message passing neural network (RMP), that aggregates the contextual information of an image considering the predicate type between objects. Our extensive evaluations demonstrate that HetSGG outperforms state-of-the-art methods, especially outperforming on tail predicate classes.

Today's scene graph generation (SGG) task is still far from practical, mainly due to the severe training bias, e.g., collapsing diverse human walk on/ sit on/lay on beach into human on beach. Given such SGG, the down-stream tasks such as VQA can hardly infer better scene structures than merely a bag of objects. However, debiasing in SGG is not trivial because traditional debiasing methods cannot distinguish between the good and bad bias, e.g., good context prior (e.g., person read book rather than eat) and bad long-tailed bias (e.g., near dominating behind/in front of). In this paper, we present a novel SGG framework based on causal inference but not the conventional likelihood. We first build a causal graph for SGG, and perform traditional biased training with the graph. Then, we propose to draw the counterfactual causality from the trained graph to infer the effect from the bad bias, which should be removed. In particular, we use Total Direct Effect as the proposed final predicate score for unbiased SGG. Note that our framework is agnostic to any SGG model and thus can be widely applied in the community who seeks unbiased predictions. By using the proposed Scene Graph Diagnosis toolkit 1 on the SGG benchmark Visual Genome and several prevailing models, we observed significant improvements over the previous state-of-the-art methods.

当前独立于域的经典计划者需要问题域和实例作为输入的符号模型，从而导致知识采集瓶颈。同时，尽管深度学习在许多领域都取得了重大成功，但知识是在与符号系统（例如计划者）不兼容的亚符号表示中编码的。我们提出了Latplan，这是一种无监督的建筑，结合了深度学习和经典计划。只有一组未标记的图像对，显示了环境中允许的过渡子集（训练输入），Latplan学习了环境的完整命题PDDL动作模型。稍后，当给出代表初始状态和目标状态（计划输入）的一对图像时，Latplan在符号潜在空间中找到了目标状态的计划，并返回可视化的计划执行。我们使用6个计划域的基于图像的版本来评估LATPLAN：8个插头，15个式嘴，Blockworld，Sokoban和两个LightsOut的变体。

同一场景中的不同对象彼此之间或多或少相关，但是只有有限数量的这些关系值得注意。受到对象检测效果的DETR的启发，我们将场景图生成视为集合预测问题，并提出了具有编码器decoder架构的端到端场景图生成模型RELTR。关于视觉特征上下文的编码器原因是，解码器使用带有耦合主题和对象查询的不同类型的注意机制渗透了一组固定大小的三胞胎主题prodicate-object。我们设计了一套预测损失，以执行地面真相与预测三胞胎之间的匹配。与大多数现有场景图生成方法相反，Reltr是一种单阶段方法，它仅使用视觉外观直接预测一组关系，而无需结合实体并标记所有可能的谓词。视觉基因组和开放图像V6数据集的广泛实验证明了我们模型的出色性能和快速推断。

We study the problem of combining neural networks with symbolic reasoning. Recently introduced frameworks for Probabilistic Neurosymbolic Learning (PNL), such as DeepProbLog, perform exponential-time exact inference, limiting the scalability of PNL solutions. We introduce Approximate Neurosymbolic Inference (A-NeSI): a new framework for PNL that uses neural networks for scalable approximate inference. A-NeSI 1) performs approximate inference in polynomial time without changing the semantics of probabilistic logics; 2) is trained using data generated by the background knowledge; 3) can generate symbolic explanations of predictions; and 4) can guarantee the satisfaction of logical constraints at test time, which is vital in safety-critical applications. Our experiments show that A-NeSI is the first end-to-end method to scale the Multi-digit MNISTAdd benchmark to sums of 15 MNIST digits, up from 4 in competing systems. Finally, our experiments show that A-NeSI achieves explainability and safety without a penalty in performance.

场景图是一个场景的结构化表示，可以清楚地表达场景中对象之间的对象，属性和关系。随着计算机视觉技术继续发展，只需检测和识别图像中的对象，人们不再满足。相反，人们期待着对视觉场景更高的理解和推理。例如，给定图像，我们希望不仅检测和识别图像中的对象，还要知道对象之间的关系（视觉关系检测），并基于图像内容生成文本描述（图像标题）。或者，我们可能希望机器告诉我们图像中的小女孩正在做什么（视觉问题应答（VQA）），甚至从图像中移除狗并找到类似的图像（图像编辑和检索）等。这些任务需要更高水平的图像视觉任务的理解和推理。场景图只是场景理解的强大工具。因此，场景图引起了大量研究人员的注意力，相关的研究往往是跨模型，复杂，快速发展的。然而，目前没有对场景图的相对系统的调查。为此，本调查对现行场景图研究进行了全面调查。更具体地说，我们首先总结了场景图的一般定义，随后对场景图（SGG）和SGG的发电方法进行了全面和系统的讨论，借助于先验知识。然后，我们调查了场景图的主要应用，并汇总了最常用的数据集。最后，我们对场景图的未来发展提供了一些见解。我们相信这将是未来研究场景图的一个非常有帮助的基础。

Deep learning models, though having achieved great success in many different fields over the past years, are usually data hungry, fail to perform well on unseen samples, and lack of interpretability. Various prior knowledge often exists in the target domain and their use can alleviate the deficiencies with deep learning. To better mimic the behavior of human brains, different advanced methods have been proposed to identify domain knowledge and integrate it into deep models for data-efficient, generalizable, and interpretable deep learning, which we refer to as knowledge-augmented deep learning (KADL). In this survey, we define the concept of KADL, and introduce its three major tasks, i.e., knowledge identification, knowledge representation, and knowledge integration. Different from existing surveys that are focused on a specific type of knowledge, we provide a broad and complete taxonomy of domain knowledge and its representations. Based on our taxonomy, we provide a systematic review of existing techniques, different from existing works that survey integration approaches agnostic to taxonomy of knowledge. This survey subsumes existing works and offers a bird's-eye view of research in the general area of knowledge-augmented deep learning. The thorough and critical reviews of numerous papers help not only understand current progresses but also identify future directions for the research on knowledge-augmented deep learning.

Visual relationships capture a wide variety of interactions between pairs of objects in images (e.g. "man riding bicycle" and "man pushing bicycle"). Consequently, the set of possible relationships is extremely large and it is difficult to obtain sufficient training examples for all possible relationships. Because of this limitation, previous work on visual relationship detection has concentrated on predicting only a handful of relationships. Though most relationships are infrequent, their objects (e.g. "man" and "bicycle") and predicates (e.g. "riding" and "pushing") independently occur more frequently. We propose a model that uses this insight to train visual models for objects and predicates individually and later combines them together to predict multiple relationships per image. We improve on prior work by leveraging language priors from semantic word embeddings to finetune the likelihood of a predicted relationship. Our model can scale to predict thousands of types of relationships from a few examples. Additionally, we localize the objects in the predicted relationships as bounding boxes in the image. We further demonstrate that understanding relationships can improve content based image retrieval.

场景图生成（SGG）任务旨在在给定图像中检测所有对象及其成对的视觉关系。尽管SGG在过去几年中取得了显着的进展，但几乎所有现有的SGG模型都遵循相同的训练范式：他们将SGG中的对象和谓词分类视为单标签分类问题，而地面真实性是一个hot目标。标签。但是，这种普遍的训练范式忽略了当前SGG数据集的两个特征：1）对于正样本，某些特定的主题对象实例可能具有多个合理的谓词。 2）对于负样本，有许多缺失的注释。不管这两个特征如何，SGG模型都很容易被混淆并做出错误的预测。为此，我们为无偏SGG提出了一种新颖的模型不合命相的标签语义知识蒸馏（LS-KD）。具体而言，LS-KD通过将预测的标签语义分布（LSD）与其原始的单热目标标签融合来动态生成每个主题对象实例的软标签。 LSD反映了此实例和多个谓词类别之间的相关性。同时，我们提出了两种不同的策略来预测LSD：迭代自我KD和同步自我KD。大量的消融和对三项SGG任务的结果证明了我们所提出的LS-KD的优势和普遍性，这些LS-KD可以始终如一地实现不同谓词类别之间的不错的权衡绩效。