对象接地任务旨在通过口头通信定位图像中的目标对象。了解人类命令是有效人体机器人通信所需的重要过程。然而,这是具有挑战性的,因为人类命令可能是暧昧和错误的。本文旨在消除人类的引用表达式,允许代理基于从场景图获得的语义数据提出相关问题。我们测试如果我们的代理可以从场景图之间使用对象之间的关系,以便询问可以消除原始用户命令的语义相关问题。在本文中,我们使用场景图(IGSG)提出增量接地,消歧模型使用从图像场景图和语言场景图到基于人类命令的地面对象的语义数据的歧义模型。与基线相比,IGSG显示了有希望的成果,在有多个相同的目标对象的复杂现实场景中。 IGSG可以通过要求消除歧义问题回到用户来有效消除歧义或错误的表达式。
translated by 谷歌翻译
场景图是一个场景的结构化表示,可以清楚地表达场景中对象之间的对象,属性和关系。随着计算机视觉技术继续发展,只需检测和识别图像中的对象,人们不再满足。相反,人们期待着对视觉场景更高的理解和推理。例如,给定图像,我们希望不仅检测和识别图像中的对象,还要知道对象之间的关系(视觉关系检测),并基于图像内容生成文本描述(图像标题)。或者,我们可能希望机器告诉我们图像中的小女孩正在做什么(视觉问题应答(VQA)),甚至从图像中移除狗并找到类似的图像(图像编辑和检索)等。这些任务需要更高水平的图像视觉任务的理解和推理。场景图只是场景理解的强大工具。因此,场景图引起了大量研究人员的注意力,相关的研究往往是跨模型,复杂,快速发展的。然而,目前没有对场景图的相对系统的调查。为此,本调查对现行场景图研究进行了全面调查。更具体地说,我们首先总结了场景图的一般定义,随后对场景图(SGG)和SGG的发电方法进行了全面和系统的讨论,借助于先验知识。然后,我们调查了场景图的主要应用,并汇总了最常用的数据集。最后,我们对场景图的未来发展提供了一些见解。我们相信这将是未来研究场景图的一个非常有帮助的基础。
translated by 谷歌翻译
搭配机器人的效用在很大程度上取决于人类的简单和直观的相互作用机制。如果机器人在自然语言中接受任务指令,首先,它必须通过解码指令来了解用户的意图。然而,在执行任务时,由于观察到的场景的变化,机器人可能面临不可预见的情况,因此需要进一步的用户干预。在本文中,我们提出了一个称为谈话的系统,该系统使机器人能够通过在视觉上观察僵局来启动与教师的相干对话交换。通过对话,它要么在原始计划中找到一个提示,它是一个可接受的替代原始计划的替代方案,或者完全肯定地中止任务。为了实现可能的僵局,我们利用观察到的场景的密集标题和给定的指令,共同计算机器人的下一个动作。我们基于初始指令和情境场景对的数据集评估我们的系统。我们的系统可以识别僵局,并以适当的对话交换来解决82%的准确性。此外,与现有技术相比,用户学习表明,我们的系统的问题更自然(4.02平均为1到5的平均值)(平均3.08)。
translated by 谷歌翻译
Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in
translated by 谷歌翻译
机器人任务说明通常涉及机器人必须在环境中定位(地面)的引用对象。尽管任务意图理解是自然语言理解的重要组成部分,但努力却减少了解决任务时可能出现的歧义的努力。现有作品使用基于视觉的任务接地和歧义检测,适用于固定视图和静态机器人。但是,该问题对移动机器人进行了放大,其中未知的理想视图是未知的。此外,单个视图可能不足以定位给定区域中的所有对象实例,从而导致歧义检测不准确。只有机器人能够传达其面临的歧义,人类干预才能有所帮助。在本文中,我们介绍了doro(对对象的歧义),该系统可以帮助体现的代理在需要时提出合适的查询来消除引用对象的歧义。给定预期对象所处的区域,Doro通过在探索和扫描该区域的同时从多个视图中汇总观察结果来找到对象的所有实例。然后,它使用接地对象实例的信息提出合适的查询。使用AI2thor模拟器进行的实验表明,Doro不仅更准确地检测到歧义,而且还通过从视觉语言接地中获得了更准确的信息来提高冗长的查询。
translated by 谷歌翻译
深度学习技术导致了通用对象检测领域的显着突破,近年来产生了很多场景理解的任务。由于其强大的语义表示和应用于场景理解,场景图一直是研究的焦点。场景图生成(SGG)是指自动将图像映射到语义结构场景图中的任务,这需要正确标记检测到的对象及其关系。虽然这是一项具有挑战性的任务,但社区已经提出了许多SGG方法并取得了良好的效果。在本文中,我们对深度学习技术带来了近期成就的全面调查。我们审查了138个代表作品,涵盖了不同的输入方式,并系统地将现有的基于图像的SGG方法从特征提取和融合的角度进行了综述。我们试图通过全面的方式对现有的视觉关系检测方法进行连接和系统化现有的视觉关系检测方法,概述和解释SGG的机制和策略。最后,我们通过深入讨论当前存在的问题和未来的研究方向来完成这项调查。本调查将帮助读者更好地了解当前的研究状况和想法。
translated by 谷歌翻译
We propose a novel scene graph generation model called Graph R-CNN, that is both effective and efficient at detecting objects and their relations in images. Our model contains a Relation Proposal Network (RePN) that efficiently deals with the quadratic number of potential relations between objects in an image. We also propose an attentional Graph Convolutional Network (aGCN) that effectively captures contextual information between objects and relations. Finally, we introduce a new evaluation metric that is more holistic and realistic than existing metrics. We report state-of-the-art performance on scene graph generation as evaluated using both existing and our proposed metrics.
translated by 谷歌翻译
We introduce GQA, a new dataset for real-world visual reasoning and compositional question answering, seeking to address key shortcomings of previous VQA datasets. We have developed a strong and robust question engine that leverages Visual Genome scene graph structures to create 22M diverse reasoning questions, which all come with functional programs that represent their semantics. We use the programs to gain tight control over the answer distribution and present a new tunable smoothing technique to mitigate question biases. Accompanying the dataset is a suite of new metrics that evaluate essential qualities such as consistency, grounding and plausibility. A careful analysis is performed for baselines as well as state-of-the-art models, providing fine-grained results for different question types and topologies. Whereas a blind LSTM obtains a mere 42.1%, and strong VQA models achieve 54.1%, human performance tops at 89.3%, offering ample opportunity for new research to explore. We hope GQA will provide an enabling resource for the next generation of models with enhanced robustness, improved consistency, and deeper semantic understanding of vision and language.
translated by 谷歌翻译
Learning descriptive 3D features is crucial for understanding 3D scenes with diverse objects and complex structures. However, it is usually unknown whether important geometric attributes and scene context obtain enough emphasis in an end-to-end trained 3D scene understanding network. To guide 3D feature learning toward important geometric attributes and scene context, we explore the help of textual scene descriptions. Given some free-form descriptions paired with 3D scenes, we extract the knowledge regarding the object relationships and object attributes. We then inject the knowledge to 3D feature learning through three classification-based auxiliary tasks. This language-assisted training can be combined with modern object detection and instance segmentation methods to promote 3D semantic scene understanding, especially in a label-deficient regime. Moreover, the 3D feature learned with language assistance is better aligned with the language features, which can benefit various 3D-language multimodal tasks. Experiments on several benchmarks of 3D-only and 3D-language tasks demonstrate the effectiveness of our language-assisted 3D feature learning. Code is available at https://github.com/Asterisci/Language-Assisted-3D.
translated by 谷歌翻译
在图像理解项目中越来越多的情况下,场景图一代在电脑视觉研究中获得了很多关注,如视觉问题应答,图像标题,自动驾驶汽车,人群行为分析,活动识别等等。场景图,图像的视觉图形结构,非常有助于简化图像理解任务。在这项工作中,我们介绍了一个称为几何上下文的后处理算法,以了解视觉场景更好的几何上。我们使用该后处理算法在对象对与先前模型之间添加和改进几何关系。我们通过计算对象对之间的方向和距离来利用此上下文。我们使用知识嵌入式路由网络(KERN)作为我们的基准模型,将工作与我们的算法扩展,并显示最近最先进的算法上的可比结果。
translated by 谷歌翻译
服务机器人应该能够与非专家用户自然互动,不仅可以帮助他们完成各种任务,还可以接收指导,以解决指导中可能存在的歧义。我们考虑了视觉接地的任务,在这种情况下,代理将对象从拥挤的场景中分离出自然语言描述。现代的整体视觉接地方法通常忽略语言结构,而努力覆盖通用领域,因此很大程度上依靠大型数据集。此外,由于基准和目标域之间的高视觉差异,它们在RGB-D数据集中的传输性能受到了影响。模块化方法将学习与领域建模结合并利用语言的组成性质,以使视觉表示从语言解析中解脱出来,但由于缺乏强大的监督,要么依靠外部解析或以端到端的方式进行训练。在这项工作中,我们试图通过引入一个完全脱钩的模块化框架来解决这些局限性,以构成实体,属性和空间关系的组成视觉基础。我们利用在合成域中生成的丰富场景图表注释,并独立训练每个模块。我们的方法在模拟和两个真实的RGB-D场景数据集中进行了评估。实验结果表明,我们的框架的解耦性可以轻松地与域适应方法相结合,以实现SIMS到现实的视觉识别,从而为机器人应用中的视觉接地提供了数据效率,健壮且可解释的解决方案。
translated by 谷歌翻译
Humans refer to objects in their environments all the time, especially in dialogue with other people. We explore generating and comprehending natural language referring expressions for objects in images.In particular, we focus on incorporating better measures of visual context into referring expression models and find that visual comparison to other objects within an image helps improve performance significantly. We also develop methods to tie the language generation process together, so that we generate expressions for all objects of a particular category jointly. Evaluation on three recent datasets -RefCOCO, RefCOCO+, and RefCOCOg 1 , shows the advantages of our methods for both referring expression generation and comprehension.
translated by 谷歌翻译
3D场景理解是一个相对新兴的研究领域。在本文中,我们介绍了3D现实世界场景(VQA-3D)中的视觉问题应答任务,旨在给出3D场景的所有可能的问题。为了解决这个问题,提出了第一个VQA-3D数据集,即CLEVR3D,其中包含在1,129个现实世界场景中的60k个问题。具体而言,我们开发一个问题发动机利用3D场景图结构来生成不同的推理问题,涵盖物体属性的问题(即,大小,颜色和材料)及其空间关系。建立在此数据集之上,我们进一步设计了第一个VQA-3D基线模型TransVQA3D。 TransVQA3D型号采用精心设计的变压器架构,实现优越的VQA-3D性能,与纯语言基线和先前的3D推理方法直接应用于3D场景。实验结果验证了VQA-3D作为辅助任务可以提高3D场景理解的性能,包括节点明智分类和全图识别的场景图分析。
translated by 谷歌翻译
现有的研究解决场景图生成(SGG) - 图像中场景理解的关键技术 - 从检测角度,即使用边界框检测到对象,然后预测其成对关系。我们认为这种范式引起了几个阻碍该领域进步的问题。例如,当前数据集中的基于框的标签通常包含冗余类,例如头发,并遗漏对上下文理解至关重要的背景信息。在这项工作中,我们介绍了Panoptic场景图生成(PSG),这是一项新的问题任务,要求该模型基于全景分割而不是刚性边界框生成更全面的场景图表示。一个高质量的PSG数据集包含可可和视觉基因组的49k井被宣传的重叠图像,是为社区创建的,以跟踪其进度。为了进行基准测试,我们构建了四个两阶段基线,这些基线是根据SGG中的经典方法修改的,以及两个单阶段基准,称为PSGTR和PSGFORMER,它们基于基于高效的变压器检测器,即detr。虽然PSGTR使用一组查询来直接学习三重态,但PSGFormer以来自两个变压器解码器的查询形式分别模拟对象和关系,然后是一种迅速的关系 - 对象对象匹配机制。最后,我们分享了关于公开挑战和未来方向的见解。
translated by 谷歌翻译
Scene graph generation from images is a task of great interest to applications such as robotics, because graphs are the main way to represent knowledge about the world and regulate human-robot interactions in tasks such as Visual Question Answering (VQA). Unfortunately, its corresponding area of machine learning is still relatively in its infancy, and the solutions currently offered do not specialize well in concrete usage scenarios. Specifically, they do not take existing "expert" knowledge about the domain world into account; and that might indeed be necessary in order to provide the level of reliability demanded by the use case scenarios. In this paper, we propose an initial approximation to a framework called Ontology-Guided Scene Graph Generation (OG-SGG), that can improve the performance of an existing machine learning based scene graph generator using prior knowledge supplied in the form of an ontology (specifically, using the axioms defined within); and we present results evaluated on a specific scenario founded in telepresence robotics. These results show quantitative and qualitative improvements in the generated scene graphs.
translated by 谷歌翻译
感知,规划,估算和控制的当代方法允许机器人在不确定,非结构化环境中的远程代理中稳健运行。此进度现在创造了机器人不仅在隔离,而且在我们的复杂环境中运行的机器人。意识到这个机会需要一种高效且灵活的媒介,人类可以与协作机器人沟通。自然语言提供了一种这样的媒体,通过对自然语言理解的统计方法的重大进展,现在能够解释各种自由形式命令。然而,大多数当代方法需要机器人环境的详细,现有的空间语义地图,这些环境模拟了话语可能引用的可能引用的空间。因此,当机器人部署在新的,先前未知或部分观察到的环境中时,这些方法发生故障,特别是当环境的心理模型在人类运营商和机器人之间不同时。本文提供了一种新的学习框架的全面描述,允许现场和服务机器人解释并正确执行先验未知,非结构化环境中的自然语言指令。对于我们的方法而不是我们的语言作为“传感器” - 在话语中隐含的“传感器” - 推断的空间,拓扑和语义信息,然后利用这些信息来学习在潜在环境模型上的分布。我们将此分布纳入概率,语言接地模型中,并在机器人的动作空间的象征性表示中推断出分布。我们使用模仿学习来确定对环境和行为分布的原因的信仰空间政策。我们通过各种导航和移动操纵实验评估我们的框架。
translated by 谷歌翻译
This paper presents a framework for jointly grounding objects that follow certain semantic relationship constraints given in a scene graph. A typical natural scene contains several objects, often exhibiting visual relationships of varied complexities between them. These inter-object relationships provide strong contextual cues toward improving grounding performance compared to a traditional object query-only-based localization task. A scene graph is an efficient and structured way to represent all the objects and their semantic relationships in the image. In an attempt towards bridging these two modalities representing scenes and utilizing contextual information for improving object localization, we rigorously study the problem of grounding scene graphs on natural images. To this end, we propose a novel graph neural network-based approach referred to as Visio-Lingual Message PAssing Graph Neural Network (VL-MPAG Net). In VL-MPAG Net, we first construct a directed graph with object proposals as nodes and an edge between a pair of nodes representing a plausible relation between them. Then a three-step inter-graph and intra-graph message passing is performed to learn the context-dependent representation of the proposals and query objects. These object representations are used to score the proposals to generate object localization. The proposed method significantly outperforms the baselines on four public datasets.
translated by 谷歌翻译
当前的自动驾驶汽车技术主要集中于将乘客从A点带到B。但是,已经证明乘客害怕乘坐自动驾驶汽车。减轻此问题的一种方法是允许乘客给汽车提供自然语言命令。但是,汽车可能会误解发布的命令或视觉环境,这可能导致不确定的情况。希望自动驾驶汽车检测到这些情况并与乘客互动以解决它们。本文提出了一个模型,该模型检测到命令时不确定的情况并找到引起该命令的视觉对象。可选地,包括描述不确定对象的系统生成的问题。我们认为,如果汽车可以以人类的方式解释这些物体,乘客就可以对汽车能力获得更多信心。因此,我们研究了如何(1)检测不确定的情况及其根本原因,以及(2)如何为乘客产生澄清的问题。在对Talk2CAR数据集进行评估时,我们表明所提出的模型\ acrfull {pipeline},改善\ gls {m:模棱两可 - absolute-Increse},与$ iou _ {.5} $相比,与不使用\ gls {pipeline {pipeline {pipeline { }。此外,我们设计了一个引用表达生成器(reg)\ acrfull {reg_model}量身定制的自动驾驶汽车设置,该设置可产生\ gls {m:流星伴侣} Meteor的相对改进,\ gls \ gls {m:rouge felative}}与最先进的REG模型相比,Rouge-L的速度快三倍。
translated by 谷歌翻译
Understanding a visual scene goes beyond recognizing individual objects in isolation. Relationships between objects also constitute rich semantic information about the scene. In this work, we explicitly model the objects and their relationships using scene graphs, a visually-grounded graphical structure of an image. We propose a novel endto-end model that generates such structured scene representation from an input image. The model solves the scene graph inference problem using standard RNNs and learns to iteratively improves its predictions via message passing. Our joint inference model can take advantage of contextual cues to make better predictions on objects and their relationships. The experiments show that our model significantly outperforms previous methods for generating scene graphs using Visual Genome dataset and inferring support relations with NYU Depth v2 dataset.
translated by 谷歌翻译
From a visual scene containing multiple people, human is able to distinguish each individual given the context descriptions about what happened before, their mental/physical states or intentions, etc. Above ability heavily relies on human-centric commonsense knowledge and reasoning. For example, if asked to identify the "person who needs healing" in an image, we need to first know that they usually have injuries or suffering expressions, then find the corresponding visual clues before finally grounding the person. We present a new commonsense task, Human-centric Commonsense Grounding, that tests the models' ability to ground individuals given the context descriptions about what happened before, and their mental/physical states or intentions. We further create a benchmark, HumanCog, a dataset with 130k grounded commonsensical descriptions annotated on 67k images, covering diverse types of commonsense and visual scenes. We set up a context-object-aware method as a strong baseline that outperforms previous pre-trained and non-pretrained models. Further analysis demonstrates that rich visual commonsense and powerful integration of multi-modal commonsense are essential, which sheds light on future works. Data and code will be available https://github.com/Hxyou/HumanCog.
translated by 谷歌翻译