In this paper, we address referring expression comprehension: localizing an image region described by a natural language expression. While most recent work treats expressions as a single unit, we propose to decompose them into three modular components related to subject appearance, location, and relationship to other objects. This allows us to flexibly adapt to expressions containing different types of information in an end-to-end framework. In our model, which we call the Modular Attention Network (MAttNet), two types of attention are utilized: languagebased attention that learns the module weights as well as the word/phrase attention that each module should focus on; and visual attention that allows the subject and relationship modules to focus on relevant image components. Module weights combine scores from all three modules dynamically to output an overall score. Experiments show that MAttNet outperforms previous state-of-the-art methods by a large margin on both bounding-box-level and pixel-level comprehension tasks. Demo 1 and code 2 are provided.

弱监督的参考表达接地（REG）旨在将特定目标扎根于语言表达描述的图像中，同时缺乏目标和表达之间的对应关系。弱监督的REG存在两个主要问题。首先，缺乏区域级注释会引入建议和查询之间的歧义。其次，大多数以前的弱监督的REG方法忽略了指南的判别位置和上下文，从而在将目标与其他相同类别对象区分开时造成了困难。为了应对上述挑战，我们设计了实体增强的自适应重建网络（enail）。具体而言，赚取包括三个模块：实体增强，自适应接地和协作重建。在实体增强中，我们计算语义相似性作为监督选择候选建议。自适应接地可以在主题，位置和背景下以分层关注计算候选提案的排名评分。协作重建从三个角度衡量排名结果：自适应重建，语言重建和属性分类。自适应机制有助于减轻不同参考表达式的差异。五个数据集的实验表明，赚取胜于现有的最新方法。定性结果表明，提议的收入可以更好地处理特定类别的多个对象在一起的情况。

We propose a method that can generate an unambiguous description (known as a referring expression) of a specific object or region in an image, and which can also comprehend or interpret such an expression to infer which object is being described. We show that our method outperforms previous methods that generate descriptions of objects without taking into account other potentially ambiguous objects in the scene. Our model is inspired by recent successes of deep learning methods for image captioning, but while image captioning is difficult to evaluate, our task allows for easy objective evaluation. We also present a new large-scale dataset for referring expressions, based on MS-COCO. We have released the dataset and a toolbox for visualization and evaluation, see https://github.com/ mjhucla/Google_Refexp_toolbox.

基于文本的视频细分旨在通过用文本查询指定演员及其表演动作来细分视频序列中的演员。由于\ emph {emph {语义不对称}的问题，以前的方法无法根据演员及其动作以细粒度的方式将视频内容与文本查询对齐。 \ emph {语义不对称}意味着在多模式融合过程中包含不同量的语义信息。为了减轻这个问题，我们提出了一个新颖的演员和动作模块化网络，该网络将演员及其动作分别定位在两个单独的模块中。具体来说，我们首先从视频和文本查询中学习与参与者相关的内容，然后以对称方式匹配它们以定位目标管。目标管包含所需的参与者和动作，然后将其送入完全卷积的网络，以预测演员的分割掩模。我们的方法还建立了对象的关联，使其与所提出的时间建议聚合机制交叉多个框架。这使我们的方法能够有效地细分视频并保持预测的时间一致性。整个模型允许联合学习参与者的匹配和细分，并在A2D句子和J-HMDB句子数据集上实现单帧细分和完整视频细分的最新性能。

Humans refer to objects in their environments all the time, especially in dialogue with other people. We explore generating and comprehending natural language referring expressions for objects in images.In particular, we focus on incorporating better measures of visual context into referring expression models and find that visual comparison to other objects within an image helps improve performance significantly. We also develop methods to tie the language generation process together, so that we generate expressions for all objects of a particular category jointly. Evaluation on three recent datasets -RefCOCO, RefCOCO+, and RefCOCOg 1 , shows the advantages of our methods for both referring expression generation and comprehension.

参考表达理解（REC）是视觉推理中最重要的任务之一，该任务需要模型来检测自然语言表达式所引用的目标对象。在拟议的管道中，一阶段参考表达理解（OSREC）已成为主要趋势，因为它合并了区域建议和选择阶段。许多最先进的OSREC模型采用多跳的推理策略，因为单个表达式中经常提到一系列对象，该对象需要多跳的推理来分析语义关系。但是，这些模型的一个未解决的问题是，在推理之前需要预先定义和固定推理步骤的数量，而忽略了表达式的不同复杂性。在本文中，我们提出了一个动态的多步推理网络，该网络允许根据推理状态和表达复杂性动态调整推理步骤。具体而言，我们采用变压器模块来记住和处理推理状态和增强学习策略，以动态推断推理步骤。这项工作可在几个REC数据集上实现最新性能或重大改进，从具有短表达式的Refcoco（+，G）到具有长而复杂的组成表达式的数据集。

The prevailing framework for matching multimodal inputs is based on a two-stage process: 1) detecting proposals with an object detector and 2) matching text queries with proposals. Existing two-stage solutions mostly focus on the matching step. In this paper, we argue that these methods overlook an obvious \emph{mismatch} between the roles of proposals in the two stages: they generate proposals solely based on the detection confidence (i.e., query-agnostic), hoping that the proposals contain all instances mentioned in the text query (i.e., query-aware). Due to this mismatch, chances are that proposals relevant to the text query are suppressed during the filtering process, which in turn bounds the matching performance. To this end, we propose VL-NMS, which is the first method to yield query-aware proposals at the first stage. VL-NMS regards all mentioned instances as critical objects, and introduces a lightweight module to predict a score for aligning each proposal with a critical object. These scores can guide the NMS operation to filter out proposals irrelevant to the text query, increasing the recall of critical objects, resulting in a significantly improved matching performance. Since VL-NMS is agnostic to the matching step, it can be easily integrated into any state-of-the-art two-stage matching methods. We validate the effectiveness of VL-NMS on two multimodal matching tasks, namely referring expression grounding and image-text matching. Extensive ablation studies on several baselines and benchmarks consistently demonstrate the superiority of VL-NMS.

在本文中，我们研究了如何使用现代视觉语言变形金刚实现更好的视觉接地，并为这项具有挑战性的任务提出了一种简单而强大的选择性训练（SIRI）机制。特别是，Siri传达了视觉接地研究的重要原则，即更好的初始视觉语言编码器将帮助该模型收敛到更好的局部最低限度，从而相应地提高性能。具体而言，随着训练的进行，我们不断更新编码器的参数，而定期重新定位的其余参数则可以根据增强的编码来更好地优化模型。 Siri在三个流行的基准测试中可以大大优于以前的方法。具体而言，我们的方法在Refcoco+ Testa上达到了83.04％的TOP1精度，超过了最先进的方法（从头开始训练）超过10.21％。此外，我们透露，即使培训数据有限，Siri也表现出色。我们还将其扩展到基于变压器的视觉接地模型和其他视觉语言任务，以验证有效性。

深度学习技术导致了通用对象检测领域的显着突破，近年来产生了很多场景理解的任务。由于其强大的语义表示和应用于场景理解，场景图一直是研究的焦点。场景图生成（SGG）是指自动将图像映射到语义结构场景图中的任务，这需要正确标记检测到的对象及其关系。虽然这是一项具有挑战性的任务，但社区已经提出了许多SGG方法并取得了良好的效果。在本文中，我们对深度学习技术带来了近期成就的全面调查。我们审查了138个代表作品，涵盖了不同的输入方式，并系统地将现有的基于图像的SGG方法从特征提取和融合的角度进行了综述。我们试图通过全面的方式对现有的视觉关系检测方法进行连接和系统化现有的视觉关系检测方法，概述和解释SGG的机制和策略。最后，我们通过深入讨论当前存在的问题和未来的研究方向来完成这项调查。本调查将帮助读者更好地了解当前的研究状况和想法。

Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and topdown attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.

服务机器人应该能够与非专家用户自然互动，不仅可以帮助他们完成各种任务，还可以接收指导，以解决指导中可能存在的歧义。我们考虑了视觉接地的任务，在这种情况下，代理将对象从拥挤的场景中分离出自然语言描述。现代的整体视觉接地方法通常忽略语言结构，而努力覆盖通用领域，因此很大程度上依靠大型数据集。此外，由于基准和目标域之间的高视觉差异，它们在RGB-D数据集中的传输性能受到了影响。模块化方法将学习与领域建模结合并利用语言的组成性质，以使视觉表示从语言解析中解脱出来，但由于缺乏强大的监督，要么依靠外部解析或以端到端的方式进行训练。在这项工作中，我们试图通过引入一个完全脱钩的模块化框架来解决这些局限性，以构成实体，属性和空间关系的组成视觉基础。我们利用在合成域中生成的丰富场景图表注释，并独立训练每个模块。我们的方法在模拟和两个真实的RGB-D场景数据集中进行了评估。实验结果表明，我们的框架的解耦性可以轻松地与域适应方法相结合，以实现SIMS到现实的视觉识别，从而为机器人应用中的视觉接地提供了数据效率，健壮且可解释的解决方案。

人群的理解由于其重要的实际意义引起了人们对视觉领域的普遍兴趣。不幸的是，没有努力探索桥接自然语言和计算机视觉的多模式领域中的人群理解。参考表达理解（REF）是一项代表性的多模式任务。当前的REF研究更多地集中在一般情况下从多个独特类别中扎根目标对象。很难应用于复杂的现实世界人群的理解。为了填补这一空白，我们提出了一个新的挑战性数据集，称为Refcrowd，该数据集旨在通过参考表达方式寻找人群中的目标人。它不仅需要充分挖掘自然语言信息，而且还需要仔细地专注于目标与具有相似外观的人群之间的细微差异，以实现从语言到视觉的细粒度映射。此外，我们提出了一个细粒度的多模式属性对比网络（FMAC），以在人群的理解中处理参考。它首先将复杂的视觉和语言特征分解为属性感知的多模式特征，然后捕获歧视性但健壮性的细粒属性特征，以有效地区分相似人之间的这些细微差异。所提出的方法优于我们的档案数据集和现有参考数据集中的现有最新方法（SOTA）方法。此外，我们为多模式域中的更深入研究实施了端到端的REF工具箱。我们的数据集和代码可以在：\ url {https://qiuheqian.github.io/datasets/refcrowd/}中可用。

如何最好地将涉及语言和视觉的多模式任务中的语言和感知处理整合在一起是一个重要的开放问题。在这项工作中，我们认为，以自上而下的方式使用语言的常见做法，直接通过高级视觉特征进行视觉关注，这可能不是最佳的。我们假设使用语言还可以调节从像素到高级功能的自下而上处理可以为整体性能带来好处。为了支持我们的主张，我们提出了一个基于U-NET的模型，并对两个语言视觉密集预测任务进行实验：引用表达式分割和语言引导的图像着色。我们比较结果，其中一个或自下而上的视觉分支都以语言为条件。我们的实验表明，除了自上而下的注意力外，使用语言来控制自下而上的视觉处理，从而为任务和实现竞争性表现提供更好的结果。我们的语言分析表明，自下而上的条件改善对象的细分，尤其是当输入文本是指低级视觉概念时。代码可在https://github.com/ilkerkesen/bvpr上找到。

In this paper, we study the problem of image-text matching. Inferring the latent semantic alignment between objects or other salient stuff (e.g. snow, sky, lawn) and the corresponding words in sentences allows to capture fine-grained interplay between vision and language, and makes image-text matching more interpretable. Prior work either simply aggregates the similarity of all possible pairs of regions and words without attending differentially to more and less important words or regions, or uses a multi-step attentional process to capture limited number of semantic alignments which is less interpretable. In this paper, we present Stacked Cross Attention to discover the full latent alignments using both image regions and words in a sentence as context and infer image-text similarity. Our approach achieves the state-of-the-art results on the MS-COCO and Flickr30K datasets. On Flickr30K, our approach outperforms the current best methods by 22.1% relatively in text retrieval from image query, and 18.2% relatively in image retrieval with text query (based on Recall@1). On MS-COCO, our approach improves sentence retrieval by 17.8% relatively and image retrieval by 16.6% relatively (based on Recall@1 using the 5K test set). Code has been made available at: https: //github.com/kuanghuei/SCAN.

Learning descriptive 3D features is crucial for understanding 3D scenes with diverse objects and complex structures. However, it is usually unknown whether important geometric attributes and scene context obtain enough emphasis in an end-to-end trained 3D scene understanding network. To guide 3D feature learning toward important geometric attributes and scene context, we explore the help of textual scene descriptions. Given some free-form descriptions paired with 3D scenes, we extract the knowledge regarding the object relationships and object attributes. We then inject the knowledge to 3D feature learning through three classification-based auxiliary tasks. This language-assisted training can be combined with modern object detection and instance segmentation methods to promote 3D semantic scene understanding, especially in a label-deficient regime. Moreover, the 3D feature learned with language assistance is better aligned with the language features, which can benefit various 3D-language multimodal tasks. Experiments on several benchmarks of 3D-only and 3D-language tasks demonstrate the effectiveness of our language-assisted 3D feature learning. Code is available at https://github.com/Asterisci/Language-Assisted-3D.

The Flickr30k dataset has become a standard benchmark for sentence-based image description. This paper presents Flickr30k Entities, which augments the 158k captions from Flickr30k with 244k coreference chains, linking mentions of the same entities across different captions for the same image, and associating them with 276k manually annotated bounding boxes. Such annotations are essential for continued progress in automatic image description and grounded language understanding. They enable us to define a new benchmark for localization of textual entity mentions in an image. We present a strong baseline for this task that combines an image-text embedding, detectors for common objects, a color classifier, and a bias towards selecting larger objects. While our baseline rivals in accuracy more complex state-of-the-art models, we show that its gains cannot be easily parlayed into improvements on such tasks as image-sentence retrieval, thus underlining the limitations of current methods and the need for further research.

We propose a technique for producing 'visual explanations' for decisions from a large class of Convolutional Neural Network (CNN)-based models, making them more transparent and explainable.Our approach -Gradient-weighted Class Activation Mapping (Grad-CAM), uses the gradients of any target concept (say 'dog' in a classification network or a sequence of words in captioning network) flowing into the final convolutional layer to produce a coarse localization map highlighting the important regions in the image for predicting the concept.Unlike previous approaches, Grad-CAM is applicable to a wide variety of CNN model-families: (1) CNNs with fullyconnected layers (e.g. VGG), (2) CNNs used for structured outputs (e.g. captioning), (3) CNNs used in tasks with multimodal inputs (e.g. visual question answering) or reinforcement learning, all without architectural changes or re-training. We combine Grad-CAM with existing fine-grained visualizations to create a high-resolution class-discriminative vi-

我们建议对视觉模型预处理的基于利润的损失，以鼓励基于梯度的解释，这些解释与区域级注释一致。我们将该目标称为注意面罩的一致性（AMC），并证明它与依赖于区域级注释的模型相比，它产生了卓越的视觉接地性能，以显式训练对象检测器，例如更快的R-CNN。 AMC通过鼓励基于梯度的解释掩盖来工作，该掩盖的注意力分数主要集中在包含这种注释的图像的注释区域中。尤其是，在标准视觉建模目标之上接受AMC训练的模型在FlickR30K视觉接地基准中获得了86.59％的最新精度，与最佳先前模型相比，绝对改善了5.48％。我们的方法在既定的基准中都表现出表达理解，并通过设计基于梯度的解释来更好地与人类注释保持一致，从而提供了极大的表现。

Referring image segmentation aims to segment the target object described by a given natural language expression. Typically, referring expressions contain complex relationships between the target and its surrounding objects. The main challenge of this task is to understand the visual and linguistic content simultaneously and to find the referred object accurately among all instances in the image. Currently, the most effective way to solve the above problem is to obtain aligned multi-modal features by computing the correlation between visual and linguistic feature modalities under the supervision of the ground-truth mask. However, existing paradigms have difficulty in thoroughly understanding visual and linguistic content due to the inability to perceive information directly about surrounding objects that refer to the target. This prevents them from learning aligned multi-modal features, which leads to inaccurate segmentation. To address this issue, we present a position-aware contrastive alignment network (PCAN) to enhance the alignment of multi-modal features by guiding the interaction between vision and language through prior position information. Our PCAN consists of two modules: 1) Position Aware Module (PAM), which provides position information of all objects related to natural language descriptions, and 2) Contrastive Language Understanding Module (CLUM), which enhances multi-modal alignment by comparing the features of the referred object with those of related objects. Extensive experiments on three benchmarks demonstrate our PCAN performs favorably against the state-of-the-art methods. Our code will be made publicly available.

多模式变压器表现出高容量和灵活性，可将图像和文本对齐以进行视觉接地。然而，由于自我发挥操作的二次时间复杂性，仅编码的接地框架（例如，transvg）遭受了沉重的计算。为了解决这个问题，我们通过将整个接地过程解散为编码和解码阶段，提出了一种新的多模式变压器体系结构，以动态MDETR形成。关键观察是，图像中存在很高的空间冗余。因此，我们通过在加快视觉接地过程之前利用这种稀疏性来设计一种新的动态多模式变压器解码器。具体而言，我们的动态解码器由2D自适应采样模块和文本引导的解码模块组成。采样模块旨在通过预测参考点的偏移来选择这些信息补丁，而解码模块则可以通过在图像功能和文本功能之间执行交叉注意来提取接地对象信息。这两个模块也被堆叠起来，以逐渐弥合模态间隙，并迭代地完善接地对象的参考点，最终实现了视觉接地的目的。对五个基准测试的广泛实验表明，我们提出的动态MDETR实现了计算和准确性之间的竞争权衡。值得注意的是，在解码器中仅使用9％的特征点，我们可以降低〜44％的多模式变压器的GLOP，但仍然比仅编码器的对应物更高的精度。此外，为了验证其概括能力并扩展我们的动态MDETR，我们构建了第一个单级剪辑授权的视觉接地框架，并在这些基准测试中实现最先进的性能。