当前的自动驾驶汽车技术主要集中于将乘客从A点带到B。但是，已经证明乘客害怕乘坐自动驾驶汽车。减轻此问题的一种方法是允许乘客给汽车提供自然语言命令。但是，汽车可能会误解发布的命令或视觉环境，这可能导致不确定的情况。希望自动驾驶汽车检测到这些情况并与乘客互动以解决它们。本文提出了一个模型，该模型检测到命令时不确定的情况并找到引起该命令的视觉对象。可选地，包括描述不确定对象的系统生成的问题。我们认为，如果汽车可以以人类的方式解释这些物体，乘客就可以对汽车能力获得更多信心。因此，我们研究了如何（1）检测不确定的情况及其根本原因，以及（2）如何为乘客产生澄清的问题。在对Talk2CAR数据集进行评估时，我们表明所提出的模型\ acrfull {pipeline}，改善\ gls {m：模棱两可 - absolute-Increse}，与$ iou _ {.5} $相比，与不使用\ gls {pipeline {pipeline {pipeline { }。此外，我们设计了一个引用表达生成器（reg）\ acrfull {reg_model}量身定制的自动驾驶汽车设置，该设置可产生\ gls {m：流星伴侣} Meteor的相对改进，\ gls \ gls {m：rouge felative}}与最先进的REG模型相比，Rouge-L的速度快三倍。

近年来，我们在自动驾驶汽车的发展中看到了显着的步骤。多家公司开始推出令人印象深刻的系统，这些系统在各种环境中工作。这些系统有时可以给人一种印象，即完全自动驾驶即将到来，我们很快就会在没有方向盘的情况下建造汽车。给予AI的自主权和控制水平的增加为人道交互的新模式提供了机会。然而，调查表明，在自动驾驶汽车中对AI提供更多控制伴随着乘客的一定程度的不安。在尝试缓解这个问题时，最近的作品通过允许乘客提供指导到视觉场景中的特定对象的命令来采取自然语言的方法。尽管如此，这只是汽车的一半，因为汽车也应该了解该命令的物理目的地，这就是我们在本文中的专注。我们提出了一个扩展，其中我们向3D目的地注释了在执行给定的命令之后需要达到的3D目的地，并在预测该目的地位置进行多个不同的基线进行评估。此外，我们介绍一个胜过适用于这种特定设置的先前作品的模型。

Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in

We propose a method that can generate an unambiguous description (known as a referring expression) of a specific object or region in an image, and which can also comprehend or interpret such an expression to infer which object is being described. We show that our method outperforms previous methods that generate descriptions of objects without taking into account other potentially ambiguous objects in the scene. Our model is inspired by recent successes of deep learning methods for image captioning, but while image captioning is difficult to evaluate, our task allows for easy objective evaluation. We also present a new large-scale dataset for referring expressions, based on MS-COCO. We have released the dataset and a toolbox for visualization and evaluation, see https://github.com/ mjhucla/Google_Refexp_toolbox.

Humans refer to objects in their environments all the time, especially in dialogue with other people. We explore generating and comprehending natural language referring expressions for objects in images.In particular, we focus on incorporating better measures of visual context into referring expression models and find that visual comparison to other objects within an image helps improve performance significantly. We also develop methods to tie the language generation process together, so that we generate expressions for all objects of a particular category jointly. Evaluation on three recent datasets -RefCOCO, RefCOCO+, and RefCOCOg 1 , shows the advantages of our methods for both referring expression generation and comprehension.

Visual understanding goes well beyond object recognition. With one glance at an image, we can effortlessly imagine the world beyond the pixels: for instance, we can infer people's actions, goals, and mental states. While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world. We formalize this task as Visual Commonsense Reasoning. Given a challenging question about an image, a machine must answer correctly and then provide a rationale justifying its answer.Next, we introduce a new dataset, VCR, consisting of 290k multiple choice QA problems derived from 110k movie scenes. The key recipe for generating non-trivial and highquality problems at scale is Adversarial Matching, a new approach to transform rich annotations into multiple choice questions with minimal bias. Experimental results show that while humans find VCR easy (over 90% accuracy), state-of-the-art vision models struggle (∼45%).To move towards cognition-level understanding, we present a new reasoning engine, Recognition to Cognition Networks (R2C), that models the necessary layered inferences for grounding, contextualization, and reasoning. R2C helps narrow the gap between humans and machines (∼65%); still, the challenge is far from solved, and we provide analysis that suggests avenues for future work.

Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and topdown attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.

我们提出了一种新的四管齐下的方法，在文献中首次建立消防员的情境意识。我们构建了一系列深度学习框架，彼此之叠，以提高消防员在紧急首次响应设置中进行的救援任务的安全性，效率和成功完成。首先，我们使用深度卷积神经网络（CNN）系统，以实时地分类和识别来自热图像的感兴趣对象。接下来，我们将此CNN框架扩展了对象检测，跟踪，分割与掩码RCNN框架，以及具有多模级自然语言处理（NLP）框架的场景描述。第三，我们建立了一个深入的Q学习的代理，免受压力引起的迷失方向和焦虑，能够根据现场消防环境中观察和存储的事实来制定明确的导航决策。最后，我们使用了一种低计算无监督的学习技术，称为张量分解，在实时对异常检测进行有意义的特征提取。通过这些临时深度学习结构，我们建立了人工智能系统的骨干，用于消防员的情境意识。要将设计的系统带入消防员的使用，我们设计了一种物理结构，其中处理后的结果被用作创建增强现实的投入，这是一个能够建议他们所在地的消防员和周围的关键特征，这对救援操作至关重要在手头，以及路径规划功能，充当虚拟指南，以帮助迷彩的第一个响应者恢复安全。当组合时，这四种方法呈现了一种新颖的信息理解，转移和综合方法，这可能会大大提高消防员响应和功效，并降低寿命损失。

连接视觉和语言在生成智能中起着重要作用。因此，已经致力于图像标题的大型研究工作，即用句法和语义有意义的句子描述图像。从2015年开始，该任务通常通过由Visual Encoder组成的管道和文本生成的语言模型来解决任务。在这些年来，两种组件通过对象区域，属性，介绍多模态连接，完全关注方法和伯特早期融合策略的利用而显着发展。但是，无论令人印象深刻的结果，图像标题的研究还没有达到结论性答案。这项工作旨在提供图像标题方法的全面概述，从视觉编码和文本生成到培训策略，数据集和评估度量。在这方面，我们量化地比较了许多相关的最先进的方法来确定架构和培训策略中最有影响力的技术创新。此外，讨论了问题的许多变体及其开放挑战。这项工作的最终目标是作为理解现有文献的工具，并突出显示计算机视觉和自然语言处理的研究领域的未来方向可以找到最佳的协同作用。

搭配机器人的效用在很大程度上取决于人类的简单和直观的相互作用机制。如果机器人在自然语言中接受任务指令，首先，它必须通过解码指令来了解用户的意图。然而，在执行任务时，由于观察到的场景的变化，机器人可能面临不可预见的情况，因此需要进一步的用户干预。在本文中，我们提出了一个称为谈话的系统，该系统使机器人能够通过在视觉上观察僵局来启动与教师的相干对话交换。通过对话，它要么在原始计划中找到一个提示，它是一个可接受的替代原始计划的替代方案，或者完全肯定地中止任务。为了实现可能的僵局，我们利用观察到的场景的密集标题和给定的指令，共同计算机器人的下一个动作。我们基于初始指令和情境场景对的数据集评估我们的系统。我们的系统可以识别僵局，并以适当的对话交换来解决82％的准确性。此外，与现有技术相比，用户学习表明，我们的系统的问题更自然（4.02平均为1到5的平均值）（平均3.08）。

图像字幕是当前的研究任务，用于使用场景中的对象及其关系来描述图像内容。为了应对这项任务，使用了两个重要的研究领域，人为的视觉和自然语言处理。在图像字幕中，就像在任何计算智能任务中一样，性能指标对于知道方法的性能（或坏）至关重要。近年来，已经观察到，基于n-gram的经典指标不足以捕获语义和关键含义来描述图像中的内容。为了衡量或不进行最新指标的集合，在本手稿中，我们对使用众所周知的COCO数据集进行了对几种图像字幕指标的评估以及它们之间的比较。为此，我们设计了两种情况。 1）一组人工构建字幕，以及2）比较某些最先进的图像字幕方法的比较。我们试图回答问题：当前的指标是否有助于制作高质量的标题？实际指标如何相互比较？指标真正测量什么？

We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations.

The International Workshop on Reading Music Systems (WoRMS) is a workshop that tries to connect researchers who develop systems for reading music, such as in the field of Optical Music Recognition, with other researchers and practitioners that could benefit from such systems, like librarians or musicologists. The relevant topics of interest for the workshop include, but are not limited to: Music reading systems; Optical music recognition; Datasets and performance evaluation; Image processing on music scores; Writer identification; Authoring, editing, storing and presentation systems for music scores; Multi-modal systems; Novel input-methods for music to produce written music; Web-based Music Information Retrieval services; Applications and projects; Use-cases related to written music. These are the proceedings of the 3rd International Workshop on Reading Music Systems, held in Alicante on the 23rd of July 2021.

图像标题是自动生成句子的任务，以最好的方式生成描述输入图像。最近用于自动生成图像标题的最成功的技术最近使用了细心的深度学习模型。设计了深入学习模型的设计方式有变化。在本调查中，我们为图像标题的细心深度学习模型提供了相关的文献述评。而不是对深度图像标题模型的所有先前工作进行全面审查，我们解释了用于深度学习模型中的图像标题任务的各种类型的注意机制。用于图像标题的最成功的深度学习模型遵循编码器解码器架构，尽管这些模型采用注意机制的方式存在差异。通过分析图像标题的不同细节深层模型的性能结果，我们的目标是在图像标题中找到深度模型中最成功的注意机制。柔软的关注，自下而上的关注和多主题是一种广泛应用于图像标题的最先进的深度学习模型的关注机构的类型。在当前时，最佳结果是从多针关注的变体实现的，以自下而上的关注。

在本文中，我们试图通过引入深度学习模型的句法归纳偏见来建立两所学校之间的联系。我们提出了两个归纳偏见的家族，一个家庭用于选区结构，另一个用于依赖性结构。选区归纳偏见鼓励深度学习模型使用不同的单位（或神经元）分别处理长期和短期信息。这种分离为深度学习模型提供了一种方法，可以从顺序输入中构建潜在的层次表示形式，即更高级别的表示由高级表示形式组成，并且可以分解为一系列低级表示。例如，在不了解地面实际结构的情况下，我们提出的模型学会通过根据其句法结构组成变量和运算符的表示来处理逻辑表达。另一方面，依赖归纳偏置鼓励模型在输入序列中找到实体之间的潜在关系。对于自然语言，潜在关系通常被建模为一个定向依赖图，其中一个单词恰好具有一个父节点和零或几个孩子的节点。将此约束应用于类似变压器的模型之后，我们发现该模型能够诱导接近人类专家注释的有向图，并且在不同任务上也优于标准变压器模型。我们认为，这些实验结果为深度学习模型的未来发展展示了一个有趣的选择。

The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the chal-

The Flickr30k dataset has become a standard benchmark for sentence-based image description. This paper presents Flickr30k Entities, which augments the 158k captions from Flickr30k with 244k coreference chains, linking mentions of the same entities across different captions for the same image, and associating them with 276k manually annotated bounding boxes. Such annotations are essential for continued progress in automatic image description and grounded language understanding. They enable us to define a new benchmark for localization of textual entity mentions in an image. We present a strong baseline for this task that combines an image-text embedding, detectors for common objects, a color classifier, and a bias towards selecting larger objects. While our baseline rivals in accuracy more complex state-of-the-art models, we show that its gains cannot be easily parlayed into improvements on such tasks as image-sentence retrieval, thus underlining the limitations of current methods and the need for further research.

Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly "intelligent" behavior. Here we apply recent techniques for explaining decisions of state-of-the-art learning machines and analyze various tasks from computer vision and arcade games. This showcases a spectrum of problem-solving behaviors ranging from naive and short-sighted, to wellinformed and strategic. We observe that standard performance evaluation metrics can be oblivious to distinguishing these diverse problem solving behaviors. Furthermore, we propose our semi-automated Spectral Relevance Analysis that provides a practically effective way of characterizing and validating the behavior of nonlinear learning machines. This helps to assess whether a learned model indeed delivers reliably for the problem that it was conceived for. Furthermore, our work intends to add a voice of caution to the ongoing excitement about machine intelligence and pledges to evaluate and judge some of these recent successes in a more nuanced manner.

考虑到安全至关重要自动化系统中情境意识的功能，对驾驶场景的风险及其解释性的感知对于自主和合作驾驶特别重要。为了实现这一目标，本文提出了在驾驶场景中的共同风险定位的新研究方向及其作为自然语言描述的风险解释。由于缺乏标准基准，我们收集了一个大规模数据集，戏剧性（带有字幕模块的驾驶风险评估机制），该数据集由17,785个在日本东京收集的互动驾驶场景组成。我们的戏剧数据集适用于带有相关重要对象的驾驶风险的视频和对象级别的问题，以实现视觉字幕的目标，作为一种自由形式的语言描述，利用封闭式和开放式响应用于多层次问题，可以用来使用这些响应，可用于在驾驶场景中评估一系列视觉字幕功能。我们将这些数据提供给社区以进行进一步研究。使用戏剧，我们探索了在互动驾驶场景中的联合风险定位和字幕的多个方面。特别是，我们基准了各种多任务预测架构，并提供了关节风险定位和风险字幕的详细分析。数据集可在https://usa.honda-ri.com/drama上获得

视觉问题应答（VQA）任务利用视觉图像和语言分析来回回答图像的文本问题。它是一个流行的研究课题，在过去十年中越来越多的现实应用。本文介绍了我们最近对AliceMind-MMU的研究（阿里巴巴的编码器 - 解码器来自Damo Academy - 多媒体理解的机器智能实验室），其比人类在VQA上获得相似甚至略微更好的结果。这是通过系统地改善VQA流水线来实现的，包括：（1）具有全面的视觉和文本特征表示的预培训; （2）与学习参加的有效跨模型互动; （3）一个新颖的知识挖掘框架，具有专门的专业专家模块，适用于复杂的VQA任务。处理不同类型的视觉问题，需要具有相应的专业知识在提高我们的VQA架构的表现方面发挥着重要作用，这取决于人力水平。进行了广泛的实验和分析，以证明新的研究工作的有效性。