在过去的几年中，在文化遗产领域中使用深度学习和计算机视觉在文化遗产领域变得非常相关，其中包括有关音频智能指南，互动博物馆和增强现实的大量应用。所有这些技术都需要大量数据才能有效工作并对用户有用。在艺术品的背景下，专家在昂贵且耗时的过程中注释了此类数据。特别是，对于每件艺术品，必须收集艺术品和描述表的图像，以执行诸如视觉问题回答之类的常见任务。在本文中，我们提出了一种视觉问题回答的方法，该方法允许在运行时生成一个描述表，该表可用于回答有关艺术品的视觉和上下文问题，从而完全避免了图像和注释过程。为此，我们研究了使用GPT-3来生成描述用于艺术品，以分析通过字幕指标分析生成的描述的质量。最后，我们评估了视觉问答答案和字幕任务的性能。

连接视觉和语言在生成智能中起着重要作用。因此，已经致力于图像标题的大型研究工作，即用句法和语义有意义的句子描述图像。从2015年开始，该任务通常通过由Visual Encoder组成的管道和文本生成的语言模型来解决任务。在这些年来，两种组件通过对象区域，属性，介绍多模态连接，完全关注方法和伯特早期融合策略的利用而显着发展。但是，无论令人印象深刻的结果，图像标题的研究还没有达到结论性答案。这项工作旨在提供图像标题方法的全面概述，从视觉编码和文本生成到培训策略，数据集和评估度量。在这方面，我们量化地比较了许多相关的最先进的方法来确定架构和培训策略中最有影响力的技术创新。此外，讨论了问题的许多变体及其开放挑战。这项工作的最终目标是作为理解现有文献的工具，并突出显示计算机视觉和自然语言处理的研究领域的未来方向可以找到最佳的协同作用。

Large language models (LLMs) have demonstrated excellent zero-shot generalization to new language tasks. However, effective utilization of LLMs for zero-shot visual question-answering (VQA) remains challenging, primarily due to the modality disconnection and task disconnection between LLM and VQA task. End-to-end training on vision and language data may bridge the disconnections, but is inflexible and computationally expensive. To address this issue, we propose \emph{Img2Prompt}, a plug-and-play module that provides the prompts that can bridge the aforementioned modality and task disconnections, so that LLMs can perform zero-shot VQA tasks without end-to-end training. In order to provide such prompts, we further employ LLM-agnostic models to provide prompts that can describe image content and self-constructed question-answer pairs, which can effectively guide LLM to perform zero-shot VQA tasks. Img2Prompt offers the following benefits: 1) It can flexibly work with various LLMs to perform VQA. 2)~Without the needing of end-to-end training, it significantly reduces the cost of deploying LLM for zero-shot VQA tasks. 3) It achieves comparable or better performance than methods relying on end-to-end training. For example, we outperform Flamingo~\cite{Deepmind:Flamingo2022} by 5.6\% on VQAv2. On the challenging A-OKVQA dataset, our method even outperforms few-shot methods by as much as 20\%.

基于知识的视觉问题答案（VQA）涉及回答图像中不存在外部知识的问题。现有方法首先从外部资源中检索知识，然后通过所选知识，输入图像和答案预测的问题进行理性。但是，这种两步方法可能导致不匹配，可能会限制VQA性能。例如，检索到的知识可能与该问题无关紧要，并且在推理过程中重新安装的知识特征可能会偏离其在知识库中的最初含义（KB）。为了应对这一挑战，我们提出了PICA，这是一种简单而有效的方法，该方法通过使用图像字幕提示GPT3用于基于知识的VQA。受GPT-3在知识检索和问题答案中的力量的启发，而不是像以前的工作那样使用结构化的KB，而是将GPT-3视为一种隐式和非结构化的KB，可以共同获取和处理相关的知识。具体来说，我们首先将图像转换为GPT-3可以理解的标题（或标签），然后通过提供一些文字中的VQA示例来调整GPT-3以几个弹射方式解决VQA任务。我们通过仔细研究进一步提高绩效：（i）哪种文本格式最能描述图像内容，以及（ii）如何更好地选择和使用中文示例。 PICA解锁了GPT-3用于多模式任务的首次使用。通过仅使用16个示例，PICA超过了OK-VQA数据集上的绝对+8.6点。我们还在VQAV2上基准了PICA，PICA还显示出不错的表现。

人们说：“一张照片值一千字”。那么，我们如何从图像中获取丰富的信息？我们认为，通过使用视觉线索来桥接大型的识别视觉基础模型和语言模型，我们可以无需任何额外的跨模式训练。得益于基础模型的强大零拍功能，我们首先构建图像的丰富语义表示（例如，图像标签，对象属性 /位置，字幕）作为结构化的文本提示，称为视觉线索，使用视觉基础模型。基于视觉线索，我们使用大型语言模型为视觉内容生成一系列综合描述，然后再次通过视觉模型验证，以选择与图像最合适的候选人。我们通过定量和定性测量评估生成的描述的质量。结果证明了这种结构化语义表示的有效性。

We propose the task of free-form and open-ended Visual Question Answering (VQA). Given an image and a natural language question about the image, the task is to provide an accurate natural language answer. Mirroring real-world scenarios, such as helping the visually impaired, both the questions and answers are open-ended. Visual questions selectively target different areas of an image, including background details and underlying context. As a result, a system that succeeds at VQA typically needs a more detailed understanding of the image and complex reasoning than a system producing generic image captions. Moreover, VQA is amenable to automatic evaluation, since many open-ended answers contain only a few words or a closed set of answers that can be provided in a multiple-choice format. We provide a dataset containing ∼0.25M images, ∼0.76M questions, and ∼10M answers (www.visualqa.org), and discuss the information it provides. Numerous baselines and methods for VQA are provided and compared with human performance. Our VQA demo is available on CloudCV (http://cloudcv.org/vqa).

最近的文本到图像匹配模型对大型图像和句子的大公司进行了对比学习。虽然这些模型可以提供用于匹配和随后的零拍任务的强大分数，但它们不能给出给定图像的标题。在这项工作中，我们重新利用这些模型来生成在推理时间的图像时生成描述性文本，而无需进一步的训练或调整步骤。这是通过将具有大语言模型的视觉语义模型组合，从两种网络级模型中的知识中获益。由受监督标题方法获得的标题的限制性较小。此外，作为零射击学习方法，它非常灵活，我们展示了执行图像算法的能力，其中输入可以是图像或文本，输出是句子。这使得新颖的高级视觉能力，例如比较两个图像或解决视觉类比测试。

人类利用先验知识来描述图像，并能够使其解释适应特定的上下文信息，即使在上下文信息和图像不匹配时，也可以在发明合理的解释的范围内。在这项工作中，我们提出了通过整合上下文知识来字幕Wikipedia图像的新颖任务。具体而言，我们制作的模型共同推理了Wikipedia文章，Wikimedia图像及其相关描述以产生上下文化的标题。特别是，可以使用类似的Wikimedia图像来说明不同的文章，并且所产生的标题需要适应特定的上下文，因此使我们能够探索模型的限制以调整标题为不同的上下文信息。该领域中的一个特殊挑战性的任务是处理量不多的单词和命名实体。为了解决这个问题，我们提出了一个预训练目标，掩盖了命名实体建模（MNEM），并表明与基线模型相比，此借口任务可以改善。此外，我们验证了Wikipedia中使用MNEM目标预先训练的模型可以很好地推广到新闻字幕数据集。此外，我们根据字幕任务的难度定义了两种不同的测试拆分。我们提供有关每种方式的作用和重要性的见解，并突出我们模型的局限性。接受时，代码，模型和数据拆分可公开可用。

从预训练的语言模型中进行的引导已被证明是用于建立基础视觉模型（VLM）的有效方法，例如图像字幕或视觉问题的答案。但是，很难用它来使模型符合用户的理由来获得特定答案。为了引起和加强常识性原因，我们提出了一个迭代采样和调整范式，称为Illume，执行以下循环：给定图像问题提示提示，VLM采样了多个候选人，并通过人类评论家通过偏好提供最小的反馈。选择，用于微调。该循环增加了训练数据，并逐渐雕刻出VLM的合理化功能。我们的详尽实验表明，Illume在使用较少的培训数据的同时，仅需要最少的反馈，与标准监督的微调竞争。

图像标题是视觉语言理解的基本任务，其中模型将文本信息标题预测到给定输入图像。在本文中，我们提出了一种解决此任务的简单方法。我们使用剪辑编码作为标题的前缀，通过采用简单的映射网络，然后微调语言模型以生成图像标题。最近提出的剪辑模型包含丰富的语义特征，这些功能培训了文本背景，使其最适合视觉语言感知。我们的关键思想与预先接受训练的语言模型（GPT2）一起，我们获得了广泛了解视觉和文本数据。因此，我们的方法只需要相当快速的培训来产生称职的标题模型。如果没有额外的注释或预训练，它有效地为大规模和多样化的数据集生成有意义的标题。令人惊讶的是，即使仅在训练映射网络时，我们的方法也很好地运行良好，而剪辑和语言模型仍然冻结，则允许较轻的培训参数较轻的架构。通过定量评估，我们展示了我们的模型在充满挑战的概念标题和Nocaps数据集上实现了最先进的方法的可比结果，而它更简单，更快，更轻。我们的代码在https://github.com/rmokady/clip_prefix_caption中提供。

在传统的视觉问题（VQG）中，大多数图像具有多个概念（例如，对象和类别），可以生成问题，但培训模型以模仿培训数据中给出的任意选择概念。这使得训练困难并且还造成评估问题 - 对于大多数图像而言，存在多个有效问题，但人类参考资料只捕获一个或多个。我们呈现指导视觉问题 - VQG的变体，它根据对问题类型和应该探索的对象的期望来解决基于分类信息的问题生成器。我们提出了两个变体：（i）明确指导的模型，使演员（人机或自动化）能够选择哪些对象和类别来生成问题; （ii）基于离散潜在变量的基于离散潜变量，了解了一个隐式导游的模型，该模型将了解条件的哪些对象和类别。在答案类别增强VQA数据集上评估所提出的模型，我们的定量结果显示了对现有技术的大大改进（超过9bleu-4增加）。人类评估验证指导有助于生成语法相干的问题，并与给定的图像和对象相关。

变压器架构已经带来了计算语言领域的根本变化，这已经由经常性神经网络主导多年。它的成功还意味着具有语言和愿景的跨模型任务的大幅度变化，许多研究人员已经解决了这个问题。在本文中，我们审查了该领域中的一些最关键的里程碑，以及变压器架构如何纳入Visuol语言跨模型任务的整体趋势。此外，我们讨论了当前的局限性，并推测了我们发现迫在眉睫的一些前景。

图像字幕是当前的研究任务，用于使用场景中的对象及其关系来描述图像内容。为了应对这项任务，使用了两个重要的研究领域，人为的视觉和自然语言处理。在图像字幕中，就像在任何计算智能任务中一样，性能指标对于知道方法的性能（或坏）至关重要。近年来，已经观察到，基于n-gram的经典指标不足以捕获语义和关键含义来描述图像中的内容。为了衡量或不进行最新指标的集合，在本手稿中，我们对使用众所周知的COCO数据集进行了对几种图像字幕指标的评估以及它们之间的比较。为此，我们设计了两种情况。 1）一组人工构建字幕，以及2）比较某些最先进的图像字幕方法的比较。我们试图回答问题：当前的指标是否有助于制作高质量的标题？实际指标如何相互比较？指标真正测量什么？

Artificial Intelligence (AI) and its applications have sparked extraordinary interest in recent years. This achievement can be ascribed in part to advances in AI subfields including Machine Learning (ML), Computer Vision (CV), and Natural Language Processing (NLP). Deep learning, a sub-field of machine learning that employs artificial neural network concepts, has enabled the most rapid growth in these domains. The integration of vision and language has sparked a lot of attention as a result of this. The tasks have been created in such a way that they properly exemplify the concepts of deep learning. In this review paper, we provide a thorough and an extensive review of the state of the arts approaches, key models design principles and discuss existing datasets, methods, their problem formulation and evaluation measures for VQA and Visual reasoning tasks to understand vision and language representation learning. We also present some potential future paths in this field of research, with the hope that our study may generate new ideas and novel approaches to handle existing difficulties and develop new applications.

Many high-level skills that are required for computer vision tasks, such as parsing questions, comparing and contrasting semantics, and writing descriptions, are also required in other domains such as natural language processing. In this paper, we ask whether this makes it possible to learn those skills from text data and then use them to complete vision tasks without ever training on visual training data. Key to our approach is exploiting the joint embedding space of contrastively trained vision and language encoders. In practice, there can be systematic differences between embedding spaces for different modalities in contrastive models, and we analyze how these differences affect our approach and study a variety of strategies to mitigate this concern. We produce models using only text training data on three tasks: image captioning, visual entailment and visual question answering, and evaluate them on standard benchmarks using images. We find that this kind of transfer is possible and results in only a small drop in performance relative to models trained on images. We also showcase a variety of stylistic image captioning models that were trained using no image data and no human-curated language data, but instead text data from books, the web, or language models.

Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in

Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and topdown attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.

Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-based language models. This advancement has led to more fluent and coherent NLG, leading to improved development in downstream tasks such as abstractive summarization, dialogue generation and data-to-text generation. However, it is also apparent that deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios. To address this issue, many studies have been presented in measuring and mitigating hallucinated texts, but these have never been reviewed in a comprehensive manner before. In this survey, we thus provide a broad overview of the research progress and challenges in the hallucination problem in NLG. The survey is organized into two parts: (1) a general overview of metrics, mitigation methods, and future directions; and (2) an overview of task-specific research progress on hallucinations in the following downstream tasks, namely abstractive summarization, dialogue generation, generative question answering, data-to-text generation, machine translation, and visual-language generation. This survey serves to facilitate collaborative efforts among researchers in tackling the challenge of hallucinated texts in NLG.

The domain of joint vision-language understanding, especially in the context of reasoning in Visual Question Answering (VQA) models, has garnered significant attention in the recent past. While most of the existing VQA models focus on improving the accuracy of VQA, the way models arrive at an answer is oftentimes a black box. As a step towards making the VQA task more explainable and interpretable, our method is built upon the SOTA VQA framework by augmenting it with an end-to-end explanation generation module. In this paper, we investigate two network architectures, including Long Short-Term Memory (LSTM) and Transformer decoder, as the explanation generator. Our method generates human-readable textual explanations while maintaining SOTA VQA accuracy on the GQA-REX (77.49%) and VQA-E (71.48%) datasets. Approximately 65.16% of the generated explanations are approved by humans as valid. Roughly 60.5% of the generated explanations are valid and lead to the correct answers.

在回答问题时，人类会利用跨不同模式可用的信息来综合一致，完整的思想链（COT）。在深度学习模型（例如大规模语言模型）的情况下，这个过程通常是黑匣子。最近，科学问题基准已用于诊断AI系统的多跳推理能力和解释性。但是，现有数据集无法为答案提供注释，或仅限于仅文本模式，小尺度和有限的域多样性。为此，我们介绍了科学问题答案（SQA），这是一个新的基准，由〜21k的多模式多种选择问题组成，其中包含各种科学主题和答案的注释，并提供相应的讲座和解释。我们进一步设计语言模型，以学习将讲座和解释作为思想链（COT），以模仿回答SQA问题时的多跳上推理过程。 SQA在语言模型中展示了COT的实用性，因为COT将问题的答案绩效提高了1.20％的GPT-3和3.99％的unifiedqa。我们还探索了模型的上限，以通过喂食输入中的那些来利用解释；我们观察到它将GPT-3的少量性能提高了18.96％。我们的分析进一步表明，与人类类似的语言模型受益于解释，从较少的数据中学习并仅使用40％的数据实现相同的性能。