Multi-modal and multi-hop question answering aims to answer a question based on multiple input sources from different modalities. Previous methods retrieve the evidence separately and feed the retrieved evidence to a language model to generate the corresponding answer. However, these methods fail to build connections between candidates and thus cannot model the inter-dependent relation during retrieval. Moreover, the reasoning process over multi-modality candidates can be unbalanced without building alignments between different modalities. To address this limitation, we propose a Structured Knowledge and Unified Retrieval Generation based method (SKURG). We align the sources from different modalities via the shared entities and map them into a shared semantic space via structured knowledge. Then, we utilize a unified retrieval-generation decoder to integrate intermediate retrieval results for answer generation and adaptively determine the number of retrieval steps. We perform experiments on two multi-modal and multi-hop datasets: WebQA and MultimodalQA. The results demonstrate that SKURG achieves state-of-the-art performance on both retrieval and answer generation.

In this paper, we propose an end-to-end Retrieval-Augmented Visual Language Model (REVEAL) that learns to encode world knowledge into a large-scale memory, and to retrieve from it to answer knowledge-intensive queries. REVEAL consists of four key components: the memory, the encoder, the retriever and the generator. The large-scale memory encodes various sources of multimodal world knowledge (e.g. image-text pairs, question answering pairs, knowledge graph triplets, etc) via a unified encoder. The retriever finds the most relevant knowledge entries in the memory, and the generator fuses the retrieved knowledge with the input query to produce the output. A key novelty in our approach is that the memory, encoder, retriever and generator are all pre-trained end-to-end on a massive amount of data. Furthermore, our approach can use a diverse set of multimodal knowledge sources, which is shown to result in significant gains. We show that REVEAL achieves state-of-the-art results on visual question answering and image captioning.

最近与大型变压器的主要工作的主要重点是优化包装到模型参数中的信息量。在这项工作中，我们问了一个不同的问题：多峰变压器可以在他们推理中利用明确的知识吗？现有，主要是单峰，方法在知识检索范例下探讨了方法，随后回答预测，但留下了关于所使用的检索知识的质量和相关性的开放性问题，以及如何集成隐含和明确知识的推理过程。为了解决这些挑战，我们提出了一种新颖的模型 - 知识增强变压器（KAT） - 在OK-VQA的开放式多模式任务上实现了强大的最先进的结果（+6分）。我们的方法在结束到终端编码器 - 解码器架构中集成了隐式和显式知识，同时在答案生成期间仍然共同推理了两个知识源。在我们分析中提高了模型预测的可解释性，可以看到明确知识集成的额外好处。

与自然语言解释的视觉结合旨在推断文本图像对之间的关​​系并生成句子以解释决策过程。先前的方法主要依靠预先训练的视觉模型来执行关系推断和语言模型来生成相应的解释。但是，预训练的视觉模型主要在文本和图像之间建立令牌级别的对齐，但忽略了短语（块）和视觉内容之间的高级语义对齐，这对于视觉推理至关重要。此外，仅基于编码的联合表示形式的解释生成器并未明确考虑关键的关系推理的决策点。因此，产生的解释不太忠于视觉语言推理。为了减轻这些问题，我们提出了一种统一的块意见对齐和基于词汇约束的方法，称为CALEC。它包含一个块感知的语义交互器（ARR。CSI），一个关系属性和词汇约束感知的发生器（arr。Lecg）。具体而言，CSI利用语言和各个图像区域固有的句子结构来构建块感知语义对齐。关系下属使用基于注意力的推理网络来合并令牌级别和块级视觉语言表示。 LECG利用词汇约束来将关系下列者重点关注的单词或块纳入解释世代，从而提高了解释的忠诚和信息性。我们在三个数据集上进行了广泛的实验，实验结果表明，CALEC在推理准确性和生成的解释的质量方面显着优于其他竞争者模型。

Question Answering (QA) is a task that entails reasoning over natural language contexts, and many relevant works augment language models (LMs) with graph neural networks (GNNs) to encode the Knowledge Graph (KG) information. However, most existing GNN-based modules for QA do not take advantage of rich relational information of KGs and depend on limited information interaction between the LM and the KG. To address these issues, we propose Question Answering Transformer (QAT), which is designed to jointly reason over language and graphs with respect to entity relations in a unified manner. Specifically, QAT constructs Meta-Path tokens, which learn relation-centric embeddings based on diverse structural and semantic relations. Then, our Relation-Aware Self-Attention module comprehensively integrates different modalities via the Cross-Modal Relative Position Bias, which guides information exchange between relevant entities of different modalities. We validate the effectiveness of QAT on commonsense question answering datasets like CommonsenseQA and OpenBookQA, and on a medical question answering dataset, MedQA-USMLE. On all the datasets, our method achieves state-of-the-art performance. Our code is available at http://github.com/mlvlab/QAT.

视觉问题回答（VQA）通常需要对视觉概念和语言语义的理解，这取决于外部知识。大多数现有方法利用了预训练的语言模型或/和非结构化文本，但是这些资源中的知识通常不完整且嘈杂。有些方法更喜欢使用经常具有强化结构知识的知识图（kgs），但是研究仍然相当初步。在本文中，我们提出了Lako，这是一种知识驱动的VQA方法，通过后期的文本注射。为了有效地纳入外部kg，我们将三元三元转移到文本中，并提出一种晚期注射机制。最后，我们将VQA作为文本生成任务，并具有有效的编码器范式。在使用OKVQA数据集的评估中，我们的方法可实现最新的结果。

现有的kg增强模型用于问题回答主要专注于设计精心图形神经网络（GNN）以模拟知识图（KG）。但是，它们忽略了（i）有效地融合和推理过问题上下文表示和kg表示，并且（ii）在推理期间自动从嘈杂的KG中选择相关节点。在本文中，我们提出了一种新颖的型号，其通过LMS和GNN的联合推理和动态KGS修剪机制解决了上述限制。具体而言，ConntLK通过新的密集双向注意模块在LMS和GNN之间执行联合推理，其中每个问题令牌参加KG节点，每个KG节点都会参加问题令牌，并且两个模态表示熔断和通过多次熔断和更新。步互动。然后，动态修剪模块使用通过联合推理产生的注意重量来递归修剪无关的kg节点。我们在CommanSENSEQA和OpenBookQA数据集上的结果表明，我们的模态融合和知识修剪方法可以更好地利用相关知识来推理。

基于多模式方面的情感分类（MABSC）是一项新兴的分类任务，旨在将给定目标的情感分类，例如具有不同模式的数据中提到的实体。在带有文本和图像的典型多模式数据中，以前的方法不能充分利用图像的细颗粒语义，尤其是与文本的语义结合在一起，并且不完全考虑对细粒图像之间的关系进行建模信息和目标，这导致图像的使用不足和不足以识别细粒度的方面和意见。为了应对这些局限性，我们提出了一个新的框架SEQCSG，包括一种构建顺序跨模式语义图和编码器模型的方法。具体而言，我们从原始图像，图像标题和场景图中提取细粒度的信息，并将它们视为跨模式语义图的元素以及文本的令牌。跨模式语义图表示为具有多模式可见矩阵的序列，指示元素之间的关系。为了有效地利用跨模式语义图，我们建议使用目标提示模板的编码器解码器方法。实验结果表明，我们的方法优于现有方法，并在两个标准数据集MABSC上实现了最新方法。进一步的分析证明了每个组件的有效性，我们的模型可以隐含地学习图像的目标和细粒度信息之间的相关性。

使用从预先接受训练的语言模型（LMS）和知识图表（LMS）和知识图表（kgs）回答问题的问题提出了两个挑战：给定QA上下文（问答选择），方法需要（i）从大型千克识别相关知识，（ii）对QA上下文和kg进行联合推理。在这项工作中，我们提出了一种新的模型，QA-GNN，它通过两个关键创新解决了上述挑战：（i）相关评分，我们使用LMS来估计KG节点相对于给定的QA上下文的重要性，以及（ii）联合推理，我们将QA上下文和kg连接到联合图，并通过图形神经网络相互更新它们的表示。我们评估了QA基准的模型（CommanSeaseQA，OpenBookQA）和生物医学（MedQa-USMLE）域名。QA-GNN优于现有的LM和LM + kg模型，并表现出可解释和结构化推理的能力，例如，正确处理问题的否定。

语言模型（LM）是否可以通过固有的关系推理能力在知识库中的地面问题解决方案（QA）任务？尽管以前仅使用LMS的模型在许多质量检查任务上都看到了一些成功，但最新的方法包括知识图（KG），以补充LMS的逻辑驱动的隐式知识。但是，有效从结构化数据（例如KGS）中提取信息，使LMS保持开放性问题，并且当前模型依靠图形技术来提取知识。在本文中，我们建议仅利用LMS将基于知识的问题的语言和知识与灵活性，覆盖范围和结构化推理相结合。具体而言，我们设计了一种知识构建方法，该方法可以通过动态跳跃来检索相关背景，该方法比传统的基于GNN的技术表达了更全面的。我们设计了一种深层融合机制，以进一步弥合语言和知识之间交换瓶颈的信息。广泛的实验表明，我们的模型始终证明了其对CommenSensenSENSENSESQA基准测试的最先进性能，从而展示了仅利用LMS将LMS稳健地质量质量质量质量质量固定到知识库的可能性。

Complex knowledge base question answering can be achieved by converting questions into sequences of predefined actions. However, there is a significant semantic and structural gap between natural language and action sequences, which makes this conversion difficult. In this paper, we introduce an alignment-enhanced complex question answering framework, called ALCQA, which mitigates this gap through question-to-action alignment and question-to-question alignment. We train a question rewriting model to align the question and each action, and utilize a pretrained language model to implicitly align the question and KG artifacts. Moreover, considering that similar questions correspond to similar action sequences, we retrieve top-k similar question-answer pairs at the inference stage through question-to-question alignment and propose a novel reward-guided action sequence selection strategy to select from candidate action sequences. We conduct experiments on CQA and WQSP datasets, and the results show that our approach outperforms state-of-the-art methods and obtains a 9.88\% improvements in the F1 metric on CQA dataset. Our source code is available at https://github.com/TTTTTTTTy/ALCQA.

我们在这项研究中的目标是研究一个更现实的环境，在这种环境中，我们可以为细粒度的产品类别进行弱监督的多模式实例级产品检索。我们首先贡献了product1m数据集，并定义了两个实际实例级检索任务，以实现价格比较和个性化建议的评估。对于两个实例级任务，如何准确地指出视觉语言数据中提到的产品目标并有效地降低了无关紧要的内容的影响非常具有挑战性。为了解决这个问题，我们利用训练一个更有效的跨模式与模型，该模型能够自适应地能够通过使用一个实体图，其节点和边缘分别表示实体和相似性，从而可以从多模式数据中合并来自多模式数据的关键概念信息。实体。具体而言，为实例级别的商品检索提出了一种新型的实体图增强的跨模式预处理（EGE-CMP）模型，该模型明确地将基于节点的基于节点的基于节点和子图的方式显式地注入实体知识。自我监管的混合流变压器可以减少不同对象内容之间的混淆，从而有效地指导网络专注于具有真实语义的实体。实验结果很好地验证了我们的EGE-CMP的功效和概括性，表现优于几个SOTA跨模式基线，例如夹子，Uniter和Capture。

目前用于开放域问题的最先进的生成模型（ODQA）专注于从非结构化文本信息生成直接答案。但是，大量的世界知识存储在结构化数据库中，并且需要使用SQL等查询语言访问。此外，查询语言可以回答需要复杂推理的问题，以及提供完全的解释性。在本文中，我们提出了一个混合框架，将文本和表格证据占据了输入，并根据哪种形式更好地回答这个问题而生成直接答案或SQL查询。然后可以在关联的数据库上执行生成的SQL查询以获得最终答案。据我们所知，这是第一种将Text2SQL与ODQA任务应用于ODQA任务的论文。凭经验，我们证明，在几个ODQA数据集上，混合方法始终如一地优于仅采用大边缘的均匀输入的基线模型。具体地，我们使用T5基础模型实现OpenSquad数据集的最先进的性能。在一个详细的分析中，我们证明能够生成结构的SQL查询可以始终带来增益，特别是对于那些需要复杂推理的问题。

Multi-modal named entity recognition (NER) and relation extraction (RE) aim to leverage relevant image information to improve the performance of NER and RE. Most existing efforts largely focused on directly extracting potentially useful information from images (such as pixel-level features, identified objects, and associated captions). However, such extraction processes may not be knowledge aware, resulting in information that may not be highly relevant. In this paper, we propose a novel Multi-modal Retrieval based framework (MoRe). MoRe contains a text retrieval module and an image-based retrieval module, which retrieve related knowledge of the input text and image in the knowledge corpus respectively. Next, the retrieval results are sent to the textual and visual models respectively for predictions. Finally, a Mixture of Experts (MoE) module combines the predictions from the two models to make the final decision. Our experiments show that both our textual model and visual model can achieve state-of-the-art performance on four multi-modal NER datasets and one multi-modal RE dataset. With MoE, the model performance can be further improved and our analysis demonstrates the benefits of integrating both textual and visual cues for such tasks.

多跳的推理需要汇总多个文档来回答一个复杂的问题。现有方法通常将多跳问题分解为更简单的单跳问题，以解决说明可解释的推理过程的问题。但是，他们忽略了每个推理步骤的支持事实的基础，这往往会产生不准确的分解。在本文中，我们提出了一个可解释的逐步推理框架，以在每个中间步骤中同时合并单跳支持句子识别和单跳问题生成，并利用当前跳跃的推断，直到推理最终结果。我们采用统一的读者模型来进行中级跳跃推理和最终的跳跃推理，并采用关节优化，以更准确，强大的多跳上推理。我们在两个基准数据集HOTPOTQA和2WIKIMULTIHOPQA上进行实验。结果表明，我们的方法可以有效地提高性能，并在不分解监督的情况下产生更好的解释推理过程。

医学视觉和语言预训练（MED-VLP）由于适用于从医学图像和文本中提取通用视觉和语言表示的适用性而受到了相当大的关注。大多数现有方法主要包含三个元素：Uni-Modal编码器（即视觉编码器和语言编码器），多模式融合模块以及借口任务，很少有研究考虑医疗领域专家知识的重要性，并明确利用此类此类此类此类此类。知识以促进Med-vlp。尽管在通用域中存在具有知识增强的视觉和语言预训练（VLP）方法，但大多数人都需要现成的工具包（例如，对象检测器和场景图解析器），这些工具包在医疗领域中是不可用的。在本文中，我们提出了一种系统有效的方法，从三个角度通过结构化医学知识来增强MED-VLP。首先，考虑知识可以被视为视觉和语言之间的中间媒介，我们通过知识对齐视觉编码器和语言编码器的表示。其次，我们将知识注入多模式融合模型，以使模型能够使用知识作为补充输入图像和文本进行推理。第三，我们指导该模型通过设计知识引起的借口任务来强调图像和文本中最关键的信息。为了进行全面的评估并促进进一步的研究，我们构建了包括三个任务的医学视觉和语言基准。实验结果说明了我们方法的有效性，在所有下游任务上都实现了最先进的性能。进一步的分析探讨了我们方法的不同组成部分和预训练的各种环境的影响。

预训练的语言模型（PTLM）已显示出在自然语言任务上表现良好。许多先前的作品都以通过知识图（KGS）标记的关系链接的实体的形式利用结构性常识来协助PTLM。检索方法使用kg作为单独的静态模块，该模块限制了覆盖范围，因为kgs包含有限的知识。生成方法训练PTLMS kg三倍以提高获得知识的规模。但是，对符号KG实体的培训限制了其在涉及自然语言文本的任务中的适用性，在这些任务中，它们忽略了整体上下文。为了减轻这种情况，我们提出了一个以句子为条件的常识性上下文化器（COSE-CO）作为输入，以使其在生成与输入文本的整体上下文相关的任务中通常可用。为了训练Cose-Co，我们提出了一个新的数据集，其中包括句子和常识知识对。 COSE-CO推断出的知识是多种多样的，并且包含了基础KG中不存在的新实体。我们增强了在多选质量质量检查和开放式常识性推理任务中产生的知识，从而改善了CSQA，ARC，QASC和OBQA数据集的当前最佳方法。我们还展示了其在改善释义生成任务的基线模型方面的适用性。

Video-language pre-training has advanced the performance of various downstream video-language tasks. However, most previous methods directly inherit or adapt typical image-language pre-training paradigms to video-language pre-training, thus not fully exploiting the unique characteristic of video, i.e., temporal. In this paper, we propose a Hierarchical Temporal-Aware video-language pre-training framework, HiTeA, with two novel pre-training tasks for modeling cross-modal alignment between moments and texts as well as the temporal relations of video-text pairs. Specifically, we propose a cross-modal moment exploration task to explore moments in videos, which results in detailed video moment representation. Besides, the inherent temporal relations are captured by aligning video-text pairs as a whole in different time resolutions with multi-modal temporal relation exploration task. Furthermore, we introduce the shuffling test to evaluate the temporal reliance of datasets and video-language pre-training models. We achieve state-of-the-art results on 15 well-established video-language understanding and generation tasks, especially on temporal-oriented datasets (e.g., SSv2-Template and SSv2-Label) with 8.6% and 11.1% improvement respectively. HiTeA also demonstrates strong generalization ability when directly transferred to downstream tasks in a zero-shot manner. Models and demo will be available on ModelScope.

Analogical reasoning is fundamental to human cognition and holds an important place in various fields. However, previous studies mainly focus on single-modal analogical reasoning and ignore taking advantage of structure knowledge. Notably, the research in cognitive psychology has demonstrated that information from multimodal sources always brings more powerful cognitive transfer than single modality sources. To this end, we introduce the new task of multimodal analogical reasoning over knowledge graphs, which requires multimodal reasoning ability with the help of background knowledge. Specifically, we construct a Multimodal Analogical Reasoning dataSet (MARS) and a multimodal knowledge graph MarKG. We evaluate with multimodal knowledge graph embedding and pre-trained Transformer baselines, illustrating the potential challenges of the proposed task. We further propose a novel model-agnostic Multimodal analogical reasoning framework with Transformer (MarT) motivated by the structure mapping theory, which can obtain better performance.

Multi-hop Question Answering over Knowledge Graph~(KGQA) aims to find the answer entities that are multiple hops away from the topic entities mentioned in a natural language question on a large-scale Knowledge Graph (KG). To cope with the vast search space, existing work usually adopts a two-stage approach: it firstly retrieves a relatively small subgraph related to the question and then performs the reasoning on the subgraph to accurately find the answer entities. Although these two stages are highly related, previous work employs very different technical solutions for developing the retrieval and reasoning models, neglecting their relatedness in task essence. In this paper, we propose UniKGQA, a novel approach for multi-hop KGQA task, by unifying retrieval and reasoning in both model architecture and parameter learning. For model architecture, UniKGQA consists of a semantic matching module based on a pre-trained language model~(PLM) for question-relation semantic matching, and a matching information propagation module to propagate the matching information along the edges on KGs. For parameter learning, we design a shared pre-training task based on question-relation matching for both retrieval and reasoning models, and then propose retrieval- and reasoning-oriented fine-tuning strategies. Compared with previous studies, our approach is more unified, tightly relating the retrieval and reasoning stages. Extensive experiments on three benchmark datasets have demonstrated the effectiveness of our method on the multi-hop KGQA task. Our codes and data are publicly available at https://github.com/RUCAIBox/UniKGQA.