Generative Knowledge Graph Construction (KGC) refers to those methods that leverage the sequence-to-sequence framework for building knowledge graphs, which is flexible and can be adapted to widespread tasks. In this study, we summarize the recent compelling progress in generative knowledge graph construction. We present the advantages and weaknesses of each paradigm in terms of different generation targets and provide theoretical insight and empirical analysis. Based on the review, we suggest promising research directions for the future. Our contributions are threefold: (1) We present a detailed, complete taxonomy for the generative KGC methods; (2) We provide a theoretical and empirical analysis of the generative KGC methods; (3) We propose several research directions that can be developed in the future.

三重提取是自然语言处理和知识图构建信息提取的重要任务。在本文中，我们重新审视了序列生成的端到端三重提取任务。由于生成三重提取可能难以捕获长期依赖性并产生不忠的三元组，因此我们引入了一种新型模型，即与生成变压器的对比度三重提取。具体而言，我们为基于编码器的生成引入了一个共享的变压器模块。为了产生忠实的结果，我们提出了一个新颖的三胞胎对比训练对象。此外，我们引入了两种机制，以进一步提高模型性能（即，批处理动态注意力掩盖和三个方面的校准）。在三个数据集（即NYT，WebNLG和MIE）上进行的实验结果表明，我们的方法比基线的方法更好。

关系提取是一项重要但具有挑战性的任务，旨在从文本中提取所有隐藏的关系事实。随着深层语言模型的发展，关系提取方法在各种基准上都取得了良好的性能。但是，我们观察到以前方法的两个缺点：首先，在各种关系提取设置下没有统一的框架可以很好地工作；其次，有效利用外部知识作为背景信息。在这项工作中，我们提出了一种知识增强的生成模型来减轻这两个问题。我们的生成模型是一个统一的框架，可在各种关系提取设置下依次生成关系三胞胎，并明确利用来自知识图（KG）的相关知识来解决歧义。我们的模型在包括WebNLG，NYT10和Tacred在内的多个基准和设置上实现了卓越的性能。

旨在从非结构化文本中提取结构信息的知识提取（KE）通常会遭受数据稀缺性和新出现的看不见类型，即低资源场景。许多低资源KE的神经方法已广泛研究并取得了令人印象深刻的表现。在本文中，我们在低资源场景中介绍了对KE的文献综述，并将现有作品分为三个范式：（1）利用更高的资源数据，（2）利用更强的模型，（3）利用数据和模型一起。此外，我们描述了有前途的应用，并概述了未来研究的一些潜在方向。我们希望我们的调查能够帮助学术和工业界更好地理解这一领域，激发更多的想法并提高更广泛的应用。

In recent years, there is a surge of generation-based information extraction work, which allows a more direct use of pre-trained language models and efficiently captures output dependencies. However, previous generative methods using lexical representation do not naturally fit document-level relation extraction (DocRE) where there are multiple entities and relational facts. In this paper, we investigate the root cause of the underwhelming performance of the existing generative DocRE models and discover that the culprit is the inadequacy of the training paradigm, instead of the capacities of the models. We propose to generate a symbolic and ordered sequence from the relation matrix which is deterministic and easier for model to learn. Moreover, we design a parallel row generation method to process overlong target sequences. Besides, we introduce several negative sampling strategies to improve the performance with balanced signals. Experimental results on four datasets show that our proposed method can improve the performance of the generative DocRE models. We have released our code at https://github.com/ayyyq/DORE.

Natural Language Processing (NLP) has been revolutionized by the use of Pre-trained Language Models (PLMs) such as BERT. Despite setting new records in nearly every NLP task, PLMs still face a number of challenges including poor interpretability, weak reasoning capability, and the need for a lot of expensive annotated data when applied to downstream tasks. By integrating external knowledge into PLMs, \textit{\underline{K}nowledge-\underline{E}nhanced \underline{P}re-trained \underline{L}anguage \underline{M}odels} (KEPLMs) have the potential to overcome the above-mentioned limitations. In this paper, we examine KEPLMs systematically through a series of studies. Specifically, we outline the common types and different formats of knowledge to be integrated into KEPLMs, detail the existing methods for building and evaluating KEPLMS, present the applications of KEPLMs in downstream tasks, and discuss the future research directions. Researchers will benefit from this survey by gaining a quick and comprehensive overview of the latest developments in this field.

Knowledge graphs (KG) have served as the key component of various natural language processing applications. Commonsense knowledge graphs (CKG) are a special type of KG, where entities and relations are composed of free-form text. However, previous works in KG completion and CKG completion suffer from long-tail relations and newly-added relations which do not have many know triples for training. In light of this, few-shot KG completion (FKGC), which requires the strengths of graph representation learning and few-shot learning, has been proposed to challenge the problem of limited annotated data. In this paper, we comprehensively survey previous attempts on such tasks in the form of a series of methods and applications. Specifically, we first introduce FKGC challenges, commonly used KGs, and CKGs. Then we systematically categorize and summarize existing works in terms of the type of KGs and the methods. Finally, we present applications of FKGC models on prediction tasks in different areas and share our thoughts on future research directions of FKGC.

Pre-trained Language Models (PLMs) which are trained on large text corpus through the self-supervised learning method, have yielded promising performance on various tasks in Natural Language Processing (NLP). However, though PLMs with huge parameters can effectively possess rich knowledge learned from massive training text and benefit downstream tasks at the fine-tuning stage, they still have some limitations such as poor reasoning ability due to the lack of external knowledge. Incorporating knowledge into PLMs has been tried to tackle these issues. In this paper, we present a comprehensive review of Knowledge-Enhanced Pre-trained Language Models (KE-PLMs) to provide a clear insight into this thriving field. We introduce appropriate taxonomies respectively for Natural Language Understanding (NLU) and Natural Language Generation (NLG) to highlight the focus of these two kinds of tasks. For NLU, we take several types of knowledge into account and divide them into four categories: linguistic knowledge, text knowledge, knowledge graph (KG), and rule knowledge. The KE-PLMs for NLG are categorized into KG-based and retrieval-based methods. Finally, we point out some promising future directions of KE-PLMs.

Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that are more natural and better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the lower level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks which may require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize CTG techniques from the perspective of PLMs. We hope it can help researchers in related fields to quickly track the academic frontier, providing them with a landscape of the area and a roadmap for future research.

使用诸如BERT，ELMO和FLAIR等模型建模上下文信息的成立具有显着改善了文字的表示学习。它还给出了几乎每个NLP任务机器翻译，文本摘要和命名实体识别的Sota结果，以命名为少。在这项工作中，除了使用这些主导的上下文感知的表示之外，我们还提出了一种用于命名实体识别（NER）的知识意识表示学习（KARL）网络。我们讨论了利用现有方法在纳入世界知识方面的挑战，并展示了如何利用我们所提出的方法来克服这些挑战。 KARL基于变压器编码器，该变压器编码器利用表示为事实三元组的大知识库，将它们转换为图形上下文，并提取驻留在内部的基本实体信息以生成用于特征增强的上下文化三联表示。实验结果表明，使用卡尔的增强可以大大提升我们的内部系统的性能，并在三个公共网络数据集中的文献中的现有方法，即Conll 2003，Conll ++和Ontonotes V5实现了比文献中现有方法的显着更好的结果。我们还观察到更好的概括和应用于从Karl上看不见的实体的真实环境。

即使预训练的语言模型共享语义编码器，自然语言的理解也遭受了各种输出模式的影响。在本文中，我们提出了基于BERT框架的统一双向语言理解模型Ubert，它可以通过Biaffine网络普遍地对不同NLU任务的训练对象进行建模。具体而言，Ubert从各个方面编码先验知识，统一地构建了多个NLU任务的学习表示，这有利于增强捕获共同语义理解的能力。使用Biaffine来模拟原始文本的开始和末端位置对，可以将各种分类和提取结构转换为通用的跨度编码方法。实验表明，UBERT在7个NLU任务，14个数据集和零拍设置上实现了最先进的性能，并实现了广泛的信息提取和语言推理任务的统一。

Triplet extraction aims to extract entities and their corresponding relations in unstructured text. Most existing methods train an extraction model on high-quality training data, and hence are incapable of extracting relations that were not observed during training. Generalizing the model to unseen relations typically requires fine-tuning on synthetic training data which is often noisy and unreliable. In this paper, we argue that reducing triplet extraction to a template filling task over a pre-trained language model can equip the model with zero-shot learning capabilities and enable it to leverage the implicit knowledge in the language model. Embodying these ideas, we propose a novel framework, ZETT (ZEro-shot Triplet extraction by Template infilling), that is based on end-to-end generative transformers. Our experiments show that without any data augmentation or pipeline systems, ZETT can outperform previous state-of-the-art models with 25% less parameters. We further show that ZETT is more robust in detecting entities and can be incorporated with automatically generated templates for relations.

Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-based language models. This advancement has led to more fluent and coherent NLG, leading to improved development in downstream tasks such as abstractive summarization, dialogue generation and data-to-text generation. However, it is also apparent that deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios. To address this issue, many studies have been presented in measuring and mitigating hallucinated texts, but these have never been reviewed in a comprehensive manner before. In this survey, we thus provide a broad overview of the research progress and challenges in the hallucination problem in NLG. The survey is organized into two parts: (1) a general overview of metrics, mitigation methods, and future directions; and (2) an overview of task-specific research progress on hallucinations in the following downstream tasks, namely abstractive summarization, dialogue generation, generative question answering, data-to-text generation, machine translation, and visual-language generation. This survey serves to facilitate collaborative efforts among researchers in tackling the challenge of hallucinated texts in NLG.

机器学习方法尤其是深度神经网络取得了巨大的成功，但其中许多往往依赖于一些标记的样品进行训练。在真实世界的应用中，我们经常需要通过例如具有新兴预测目标和昂贵的样本注释的动态上下文来解决样本短缺。因此，低资源学习，旨在学习具有足够资源（特别是培训样本）的强大预测模型，现在正在被广泛调查。在所有低资源学习研究中，许多人更喜欢以知识图（kg）的形式利用一些辅助信息，这对于知识表示变得越来越受欢迎，以减少对标记样本的依赖。在这项调查中，我们非常全面地审查了90美元的报纸关于两个主要的低资源学习设置 - 零射击学习（ZSL）的预测，从未出现过训练，而且很少拍摄的学习（FSL）预测的新类仅具有可用的少量标记样本。我们首先介绍了ZSL和FSL研究中使用的KGS以及现有的和潜在的KG施工解决方案，然后系统地分类和总结了KG感知ZSL和FSL方法，将它们划分为不同的范例，例如基于映射的映射，数据增强，基于传播和基于优化的。我们接下来呈现了不同的应用程序，包括计算机视觉和自然语言处理中的kg增强预测任务，还包括kg完成的任务，以及每个任务的一些典型评估资源。我们最终讨论了一些关于新学习和推理范式的方面的一些挑战和未来方向，以及高质量的KGs的建设。

从新闻文章中提取事件的信息论点是信息提取的一个具有挑战性的问题，这需要对每个文档的全球上下文理解。尽管有关文档级提取的最新工作已经超越了单句子，并提高了端到端模型的跨句子推理能力，但它们仍然受到某些输入序列长度约束的限制，通常忽略事件之间的全局上下文。为了解决此问题，我们通过构建文档存储器存储来记录上下文事件信息，并利用它隐含，明确地帮助解码以后事件的参数，从而引入了一个新的基于全局神经生成的框架，以用于文档级事件参数提取提取文档级别的事件参数提取。经验结果表明，我们的框架的表现要优于先验方法，并且使用约束的解码设计对对抗注释的示例更为强大。 （我们的代码和资源可在https://github.com/xinyadu/memory_docie上获得研究目的。）

基于多模式方面的情感分类（MABSC）是一项新兴的分类任务，旨在将给定目标的情感分类，例如具有不同模式的数据中提到的实体。在带有文本和图像的典型多模式数据中，以前的方法不能充分利用图像的细颗粒语义，尤其是与文本的语义结合在一起，并且不完全考虑对细粒图像之间的关系进行建模信息和目标，这导致图像的使用不足和不足以识别细粒度的方面和意见。为了应对这些局限性，我们提出了一个新的框架SEQCSG，包括一种构建顺序跨模式语义图和编码器模型的方法。具体而言，我们从原始图像，图像标题和场景图中提取细粒度的信息，并将它们视为跨模式语义图的元素以及文本的令牌。跨模式语义图表示为具有多模式可见矩阵的序列，指示元素之间的关系。为了有效地利用跨模式语义图，我们建议使用目标提示模板的编码器解码器方法。实验结果表明，我们的方法优于现有方法，并在两个标准数据集MABSC上实现了最新方法。进一步的分析证明了每个组件的有效性，我们的模型可以隐含地学习图像的目标和细粒度信息之间的相关性。

As an important fine-grained sentiment analysis problem, aspect-based sentiment analysis (ABSA), aiming to analyze and understand people's opinions at the aspect level, has been attracting considerable interest in the last decade. To handle ABSA in different scenarios, various tasks are introduced for analyzing different sentiment elements and their relations, including the aspect term, aspect category, opinion term, and sentiment polarity. Unlike early ABSA works focusing on a single sentiment element, many compound ABSA tasks involving multiple elements have been studied in recent years for capturing more complete aspect-level sentiment information. However, a systematic review of various ABSA tasks and their corresponding solutions is still lacking, which we aim to fill in this survey. More specifically, we provide a new taxonomy for ABSA which organizes existing studies from the axes of concerned sentiment elements, with an emphasis on recent advances of compound ABSA tasks. From the perspective of solutions, we summarize the utilization of pre-trained language models for ABSA, which improved the performance of ABSA to a new stage. Besides, techniques for building more practical ABSA systems in cross-domain/lingual scenarios are discussed. Finally, we review some emerging topics and discuss some open challenges to outlook potential future directions of ABSA.

开放信息提取（OpenIE）促进了独立于域的大型语料库的关系事实的发现。该技术很好地适合许多开放世界的自然语言理解场景，例如自动知识基础构建，开放域问答和明确的推理。由于深度学习技术的快速发展，已经提出了许多神经开放式体系结构并取得了可观的性能。在这项调查中，我们提供了有关状态神经开放模型的广泛概述，其关键设计决策，优势和劣势。然后，我们讨论当前解决方案的局限性以及OpenIE问题本身的开放问题。最后，我们列出了最近的趋势，这些趋势可以帮助扩大其范围和适用性，从而为Openie的未来研究设定了有希望的方向。据我们所知，本文是有关此特定主题的第一篇评论。

文本的结构化和接地表示通常是通过封闭信息提取形式化的，提取与从知识库模式的预定义实体集合和关系一致的穷举集（主题，关系，对象）三元组的问题。大多数现有的作品是管道容易出错的累积，所有方法都仅适用于不切实际的少数实体和关系。我们介绍了Genie（生成信息提取），第一端到最终的归属化闭合信息提取。 Genie自然地通过自动生成文本形式的关系和实体来利用预先训练的变压器的语言知识。由于新的双层约束生成策略，仅生产与预定义知识库模式一致的三胞胎。我们的实验表明，Genie在封闭信息提取时是最先进的，从较少的训练数据点广泛地推广到基线，并缩放到以前无管理数量的实体和关系。通过这项工作，封闭的信息提取在现实情景中变得实用，为下游任务提供了新的机会。最后，这项工作为信息提取的核心任务铺平了统一的端到端方法。在https://github.com/epfl-dlab/genie提供的代码和模型。

知识基础问题回答（KBQA）旨在通过知识库（KB）回答问题。早期研究主要集中于回答有关KB的简单问题，并取得了巨大的成功。但是，他们在复杂问题上的表现远非令人满意。因此，近年来，研究人员提出了许多新颖的方法，研究了回答复杂问题的挑战。在这项调查中，我们回顾了KBQA的最新进展，重点是解决复杂问题，这些问题通常包含多个主题，表达复合关系或涉及数值操作。详细说明，我们从介绍复杂的KBQA任务和相关背景开始。然后，我们描述用于复杂KBQA任务的基准数据集，并介绍这些数据集的构建过程。接下来，我们提出两个复杂KBQA方法的主流类别，即基于语义解析的方法（基于SP）的方法和基于信息检索的方法（基于IR）。具体而言，我们通过流程设计说明了他们的程序，并讨论了它们的主要差异和相似性。之后，我们总结了这两类方法在回答复杂问题时会遇到的挑战，并解释了现有工作中使用的高级解决方案和技术。最后，我们结论并讨论了与复杂的KBQA有关的几个有希望的方向，以进行未来的研究。