Large pretrained language models generate fluent text but are notoriously hard to controllably sample from. In this work, we study constrained sampling from such language models: generating text that satisfies user-defined constraints, while maintaining fluency and the model's performance in a downstream task. We propose MuCoLa -- a sampling procedure that combines the log-likelihood of the language model with arbitrary (differentiable) constraints in a single energy function, and then generates samples in a non-autoregressive manner. Specifically, it initializes the entire output sequence with noise and follows a Markov chain defined by Langevin Dynamics using the gradients of the energy function. We evaluate MuCoLa on text generation with soft and hard constraints as well as their combinations obtaining significant improvements over competitive baselines for toxicity avoidance, sentiment control, and keyword-guided generation.

真实世界的文本应用程序通常涉及组成广泛的文本控制操作，例如编辑文本W.R.T.属性，操纵关键字和结构，并生成所需属性的新文本。事先的工作通常会学习/芬太尼语言模型（LM）以执行操作的个人或特定子集。最近的研究以插件方式研究了合并操作，通常在复杂序列空间中以昂贵的搜索或优化进行了研究。本文提出了一种新的有效方法，用于在紧凑的文本潜在空间中进行可复合的文本操作。文本潜在矢量的低维度和不同性使我们能够基于给定的任意插入运算符（例如属性分类器）基于普通微分方程（ODE）开发有效的采样器。通过通过有效的适应性将预告片的LMS（例如GPT2）连接到潜在空间，然后我们将采样向量解码为所需的文本序列。灵活的方法允许使用来自不同域中的任何相关数据获取的各种控制操作员（情感，时态，形式，关键字等）。实验表明，在我们的方法中构成这些操作员可以生成或编辑高质量文本，从而在发电质量和效率方面显着改善了以前的方法。

Steering language generation towards objectives or away from undesired content has been a long-standing goal in utilizing language models (LM). Recent work has demonstrated reinforcement learning and weighted decoding as effective approaches to achieve a higher level of language control and quality with pros and cons. In this work, we propose a novel critic decoding method for controlled language generation (CriticControl) that combines the strengths of reinforcement learning and weighted decoding. Specifically, we adopt the actor-critic framework to train an LM-steering critic from non-differentiable reward models. And similar to weighted decoding, our method freezes the language model and manipulates the output token distribution using called critic, improving training efficiency and stability. Evaluation of our method on three controlled generation tasks, namely topic control, sentiment control, and detoxification, shows that our approach generates more coherent and well-controlled texts than previous methods. In addition, CriticControl demonstrates superior generalization ability in zero-shot settings. Human evaluation studies also corroborate our findings.

神经文本生成的主导范式是自回归语言模型的左右解码。然而，复杂的词汇约束下的受约束或可控发生的产生需要远见计划未来可行的未来路径。从A *搜索算法绘制灵感，我们提出了一种神经系统A * esque，一种解码算法包含未来成本的启发式估计。我们开发了高效的寻找高效，对大规模语言模型有效，使我们的方法成为诸如光束搜索和顶-K采样等共同技术的替代品。为了使受约束的产生，我们构建了神经系统解码（Lu等，2021），将其灵活性结合到与未来约束满足的* esque估计结合起来的逻辑限制。我们的方法在五代任务中优于竞争力的基线，并在表格到文本生成，受限机器翻译和关键字的生成中实现了新的最先进的性能。在需要复杂约束满足或少量拍摄或零拍摄设置的任务上，改进尤其显着。神经系统A * esque说明了用于改进和实现大规模语言模型的新功能的解码的力量。

Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that are more natural and better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the lower level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks which may require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize CTG techniques from the perspective of PLMs. We hope it can help researchers in related fields to quickly track the academic frontier, providing them with a landscape of the area and a roadmap for future research.

当前的语言模型达到了较低的困惑，但其产生的几代人仍然遭受有毒的反应，重复性和矛盾。标准语言建模设置无法解决这些问题。在本文中，我们介绍了一个新的体系结构{\ sc导演}，由一个统一的生成器分类器组成，具有语言建模和每个输出令牌的分类头。培训是使用标准语言建模数据共同进行的，并以所需和不良序列标记的数据。与标准语言模型相比，该模型在多种设置中的实验表明，该模型具有竞争性的培训和解码速度，同时产生了较高的结果，从而减轻了已知的问题，同时保持发电质量。就准确性和效率而言，它还优于现有的模型指导方法。

深度神经语言模型的最新进展与大规模数据集的能力相结合，加速了自然语言生成系统的发展，这些系统在多种任务和应用程序上下文中产生流利和连贯的文本（在各种成功程度上）。但是，为所需的用户控制这些模型的输出仍然是一个开放的挑战。这不仅对于自定义生成语言的内容和样式至关重要，而且对于他们在现实世界中的安全可靠部署至关重要。我们提出了一项关于受约束神经语言生成的新兴主题的广泛调查，在该主题中，我们通过区分条件和约束（后者是在输出文本上而不是输入的可检验条件），正式定义和分类自然语言生成问题，目前是可检验的）约束文本生成任务，并查看受限文本生成的现有方法和评估指标。我们的目的是强调这个新兴领域的最新进展和趋势，以告知最有希望的方向和局限性，以推动受约束神经语言生成研究的最新作品。

预处理的基于变压器的语言模型（LMS）显示出显着的自然语言生成能力。凭借其巨大的潜力，控制这种LM的文本生成引起了人们的关注。尽管有一些研究试图控制生成的文本的高级属性（例如情感和主题），但仍然缺乏对其在单词和短语级别上的内容的更精确的控制。在这里，我们建议内容调节器（COCON）以细粒度的水平控制LM的输出文本。在我们的自我监督方法中，Cocon Block学会了通过调节从LM中扣留的内容输入来帮助LM完成部分观察到的文本序列。通过实验，我们表明Cocon可以自然地将目标内容纳入生成的文本中，并以零拍的方式控制高级文本属性。

由于在开放式文本生成中取得了重大进展，衡量机器生成的文本是如何对人类语言的关键问题。我们介绍紫红色，一个开放式文本生成的比较措施，它直接将文本生成模型的学习分布与使用发散边界的分发进行了分布到人写的文本。淡紫色通过计算量化嵌入空间中的信息分流来缩放到现代文本生成模型。通过对三个开放式发电任务的广泛实证研究，我们发现紫红色标识了所生成文本的已知属性，天然存在模型大小，并与人类判断相关，而不是现有的分布评估度量的限制较少。

本文提出了一种简单的方法，用于使用自由形式分类器（即CAIF采样）基于加权逻辑来控制文本生成。使用任意文本分类器，我们将语言模型逻辑的一小部分调整为指导文本生成，以远离分类器预测。我们试验了避免毒性和情感控制任务，并表明该方法在PPL和DESS准确度指标上基于生成的文本的外部分类器而显着优于最近的PPLM，GEDI和DEXPERTS。此外，与其他方法相比，它更容易实施和调整，并且限制和要求较少。

我们提出了两种小型无监督方法，用于消除文本中的毒性。我们的第一个方法结合了最近的两个想法：（1）使用小型条件语言模型的生成过程的指导和（2）使用释义模型进行风格传输。我们使用良好的令人措辞的令人愉快的释放器，由风格培训的语言模型引导，以保持文本内容并消除毒性。我们的第二种方法使用BERT用他们的非攻击性同义词取代毒性单词。我们通过使BERT替换具有可变数量的单词的屏蔽令牌来使该方法更灵活。最后，我们介绍了毒性去除任务的风格转移模型的第一个大规模比较研究。我们将模型与许多用于样式传输的方法进行比较。使用无监督的样式传输指标的组合以可参考方式评估该模型。两种方法都建议产生新的SOTA结果。

Large language models can perform new tasks in a zero-shot fashion, given natural language prompts that specify the desired behavior. Such prompts are typically hand engineered, but can also be learned with gradient-based methods from labeled data. However, it is underexplored what factors make the prompts effective, especially when the prompts are natural language. In this paper, we investigate common attributes shared by effective prompts. We first propose a human readable prompt tuning method (F LUENT P ROMPT) based on Langevin dynamics that incorporates a fluency constraint to find a diverse distribution of effective and fluent prompts. Our analysis reveals that effective prompts are topically related to the task domain and calibrate the prior probability of label words. Based on these findings, we also propose a method for generating prompts using only unlabeled data, outperforming strong baselines by an average of 7.0% accuracy across three tasks.

Text detoxification has the potential to mitigate the harms of toxicity by rephrasing text to remove offensive meaning, but subtle toxicity remains challenging to tackle. We introduce MaRCo, a detoxification algorithm that combines controllable generation and text rewriting methods using a Product of Experts with autoencoder language models (LMs). MaRCo uses likelihoods under a non-toxic LM (expert) and a toxic LM (anti-expert) to find candidate words to mask and potentially replace. We evaluate our method on several subtle toxicity and microaggressions datasets, and show that it not only outperforms baselines on automatic metrics, but MaRCo's rewrites are preferred 2.1 $\times$ more in human evaluation. Its applicability to instances of subtle toxicity is especially promising, demonstrating a path forward for addressing increasingly elusive online hate.

We consider the task of text generation in language models with constraints specified in natural language. To this end, we first create a challenging benchmark Cognac that provides as input to the model a topic with example text, along with a constraint on text to be avoided. Unlike prior work, our benchmark contains knowledge-intensive constraints sourced from databases like Wordnet and Wikidata, which allows for straightforward evaluation while striking a balance between broad attribute-level and narrow lexical-level controls. We find that even state-of-the-art language models like GPT-3 fail often on this task, and propose a solution to leverage a language model's own internal knowledge to guide generation. Our method, called CognacGen, first queries the language model to generate guidance terms for a specified topic or constraint, and uses the guidance to modify the model's token generation probabilities. We propose three forms of guidance (binary verifier, top-k tokens, textual example), and employ prefix-tuning approaches to distill the guidance to tackle diverse natural language constraints. Through extensive empirical evaluations, we demonstrate that CognacGen can successfully generalize to unseen instructions and outperform competitive baselines in generating constraint conforming text.

基于变压器的语言模型能够生成流利的文本，并在各种自然语言生成任务中有效地适应。但是，已证明在大型未标记的网络文本语料库中鉴定的语言模型已被证明会遭受堕落的有毒内容和社会偏见行为的损害，从而阻碍了他们的安全部署。提出了各种排毒方法来减轻语言模型的毒性；但是，这些方法是在包含与性别，种族或宗教相关的特定社会身份的提示条件下进行排毒语言模型的。在这项研究中，我们提出了增强氧化。一种基于强化学习的方法，用于降低语言模型中的毒性。我们应对语言模型中的安全性挑战，并提出了一种新的奖励模型，该模型能够检测有毒内容并减轻对毒性预测中社会身份的意外偏见。该实验表明，用于语言模型排毒的增强方法化方法优于自动评估指标中现有的排毒方法，这表明我们在语言模型排毒中的方法能力和对生成内容中社会认同的意外偏见的能力较小。

Pretrained language models have demonstrated extraordinary capabilities in language generation. However, real-world tasks often require controlling the distribution of generated text in order to mitigate bias, promote fairness, and achieve personalization. Existing techniques for controlling the distribution of generated text only work with quantified distributions, which require pre-defined categories, proportions of the distribution, or an existing corpus following the desired distributions. However, many important distributions, such as personal preferences, are unquantified. In this work, we tackle the problem of generating text following arbitrary distributions (quantified and unquantified) by proposing Nano, a few-shot human-in-the-loop training algorithm that continuously learns from human feedback. Nano achieves state-of-the-art results on single topic/attribute as well as quantified distribution control compared to previous works. We also show that Nano is able to learn unquantified distributions, achieves personalization, and captures differences between different individuals' personal preferences with high sample efficiency.

控制语言模型的主要方法在控制高级属性（例如主题和情感）方面具有突出性。但是，这些方法通常需要特定于条件的数据或计算昂贵。我们提出了一种新的简单引导解码方法，伽玛采样，该方法不需要任何培训数据来实现可控制的文本生成，同时保持快速生成速度。伽玛采样将与属性相关的信息（由人类或语言模型本身提供）引入采样过程中，以指导语言模型，以生成具有所需属性的文本。由于不涉及培训，因此可以轻松地将伽马抽样应用于任何语言模型以进行可控文本。通过实验，我们表明，伽马取样的GPT2-MALL（1.17亿）优于PPLM（345m）和CTRL（1.6B）的多样性，属性相关性以及生成样品的整体质量。

Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few-shot generalization to unseen tasks. However, there is a limited understanding of the performance trade-offs of different decisions made during the instruction-tuning process. These decisions include the scale and diversity of the instruction-tuning benchmark, different task sampling strategies, fine-tuning with and without demonstrations, training using specialized datasets for reasoning and dialogue, and finally, the fine-tuning objectives themselves. In this paper, we characterize the effect of instruction-tuning decisions on downstream task performance when scaling both model and benchmark sizes. To this end, we create OPT-IML Bench: a large benchmark for Instruction Meta-Learning (IML) of 2000 NLP tasks consolidated into task categories from 8 existing benchmarks, and prepare an evaluation framework to measure three types of model generalizations: to tasks from fully held-out categories, to held-out tasks from seen categories, and to held-out instances from seen tasks. Through the lens of this framework, we first present insights about instruction-tuning decisions as applied to OPT-30B and further exploit these insights to train OPT-IML 30B and 175B, which are instruction-tuned versions of OPT. OPT-IML demonstrates all three generalization abilities at both scales on four different evaluation benchmarks with diverse tasks and input formats -- PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG. Not only does it significantly outperform OPT on all benchmarks but is also highly competitive with existing models fine-tuned on each specific benchmark. We release OPT-IML at both scales, together with the OPT-IML Bench evaluation framework.

Diffusion models have quickly become the go-to paradigm for generative modelling of perceptual signals (such as images and sound) through iterative refinement. Their success hinges on the fact that the underlying physical phenomena are continuous. For inherently discrete and categorical data such as language, various diffusion-inspired alternatives have been proposed. However, the continuous nature of diffusion models conveys many benefits, and in this work we endeavour to preserve it. We propose CDCD, a framework for modelling categorical data with diffusion models that are continuous both in time and input space. We demonstrate its efficacy on several language modelling tasks.

我们探索使用大型预用语言模型作为少量语义解析器。语义解析中的目标是给定自然语言输入的结构化含义表示。但是，培训语言模型以生成自然语言。为了弥合差距，我们使用语言模型来解释进入一个类似于英语的受控的子宫内的输入，可以自动映射到目标含义表示表示。我们的结果表明，只有少量的数据和较少的代码转换为类似英语的代表，我们为快速启动语义解析器的蓝图导致了对多个社区任务的令人惊讶的有效性能，大大超过基线方法也在相同的限制上培训数据。