本文提出了一种简单的方法，用于使用自由形式分类器（即CAIF采样）基于加权逻辑来控制文本生成。使用任意文本分类器，我们将语言模型逻辑的一小部分调整为指导文本生成，以远离分类器预测。我们试验了避免毒性和情感控制任务，并表明该方法在PPL和DESS准确度指标上基于生成的文本的外部分类器而显着优于最近的PPLM，GEDI和DEXPERTS。此外，与其他方法相比，它更容易实施和调整，并且限制和要求较少。

Large pretrained language models generate fluent text but are notoriously hard to controllably sample from. In this work, we study constrained sampling from such language models: generating text that satisfies user-defined constraints, while maintaining fluency and the model's performance in a downstream task. We propose MuCoLa -- a sampling procedure that combines the log-likelihood of the language model with arbitrary (differentiable) constraints in a single energy function, and then generates samples in a non-autoregressive manner. Specifically, it initializes the entire output sequence with noise and follows a Markov chain defined by Langevin Dynamics using the gradients of the energy function. We evaluate MuCoLa on text generation with soft and hard constraints as well as their combinations obtaining significant improvements over competitive baselines for toxicity avoidance, sentiment control, and keyword-guided generation.

Steering language generation towards objectives or away from undesired content has been a long-standing goal in utilizing language models (LM). Recent work has demonstrated reinforcement learning and weighted decoding as effective approaches to achieve a higher level of language control and quality with pros and cons. In this work, we propose a novel critic decoding method for controlled language generation (CriticControl) that combines the strengths of reinforcement learning and weighted decoding. Specifically, we adopt the actor-critic framework to train an LM-steering critic from non-differentiable reward models. And similar to weighted decoding, our method freezes the language model and manipulates the output token distribution using called critic, improving training efficiency and stability. Evaluation of our method on three controlled generation tasks, namely topic control, sentiment control, and detoxification, shows that our approach generates more coherent and well-controlled texts than previous methods. In addition, CriticControl demonstrates superior generalization ability in zero-shot settings. Human evaluation studies also corroborate our findings.

Controlled text generation is a very important task in the arena of natural language processing due to its promising applications. In order to achieve this task we mainly introduce the novel soft prompt tuning method of using soft prompts at both encoder and decoder levels together in a T5 model and investigate the performance as the behaviour of an additional soft prompt related to the decoder of a T5 model in controlled text generation remained unexplored. Then we also investigate the feasibility of steering the output of this extended soft prompted T5 model at decoder level and finally analyse the utility of generated text to be used in AI related tasks such as training AI models with an interpretability analysis of the classifier trained with synthetic text, as there is a lack of proper analysis of methodologies in generating properly labelled data to be utilized in AI tasks. Through the performed in-depth intrinsic and extrinsic evaluations of this generation model along with the artificially generated data, we found that this model produced better results compared to the T5 model with a single soft prompt at encoder level and the sentiment classifier trained using this artificially generated data can produce comparable classification results to the results of a classifier trained with real labelled data and also the classifier decision is interpretable with respect to the input text content.

当前的语言模型达到了较低的困惑，但其产生的几代人仍然遭受有毒的反应，重复性和矛盾。标准语言建模设置无法解决这些问题。在本文中，我们介绍了一个新的体系结构{\ sc导演}，由一个统一的生成器分类器组成，具有语言建模和每个输出令牌的分类头。培训是使用标准语言建模数据共同进行的，并以所需和不良序列标记的数据。与标准语言模型相比，该模型在多种设置中的实验表明，该模型具有竞争性的培训和解码速度，同时产生了较高的结果，从而减轻了已知的问题，同时保持发电质量。就准确性和效率而言，它还优于现有的模型指导方法。

The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining. Reformulating tasks as fillin-the-blanks problems (e.g., cloze tests) is a natural approach for gauging such knowledge, however, its usage is limited by the manual effort and guesswork required to write suitable prompts. To address this, we develop AUTOPROMPT, an automated method to create prompts for a diverse set of tasks, based on a gradient-guided search. Using AUTO-PROMPT, we show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning, sometimes achieving performance on par with recent state-of-the-art supervised models. We also show that our prompts elicit more accurate factual knowledge from MLMs than the manually created prompts on the LAMA benchmark, and that MLMs can be used as relation extractors more effectively than supervised relation extraction models. These results demonstrate that automatically generated prompts are a viable parameter-free alternative to existing probing methods, and as pretrained LMs become more sophisticated and capable, potentially a replacement for finetuning.

上下文学习是最近的自然语言理解的范例，其中大型预先接受的语言模型（LM）观察测试实例和一些训练示例作为其输入，并直接对输出进行解码，而不会对其参数进行任何更新。但是，表现已被证明强烈依赖于所选培训示例（称为提示）。在这项工作中，我们提出了一种有效的方法，用于使用注释的数据和LM检索内心学习的提示。给定输入输出对，我们估计给出输入和候选训练示例的输出的概率作为提示，以及基于这种概率的正面或负标记训练示例。然后，我们从该数据中培训一个有效的密集鼠尾，用于检索训练示例作为测试时间的提示。我们在三个序列到序列任务中评估我们的方法，其中语言话语映射到意义表示，并发现它基本上优于前面的工作和电路板的多个基线。

Standard language model training employs gold human documents or human-human interaction data, and treats all training data as positive examples. Growing evidence shows that even with very large amounts of positive training data, issues remain that can be alleviated with relatively small amounts of negative data -- examples of what the model should not do. In this work, we propose a novel procedure to train with such data called the CRINGE loss (ContRastive Iterative Negative GEneration). We show the effectiveness of this approach across three different experiments on the tasks of safe generation, contradiction avoidance, and open-domain dialogue. Our models outperform multiple strong baselines and are conceptually simple, easy to train and implement.

大型预训练的语言模型能够产生多种多样的文本。从提示开始，这些模型产生了一种可以不可预测的叙述。现有的可控文本生成方法，该方法指导用户指定方向的文本中的叙述，需要创建培训语料库和额外的耗时培训程序。本文提出并调查了Contocation2Text，这是一种用于俄罗斯自动可控文本生成的插件方法，不需要微调。该方法基于两个交互模型：自回归语言Rugpt-3模型和自动编码语言Ruroberta模型。该方法的想法是根据自动编码模型的输出分布将自回归模型的输出分布移动，以确保文本中叙事的连贯过渡向指南短语，其中可以包含单个单词或搭配。能够考虑到令牌的左和右下方的自动编码模型“告诉”“自动回归模型”在当前一代步骤中，该模型是令牌最不合逻辑的，从而增加或降低了相应令牌的概率。使用该方法生成新闻文章的实验显示了其对自动生成的流利文本的有效性，这些文本包含用户指定的短语之间的连贯过渡。

当前，可以将预训练的模型视为多种NLP任务的默认选择。尽管有SOTA结果，但有实际证据表明，这些模型可能需要不同数量的计算层来进行不同的输入序列，因为评估所有层都会导致错误的预测过度自信（即过度思考）。可以通过实施自适应计算时间方法来解决此问题，该方法首先旨在提高推理速度。最近提出的洪加尔网可能是通过将出口层的索引视为潜在变量来进行早期出口的有前途解决方案。但是，最初提出的退出标准依赖于从经过训练的后验分布中取样的概率，从$ i $ the层退出，引入了出口层指数的主要差异，从而大大降低了所得模型的性能。在本文中，我们提出了通过新颖的确定性Q-Exit标准和重新审视的模型体系结构提出改进的Pondernet。我们将提出的机制调整为阿尔伯特和罗伯塔，并将其与最近进行早期出口的方法进行了比较。我们观察到，在广泛的胶水任务上，可以将提出的更改视为对原始蓬松网络架构的重大改进，并胜过pabee。此外，我们还对拟议的体系结构进行了深入的消融研究，以进一步了解Lambda层及其性能。

控制语言模型的主要方法在控制高级属性（例如主题和情感）方面具有突出性。但是，这些方法通常需要特定于条件的数据或计算昂贵。我们提出了一种新的简单引导解码方法，伽玛采样，该方法不需要任何培训数据来实现可控制的文本生成，同时保持快速生成速度。伽玛采样将与属性相关的信息（由人类或语言模型本身提供）引入采样过程中，以指导语言模型，以生成具有所需属性的文本。由于不涉及培训，因此可以轻松地将伽马抽样应用于任何语言模型以进行可控文本。通过实验，我们表明，伽马取样的GPT2-MALL（1.17亿）优于PPLM（345m）和CTRL（1.6B）的多样性，属性相关性以及生成样品的整体质量。

我们提出了两种小型无监督方法，用于消除文本中的毒性。我们的第一个方法结合了最近的两个想法：（1）使用小型条件语言模型的生成过程的指导和（2）使用释义模型进行风格传输。我们使用良好的令人措辞的令人愉快的释放器，由风格培训的语言模型引导，以保持文本内容并消除毒性。我们的第二种方法使用BERT用他们的非攻击性同义词取代毒性单词。我们通过使BERT替换具有可变数量的单词的屏蔽令牌来使该方法更灵活。最后，我们介绍了毒性去除任务的风格转移模型的第一个大规模比较研究。我们将模型与许多用于样式传输的方法进行比较。使用无监督的样式传输指标的组合以可参考方式评估该模型。两种方法都建议产生新的SOTA结果。

Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that are more natural and better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the lower level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks which may require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize CTG techniques from the perspective of PLMs. We hope it can help researchers in related fields to quickly track the academic frontier, providing them with a landscape of the area and a roadmap for future research.

我们呈现隐藏状态优化（HSO），一种基于梯度的方法，用于提高推理时间的变压器语言模型的性能。类似于动态评估（KRAUE等，2018），HSO计算语言模型分配给评估文本的日志概率的渐变，但使用它来更新缓存的隐藏状态而不是模型参数。我们用预磨削的变换器-XL和GPT-2语言模型测试HSO，在困惑方面发现Wikitext103和PG-19数据集的改进，特别是在评估其培训分布之外的模型时。我们还通过在最近开发的基于少量拍摄评估设置中显示出口，再次展示下游适用性，没有额外的参数或培训数据。

基于变压器的语言模型能够生成流利的文本，并在各种自然语言生成任务中有效地适应。但是，已证明在大型未标记的网络文本语料库中鉴定的语言模型已被证明会遭受堕落的有毒内容和社会偏见行为的损害，从而阻碍了他们的安全部署。提出了各种排毒方法来减轻语言模型的毒性；但是，这些方法是在包含与性别，种族或宗教相关的特定社会身份的提示条件下进行排毒语言模型的。在这项研究中，我们提出了增强氧化。一种基于强化学习的方法，用于降低语言模型中的毒性。我们应对语言模型中的安全性挑战，并提出了一种新的奖励模型，该模型能够检测有毒内容并减轻对毒性预测中社会身份的意外偏见。该实验表明，用于语言模型排毒的增强方法化方法优于自动评估指标中现有的排毒方法，这表明我们在语言模型排毒中的方法能力和对生成内容中社会认同的意外偏见的能力较小。

We consider the task of text generation in language models with constraints specified in natural language. To this end, we first create a challenging benchmark Cognac that provides as input to the model a topic with example text, along with a constraint on text to be avoided. Unlike prior work, our benchmark contains knowledge-intensive constraints sourced from databases like Wordnet and Wikidata, which allows for straightforward evaluation while striking a balance between broad attribute-level and narrow lexical-level controls. We find that even state-of-the-art language models like GPT-3 fail often on this task, and propose a solution to leverage a language model's own internal knowledge to guide generation. Our method, called CognacGen, first queries the language model to generate guidance terms for a specified topic or constraint, and uses the guidance to modify the model's token generation probabilities. We propose three forms of guidance (binary verifier, top-k tokens, textual example), and employ prefix-tuning approaches to distill the guidance to tackle diverse natural language constraints. Through extensive empirical evaluations, we demonstrate that CognacGen can successfully generalize to unseen instructions and outperform competitive baselines in generating constraint conforming text.

We describe PromptBoosting, a query-efficient procedure for building a text classifier from a neural language model (LM) without access to the LM's parameters, gradients, or hidden representations. This form of "black-box" classifier training has become increasingly important as the cost of training and inference in large-scale LMs grows. But existing black-box LM classifier learning approaches are themselves computationally inefficient, typically specializing LMs to the target task by searching in a large space of (discrete or continuous) prompts using zeroth-order optimization methods. Instead of directly optimizing in prompt space, PromptBoosting obtains a small pool of prompts via a gradient-free approach and then constructs a large pool of weak learners by pairing these prompts with different elements of the LM's output distribution. These weak learners are then ensembled using the AdaBoost algorithm. The entire learning process requires only a small number of forward passes and no backward pass. Experiments show that PromptBoosting achieves state-of-the-art performance in multiple black-box few-shot classification tasks, and matches or outperforms full fine-tuning in both few-shot and standard learning paradigms, while training 10x faster than existing black-box methods.

预培训语言模型的浪潮一直不断提高机器生成的对话的质量，然而，一些产生的响应仍然遭受过度重复，有时重复从话语中重复单词，有时重复自我产生的响应中的单词，或者两个都。不当重复单词可以显着降低生成文本的质量。受到惩罚的采样是一种流行的解决方案，减少了推理期间现有词的采样概率，但是，它非常容易受到静态的不适当的设置。将其设置得太高可以产生奇怪和不切实际的句子，同时将其设置得太低，使得抑制重复微不足道的任务。要解决上述方法的缺点，我们设计了一个上下文感知的分类器，以明确决定何时允许重复和何时采用惩罚的采样。这种分类器可以容易地与现有的解码方法集成，在保持文本的分集的同时在适当的情况下减少重复。实验结果表明，我们的方法可以产生更高质量和更真实的对话。

Large language models can perform new tasks in a zero-shot fashion, given natural language prompts that specify the desired behavior. Such prompts are typically hand engineered, but can also be learned with gradient-based methods from labeled data. However, it is underexplored what factors make the prompts effective, especially when the prompts are natural language. In this paper, we investigate common attributes shared by effective prompts. We first propose a human readable prompt tuning method (F LUENT P ROMPT) based on Langevin dynamics that incorporates a fluency constraint to find a diverse distribution of effective and fluent prompts. Our analysis reveals that effective prompts are topically related to the task domain and calibrate the prior probability of label words. Based on these findings, we also propose a method for generating prompts using only unlabeled data, outperforming strong baselines by an average of 7.0% accuracy across three tasks.

公开可用的大型预磨语删除媒介（LMS）生成具有显着质量的文本，但仅从左右依次顺序地。因此，它们不会立即适用于打破单向假设的生成任务，例如释放或文本缺陷，需要特定于特定的监督。在本文中，我们呈现反射解码，这是一种新型无监督算法，其允许直接向非顺序任务应用单向LMS。我们的2步方法不需要监督甚至并行对象，只有两个离心的预磨损LMS相反的方向：向前和向后。首先，在上下文化步骤中，我们使用LMS生成过去和未来环境的集合，该上下文共同捕获输入（例如，索引源句）。其次，在反射步骤中，我们在这些“上下文集合”中的条件，生成与它们兼容的输出。综合经验结果表明，反思解码优于涉及释义和绑架文本缺陷的强烈无监督的基线，显着缩小无监督和监督方法之间的差距。反射解码超越了各种度量的多个监督基线，包括人为评估。