在本文中，我们呈现了Bartpho的两个版本Bartpho-symlable和Bartpho-Word，这是第一个为越南语预先培训的公共大规模单声道序列到序列模型。Bartpho使用“大”架构和序列序列去噪的预训练方案，因此特别适用于生成NLP任务。我们开展实验，以将我们的巴特照片与竞争对手MBART进行比较，以越南文本摘要的下游任务，表明：在自动和人类评估中，Bartpho优于强大的基线MBART并改善了最先进的。我们释放巴特诺以促进未来的生成越南NLP任务的研究和应用。我们的Bartpho模型可公开提供：https://github.com/vinairesearch/bartpho

The research on text summarization for low-resource Indian languages has been limited due to the availability of relevant datasets. This paper presents a summary of various deep-learning approaches used for the ILSUM 2022 Indic language summarization datasets. The ISUM 2022 dataset consists of news articles written in Indian English, Hindi, and Gujarati respectively, and their ground-truth summarizations. In our work, we explore different pre-trained seq2seq models and fine-tune those with the ILSUM 2022 datasets. In our case, the fine-tuned SoTA PEGASUS model worked the best for English, the fine-tuned IndicBART model with augmented data for Hindi, and again fine-tuned PEGASUS model along with a translation mapping-based approach for Gujarati. Our scores on the obtained inferences were evaluated using ROUGE-1, ROUGE-2, and ROUGE-4 as the evaluation metrics.

Pre-trained models have achieved remarkable success in natural language processing (NLP). However, existing pre-training methods underutilize the benefits of language understanding for generation. Inspired by the idea of Generative Adversarial Networks (GANs), we propose a GAN-style model for encoder-decoder pre-training by introducing an auxiliary discriminator, unifying the ability of language understanding and generation in a single model. Our model, named as GanLM, is trained with two pre-training objectives: replaced token detection and replaced token denoising. Specifically, given masked source sentences, the generator outputs the target distribution and the discriminator predicts whether the target sampled tokens from distribution are incorrect. The target sentence is replaced with misclassified tokens to construct noisy previous context, which is used to generate the gold sentence. In general, both tasks improve the ability of language understanding and generation by selectively using the denoising data. Extensive experiments in language generation benchmarks show that GanLM with the powerful language understanding capability outperforms various strong pre-trained language models (PLMs) and achieves state-of-the-art performance.

We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by ( 1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and many other more recent pretraining schemes. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of the original sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa with comparable training resources on GLUE and SQuAD, achieves new stateof-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 6 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining. We also report ablation experiments that replicate other pretraining schemes within the BART framework, to better measure which factors most influence end-task performance.

在本文中，我们利用了以前的预训练模型（PTM）的优势，并提出了一种新型的中国预训练的不平衡变压器（CPT）。与以前的中国PTM不同，CPT旨在利用自然语言理解（NLU）和自然语言生成（NLG）之间的共同知识来促进表现。 CPT包括三个部分：共享编码器，一个理解解码器和一代解码器。具有共享编码器的两个特定解码器分别通过蒙版语言建模（MLM）进行了预训练，并分别将自动编码（DAE）任务进行了验证。借助部分共享的体系结构和多任务预培训，CPT可以（1）使用两个解码器学习NLU或NLG任务的特定知识，并且（2）对模型的潜力充分利用了微调。此外，不平衡的变压器节省了计算和存储成本，这使CPT竞争激烈，并极大地加速了文本生成的推断。对各种中国NLU和NLG任务的实验结果显示了CPT的有效性。

This paper presents the work of restoring punctuation for ASR transcripts generated by multilingual ASR systems. The focus languages are English, Mandarin, and Malay which are three of the most popular languages in Singapore. To the best of our knowledge, this is the first system that can tackle punctuation restoration for these three languages simultaneously. Traditional approaches usually treat the task as a sequential labeling task, however, this work adopts a slot-filling approach that predicts the presence and type of punctuation marks at each word boundary. The approach is similar to the Masked-Language Model approach employed during the pre-training stages of BERT, but instead of predicting the masked word, our model predicts masked punctuation. Additionally, we find that using Jieba1 instead of only using the built-in SentencePiece tokenizer of XLM-R can significantly improve the performance of punctuating Mandarin transcripts. Experimental results on English and Mandarin IWSLT2022 datasets and Malay News show that the proposed approach achieved state-of-the-art results for Mandarin with 73.8% F1-score while maintaining a reasonable F1-score for English and Malay, i.e. 74.7% and 78% respectively. Our source code that allows reproducing the results and building a simple web-based application for demonstration purposes is available on Github.

大型审慎的语言模型最近征服了自然语言处理领域。作为BERT中引入的主要掩盖语言建模的替代方案，T5模型引入了更通用的训练目标，即序列转换的顺序，其中包括蒙版语言模型，但自然地适合文本生成任务，例如机器翻译，摘要，开放 - 开放 - 域问题回答，文本简化，对话系统等。T5模型的单语变体仅限于资源良好的语言，而大量的多语言T5模型则支持101种语言。相比之下，我们训练了两个不同尺寸的T5型序列，以使用较少的资源并分析其行为的形态丰富的斯洛文尼语的序列模型。关于分类任务，SLOT5模型主要落后于单语Slovene Sloberta模型，但应考虑生成任务。

对于多语言序列到序列预审预周序模型（多语言SEQ2SEQ PLM），例如姆巴特（Mbart），自制的预处理任务接受了多种单语言的培训，例如25种来自CommonCrawl的语言，而下游的跨语言任务通常在双语语言子集上进行，例如英语 - 德国人，存在数据差异，即领域的差异，以及跨语言学习客观差异，即在训练和填充阶段之间的任务差异。为了弥合上述跨语言域和任务差距，我们将使用额外的代码切换恢复任务扩展了香草预后管道。具体而言，第一阶段采用自我监督的代码转换还原任务作为借口任务，从而允许多语言SEQ2SEQ PLM获取一些域内对齐信息。在第二阶段，我们正常在下游数据上微调模型。 NLG评估（12个双语翻译任务，30个零射击任务和2项跨语言摘要任务）和NLU评估（7个跨语性自然语言推理任务）的实验表明，我们的模型超过了强大的基线MBART，具有标准的FINETUNNING，这表明了我们的模型策略，一致。分析表明，我们的方法可以缩小跨语性句子表示的欧几里得距离，并通过微不足道的计算成本改善模型概括。我们在：https：//github.com/zanchangtong/csr4mbart上发布代码。

对比学习模型在无监督的视觉表示学习中取得了巨大成功，这使得相同图像的不同视图的特征表示之间的相似性最大化，同时最小化不同图像的视图的特征表示之间的相似性。在文本摘要中，输出摘要是输入文档的较短形式，它们具有类似的含义。在本文中，我们提出了对监督抽象文本摘要的对比学习模型，在那里我们查看文档，它的金摘要及其模型生成的摘要，与相同的平均表示的不同视图，并在培训期间最大化它们之间的相似性。我们在三个不同的摘要数据集上改进了一个强序列到序列文本生成模型（即，BART）。人类评估还表明，与其对应物相比，我们的模型达到了更好的忠实性评级，没有对比的目标。

Powerful generative models have led to recent progress in question generation (QG). However, it is difficult to measure advances in QG research since there are no standardized resources that allow a uniform comparison among approaches. In this paper, we introduce QG-Bench, a multilingual and multidomain benchmark for QG that unifies existing question answering datasets by converting them to a standard QG setting. It includes general-purpose datasets such as SQuAD for English, datasets from ten domains and two styles, as well as datasets in eight different languages. Using QG-Bench as a reference, we perform an extensive analysis of the capabilities of language models for the task. First, we propose robust QG baselines based on fine-tuning generative language models. Then, we complement automatic evaluation based on standard metrics with an extensive manual evaluation, which in turn sheds light on the difficulty of evaluating QG models. Finally, we analyse both the domain adaptability of these models as well as the effectiveness of multilingual models in languages other than English. QG-Bench is released along with the fine-tuned models presented in the paper https://github.com/asahi417/lm-question-generation, which are also available as a demo https://autoqg.net/.

这项研究提供了对僧伽罗文本分类的预训练语言模型的性能的首次全面分析。我们测试了一组不同的Sinhala文本分类任务，我们的分析表明，在包括Sinhala（XLM-R，Labse和Laser）的预训练的多语言模型中，XLM-R是迄今为止Sinhala文本的最佳模型分类。我们还预先培训了两种基于罗伯塔的单语僧伽罗模型，它们远远优于僧伽罗的现有预训练的语言模型。我们表明，在微调时，这些预训练的语言模型为僧伽罗文本分类树立了非常强大的基线，并且在标记数据不足以进行微调的情况下非常强大。我们进一步提供了一组建议，用于使用预训练的模型进行Sinhala文本分类。我们还介绍了新的注释数据集，可用于僧伽罗文本分类的未来研究，并公开发布我们的预培训模型。

抽象性摘要领域的最新进展利用了预训练的语言模型，而不是从头开始训练模型。但是，这样的模型训练和伴随着大量的开销。研究人员提出了一些轻巧的替代方案，例如较小的适配器来减轻缺点。尽管如此，就提高效率而没有绩效不愉快的牺牲，使用使用适配器是否有利于总结的任务。在这项工作中，我们对具有不同复杂性的摘要任务进行了多方面的调查：语言，域和任务转移。在我们的实验中，对预训练的语言模型进行微调通常比使用适配器更好。性能差距与所使用的训练数据量正相关。值得注意的是，在极低的资源条件下，适配器超过微调。我们进一步提供了有关多语言，模型收敛性和鲁棒性的见解，希望能阐明抽象性摘要中微调或适配器的实用选择。

我们介绍了第一项经验研究，研究了突发性检测对意向检测和插槽填充的下游任务的影响。我们对越南人进行了这项研究，这是一种低资源语言，没有以前的研究，也没有公共数据集可用于探索。首先，我们通过手动添加上下文不满并注释它们来扩展流利的越南意图检测和插槽填充phoatis。然后，我们使用强基线进行实验进行实验，以基于预训练的语言模型，以检测和关节意图检测和插槽填充。我们发现：（i）爆发对下游意图检测和插槽填充任务的性能产生负面影响，并且（ii）在探索环境中，预先训练的多语言语言模型XLM-R有助于产生更好的意图检测和插槽比预先训练的单语言模型phobert填充表演，这与在流利性环境中通常发现的相反。

This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART -a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective . mBART is the first method for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text. Pre-training a complete model allows it to be directly fine tuned for supervised (both sentence-level and document-level) and unsupervised machine translation, with no task-specific modifications. We demonstrate that adding mBART initialization produces performance gains in all but the highest-resource settings, including up to 12 BLEU points for low resource MT and over 5 BLEU points for many document-level and unsupervised models. We also show it also enables new types of transfer to language pairs with no bi-text or that were not in the pre-training corpus, and present extensive analysis of which factors contribute the most to effective pre-training.

Bidirectional Encoder Representations from Transformers (BERT; Devlin et al. 2019) represents the latest incarnation of pretrained language models which have recently advanced a wide range of natural language processing tasks. In this paper, we showcase how BERT can be usefully applied in text summarization and propose a general framework for both extractive and abstractive models. We introduce a novel document-level encoder based on BERT which is able to express the semantics of a document and obtain representations for its sentences. Our extractive model is built on top of this encoder by stacking several intersentence Transformer layers. For abstractive summarization, we propose a new fine-tuning schedule which adopts different optimizers for the encoder and the decoder as a means of alleviating the mismatch between the two (the former is pretrained while the latter is not). We also demonstrate that a two-staged fine-tuning approach can further boost the quality of the generated summaries. Experiments on three datasets show that our model achieves stateof-the-art results across the board in both extractive and abstractive settings. 1

在这项工作中，我们证明了多种语的大规模序列到序列（SEQ2SEQ）模型，该模型是通过Denoising和因果语言建模（CLM）任务的混合物进行训练的，比仅解码器模型更有效地进行了效率的学习者在各种任务上。特别是，我们培训了一个名为Alexa教师模型（Alexatm 20b）的200亿个参数多语言SEQ2SEQ模型，并表明它在1-Shot摘要任务上实现了最先进的（SOTA）性能，超过了更大的540B PALM DOPODER模型。 Alexatm 20b还可以在1-Shot Machine翻译中实现SOTA，尤其是对于低资源语言，几乎所有语言对（阿拉伯语，英语，法语，德语，德语，印地语，意大利语，日语，以及flores-101数据集上的泰卢固语）。我们还显示了零拍设置，AlexATM 20B在SuperGlue和SqueadV2数据集上的表现优于GPT3（175B），并在XNLI，XCOPA，PAWS-X和XWINOGRAD等多语言任务上提供SOTA性能。总体而言，我们的结果为SEQ2SEQ模型提供了一个令人信服的案例，作为大型语言模型（LLM）培训的仅解码器模型的强大替代方法。

预先训练的语言模型已经建立了有关各种自然语言处理任务的最新技术，包括对话摘要，这使读者可以在会议，访谈或电话中的长时间对话中快速访问关键信息。但是，这种对话仍然很难使用当前的模型来处理，因为语言的自发性涉及在用于预先培训语言模型的语料库中很少存在的表达式。此外，在这一领域完成的绝大多数工作都集中在英语上。在这项工作中，我们介绍了一项研究，使用几种特定语言的预培训模型：Barthez和Belgpt-2以及多语言预培训的模型：MBART，MBARTHEZ和MT5。实验是在Decoda（呼叫中心）对话语料库上进行的，其任务是根据情况在呼叫中心与一个或几个代理之间的呼叫中心对话中产生抽象介绍。结果表明，Barthez型号的性能最佳，远远超过了Decoda先前的最新性能。我们进一步讨论了此类预训练模型的局限性以及总结自发对话所需的挑战。

本文介绍了Z-Code ++，这是一种针对抽象文本摘要优化的新的预训练的语言模型。该模型使用三种技术扩展了艺术编码器模型的状态。首先，我们使用两阶段的预训练过程来改善模型在低资源摘要任务上的性能。该模型首先是使用文本语料库进行语言理解的预先培训的，然后在汇总语料库中不断预先培训，以进行基础文本生成。其次，我们用分离的注意力层代替编码器中的自我发项层，其中每个单词都使用两个向量分别代表其内容和位置。第三，我们使用融合编码器，这是一种以层次方式编码长序列的简单而有效的方法。 Z-Code ++在13个文本摘要任务中的9个跨5种语言中创建了新的艺术状态。我们的模型的参数有效，因为它的表现优于XSUM上600倍较大的Palm-540b，并且在Samsum上的易经的200倍GPT3-175B较大。在零射击和少量设置中，我们的模型大大优于竞争模型。

特定于语言的预训练模型已被证明比单语说在单语法评估设置中更准确，阿拉伯语也不例外。但是，我们发现先前发布的阿拉伯伯特模型显着培训。在这本技术报告中，我们展示了Jaber，Junior Arabic Bert，我们的预用语言模型原型专用于阿拉伯语。我们进行实证研究，以系统地评估模型在各种现有阿拉伯语NLU任务中的性能。实验结果表明，Jaber实现了Alue的最先进的表演，这是阿拉伯语了解评估的新基准，以及成熟的内部基准

最近在单语数据和机器翻译（MT）进行微调的预培训方面取得了成功，但尚不清楚如何最好地利用预先训练的模型来完成给定的MT任务。本文在微调MT上的预训练模型时研究了冻结参数的好处和缺点。我们专注于1）微调仅在英语单语言数据的BART上训练的模型。2）微调一个模型，该模型对25种语言的单语言数据进行了培训，Mbart。对于Bart，我们通过冻结大多数模型参数并添加额外的位置嵌入来获得最佳性能。对于MBART，我们将大多数语言对的天真微调的性能与编码器以及大多数解码器搭配。编码器的注意参数对于微调最重要。当将自己限制为越南人对英语的室外训练套装时，我们看到了基线的最大进步。