中文拼写检查（CSC）的任务旨在检测和纠正文本中可以找到的拼写错误。虽然手动注释高质量的数据集很昂贵且耗时，因此培训数据集的规模通常很小（例如，Sighan15仅包含2339个用于培训的样本），因此基于学习的模型通常会遭受数据稀疏限制。和过度合适的问题，尤其是在大语言模型时代。在本文中，我们致力于研究\ textbf {无监督}范式来解决CSC问题，我们提出了一个名为\ textbf {uchecker}的框架，以进行无监督的拼写错误检测和校正。考虑到其强大的语言诊断能力，将蒙面审慎的语言模型（例如BERT）引入为骨干模型。从各种且灵活的掩蔽操作中受益，我们提出了一种混乱集引导的掩盖策略，以精细培训掩盖语言模型，以进一步提高无监督的检测和校正的性能。标准数据集的实验结果证明了我们提出的模型Uchecker在字符级别和句子级的准确性，精度，回忆和F1分别对拼写错误检测和校正任务分别进行的效率。

The lack of label data is one of the significant bottlenecks for Chinese Spelling Check (CSC). Existing researches use the method of automatic generation by exploiting unlabeled data to expand the supervised corpus. However, there is a big gap between the real input scenario and automatic generated corpus. Thus, we develop a competitive general speller ECSpell which adopts the Error Consistent masking strategy to create data for pretraining. This error consistency masking strategy is used to specify the error types of automatically generated sentences which is consistent with real scene. The experimental result indicates our model outperforms previous state-of-the-art models on the general benchmark. Moreover, spellers often work within a particular domain in real life. Due to lots of uncommon domain terms, experiments on our built domain specific datasets show that general models perform terribly. Inspired by the common practice of input methods, we propose to add an alterable user dictionary to handle the zero-shot domain adaption problem. Specifically, we attach a User Dictionary guided inference module (UD) to a general token classification based speller. Our experiments demonstrate that ECSpell$^{UD}$, namely ECSpell combined with UD, surpasses all the other baselines largely, even approaching the performance on the general benchmark.

由于最近的自然语言处理的进步，几种作品已经将伯特的预先接受审查的屏蔽语言模型（MLM）应用于语音识别的后校正。然而，现有的预先训练的模型仅考虑语义校正，同时忽略了单词的语音特征。因此，语义后校正将降低性能，因为在中国ASR中同音误差相当常见。在本文中，我们提出了一种集体利用了语境化表示的新方法以及错误与其替换候选人之间的语音信息来缓解中国ASR的错误率。我们对现实世界语音识别数据集的实验结果表明，我们所提出的方法明显地低于基线模型的CER，其利用预先训练的BERT MLM作为校正器。

中文拼写检查（CSC）任务旨在检测和纠正中文拼写错误。近年来，相关研究的重点是引入“混乱设置”以增强CSC模型的角色相似性，忽略了包含更丰富信息的字符的上下文。为了更好地利用上下文相似性，我们为CSC任务提供了一个简单而有效的课程学习框架。借助我们设计的模型不足框架，现有的CSC型号将从人类学习汉字并取得进一步改进的培训。对广泛使用的Sighan数据集进行了广泛的实验和详细分析表明，我们的方法的表现优于先前的最新方法。

来自变压器（BERT）的双向编码器表示显示了各种NLP任务的奇妙改进，并且已经提出了其连续的变体来进一步提高预先训练的语言模型的性能。在本文中，我们的目标是首先介绍中国伯特的全文掩蔽（WWM）策略，以及一系列中国预培训的语言模型。然后我们还提出了一种简单但有效的型号，称为Macbert，这在几种方面提高了罗伯塔。特别是，我们提出了一种称为MLM作为校正（MAC）的新掩蔽策略。为了展示这些模型的有效性，我们创建了一系列中国预先培训的语言模型，作为我们的基线，包括BERT，Roberta，Electra，RBT等。我们对十个中国NLP任务进行了广泛的实验，以评估创建的中国人托管语言模型以及提议的麦克白。实验结果表明，Macbert可以在许多NLP任务上实现最先进的表演，我们还通过几种可能有助于未来的研究的调查结果来消融细节。我们开源我们的预先培训的语言模型，以进一步促进我们的研究界。资源可用：https://github.com/ymcui/chinese-bert-wwm

孟加拉语键入大多是使用英语键盘进行的，并且由于存在化合物和类似明显的字母，因此可能是错误的。拼写错误的单词的拼写校正需要了解单词键入模式以及用法一词的上下文。我们提出了一个专业的BERT模型，Bspell针对词校正句子级别。Bspell包含一个可训练的CNN子模型，名为Semanticnet以及专门的辅助损失。这使得Bspell在存在拼写错误的情况下专门研究高度易转的孟加拉词汇。我们进一步提出了将单词级别和字符水平掩蔽组合的混合预读方案。利用这种预审前的方案，BSPELL在现实生活中的孟加拉语拼写校正验证设置中实现了91.5％的准确性。对两个孟加拉语和一个印地语拼写校正数据集进行了详细比较，显示了拟议的Bspell优于现有咒语检查器的优势。

Lexicon信息和预先训练的型号，如伯特，已被组合以探索由于各自的优势而探索中文序列标签任务。然而，现有方法通过浅和随机初始化的序列层仅熔断词典特征，并且不会将它们集成到伯特的底层中。在本文中，我们提出了用于汉语序列标记的Lexicon增强型BERT（Lebert），其直接通过Lexicon适配器层将外部词典知识集成到BERT层中。与现有方法相比，我们的模型促进了伯特下层的深层词典知识融合。关于十个任务的十个中文数据集的实验，包括命名实体识别，单词分段和言语部分标记，表明Lebert实现了最先进的结果。

Scene text spotting is of great importance to the computer vision community due to its wide variety of applications. Recent methods attempt to introduce linguistic knowledge for challenging recognition rather than pure visual classification. However, how to effectively model the linguistic rules in end-to-end deep networks remains a research challenge. In this paper, we argue that the limited capacity of language models comes from 1) implicit language modeling; 2) unidirectional feature representation; and 3) language model with noise input. Correspondingly, we propose an autonomous, bidirectional and iterative ABINet++ for scene text spotting. Firstly, the autonomous suggests enforcing explicitly language modeling by decoupling the recognizer into vision model and language model and blocking gradient flow between both models. Secondly, a novel bidirectional cloze network (BCN) as the language model is proposed based on bidirectional feature representation. Thirdly, we propose an execution manner of iterative correction for the language model which can effectively alleviate the impact of noise input. Finally, to polish ABINet++ in long text recognition, we propose to aggregate horizontal features by embedding Transformer units inside a U-Net, and design a position and content attention module which integrates character order and content to attend to character features precisely. ABINet++ achieves state-of-the-art performance on both scene text recognition and scene text spotting benchmarks, which consistently demonstrates the superiority of our method in various environments especially on low-quality images. Besides, extensive experiments including in English and Chinese also prove that, a text spotter that incorporates our language modeling method can significantly improve its performance both in accuracy and speed compared with commonly used attention-based recognizers.

Error correction in automatic speech recognition (ASR) aims to correct those incorrect words in sentences generated by ASR models. Since recent ASR models usually have low word error rate (WER), to avoid affecting originally correct tokens, error correction models should only modify incorrect words, and therefore detecting incorrect words is important for error correction. Previous works on error correction either implicitly detect error words through target-source attention or CTC (connectionist temporal classification) loss, or explicitly locate specific deletion/substitution/insertion errors. However, implicit error detection does not provide clear signal about which tokens are incorrect and explicit error detection suffers from low detection accuracy. In this paper, we propose SoftCorrect with a soft error detection mechanism to avoid the limitations of both explicit and implicit error detection. Specifically, we first detect whether a token is correct or not through a probability produced by a dedicatedly designed language model, and then design a constrained CTC loss that only duplicates the detected incorrect tokens to let the decoder focus on the correction of error tokens. Compared with implicit error detection with CTC loss, SoftCorrect provides explicit signal about which words are incorrect and thus does not need to duplicate every token but only incorrect tokens; compared with explicit error detection, SoftCorrect does not detect specific deletion/substitution/insertion errors but just leaves it to CTC loss. Experiments on AISHELL-1 and Aidatatang datasets show that SoftCorrect achieves 26.1% and 9.4% CER reduction respectively, outperforming previous works by a large margin, while still enjoying fast speed of parallel generation.

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a;Radford et al., 2018), BERT is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial taskspecific architecture modifications.BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

While pre-trained Chinese language models have demonstrated impressive performance on a wide range of NLP tasks, the Chinese Spell Checking (CSC) task remains a challenge. Previous research has explored using information such as glyphs and phonetics to improve the ability to distinguish misspelled characters, with good results. However, the generalization ability of these models is not well understood: it is unclear whether they incorporate glyph-phonetic information and, if so, whether this information is fully utilized. In this paper, we aim to better understand the role of glyph-phonetic information in the CSC task and suggest directions for improvement. Additionally, we propose a new, more challenging, and practical setting for testing the generalizability of CSC models. All code is made publicly available.

事实证明，将先验知识纳入预训练的语言模型中对知识驱动的NLP任务有效，例如实体键入和关系提取。当前的培训程序通常通过使用知识掩盖，知识融合和知识更换将外部知识注入模型。但是，输入句子中包含的事实信息尚未完全开采，并且尚未严格检查注射的外部知识。结果，无法完全利用上下文信息，并将引入额外的噪音，或者注入的知识量受到限制。为了解决这些问题，我们提出了MLRIP，该MLRIP修改了Ernie-Baidu提出的知识掩盖策略，并引入了两阶段的实体替代策略。进行全面分析的广泛实验说明了MLRIP在军事知识驱动的NLP任务中基于BERT的模型的优势。

预训练的语言模型（PLM）在自然语言理解中的许多下游任务中取得了显着的性能增长。已提出了各种中文PLM，以学习更好的中文表示。但是，大多数当前模型都使用中文字符作为输入，并且无法编码中文单词中包含的语义信息。虽然最近的预训练模型同时融合了单词和字符，但它们通常会遭受不足的语义互动，并且无法捕获单词和字符之间的语义关系。为了解决上述问题，我们提出了一个简单而有效的PLM小扣手，该小扣子采用了对单词和性格表示的对比度学习。特别是，Clower通过对多透明信息的对比学习将粗粒的信息（即单词）隐式编码为细粒度表示（即字符）。在现实的情况下，小电动器具有很大的价值，因为它可以轻松地将其纳入任何现有的基于细粒的PLM中而无需修改生产管道。在一系列下游任务上进行的扩展实验表明，小动物的卓越性能超过了几个最先进的实验 - 艺术基线。

Language model pre-training has proven to be useful in learning universal language representations. As a state-of-the-art language model pre-training model, BERT (Bidirectional Encoder Representations from Transformers) has achieved amazing results in many language understanding tasks. In this paper, we conduct exhaustive experiments to investigate different fine-tuning methods of BERT on text classification task and provide a general solution for BERT fine-tuning. Finally, the proposed solution obtains new state-of-the-art results on eight widely-studied text classification datasets. 1

Due to the ambiguity of homophones, Chinese Spell Checking (CSC) has widespread applications. Existing systems typically utilize BERT for text encoding. However, CSC requires the model to account for both phonetic and graphemic information. To adapt BERT to the CSC task, we propose a token-level self-distillation contrastive learning method. We employ BERT to encode both the corrupted and corresponding correct sentence. Then, we use contrastive learning loss to regularize corrupted tokens' hidden states to be closer to counterparts in the correct sentence. On three CSC datasets, we confirmed our method provides a significant improvement above baselines.

拼写错误纠正是自然语言处理中具有很长历史的主题之一。虽然以前的研究取得了显着的结果，但仍然存在挑战。在越南语中，任务的最先进的方法从其相邻音节中介绍了一个音节的上下文。然而，该方法的准确性可能是不令人满意的，因为如果模型可能会失去上下文，如果两个（或更多）拼写错误彼此静置。在本文中，我们提出了一种纠正越南拼写错误的新方法。我们使用深入学习模型解决错误错误和拼写错误错误的问题。特别地，嵌入层由字节对编码技术提供支持。基于变压器架构的序列模型的序列使我们的方法与上一个问题不同于同一问题的方法。在实验中，我们用大型合成数据集训练模型，这是随机引入的拼写错误。我们使用现实数据集测试所提出的方法的性能。此数据集包含11,202个以9,341不同的越南句子中的人造拼写错误。实验结果表明，我们的方法达到了令人鼓舞的表现，检测到86.8％的误差，81.5％纠正，分别提高了最先进的方法5.6％和2.2％。

Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding. We argue that informative entities in KGs can enhance language representation with external knowledge. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. The experimental results have demonstrated that ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks. The source code and experiment details of this paper can be obtained from https:// github.com/thunlp/ERNIE.

我们利用预训练的语言模型来解决两种低资源语言的复杂NER任务：中文和西班牙语。我们使用整个单词掩码（WWM）的技术来提高大型和无监督的语料库的掩盖语言建模目标。我们在微调的BERT层之上进行多个神经网络体系结构，将CRF，Bilstms和线性分类器结合在一起。我们所有的模型都优于基线，而我们的最佳性能模型在盲目测试集的评估排行榜上获得了竞争地位。

近年来，基于变压器的预训练模型已获得了很大的进步，成为自然语言处理中最重要的骨干之一。最近的工作表明，变压器内部的注意力机制可能不需要，卷积神经网络和基于多层感知器的模型也已被研究为变压器替代方案。在本文中，我们考虑了一个用于语言模型预训练的图形循环网络，该网络通过本地令牌级通信为每个序列构建一个图形结构，以及与其他代币解耦的句子级表示。原始模型在受监督培训下的特定领域特定文本分类中表现良好，但是，其通过自我监督的方式学习转移知识的潜力尚未得到充分利用。我们通过优化体系结构并验证其在更通用的语言理解任务（英语和中文）中的有效性来填补这一空白。至于模型效率，我们的模型在基于变压器的模型中而不是二次复杂性，而是具有线性复杂性，并且在推断过程中的性能更有效。此外，我们发现与现有基于注意力的模型相比，我们的模型可以生成更多样化的输出，而背景化的功能冗余性较小。

Spelling error correction is the task of identifying and rectifying misspelled words in texts. It is a potential and active research topic in Natural Language Processing because of numerous applications in human language understanding. The phonetically or visually similar yet semantically distinct characters make it an arduous task in any language. Earlier efforts on spelling error correction in Bangla and resource-scarce Indic languages focused on rule-based, statistical, and machine learning-based methods which we found rather inefficient. In particular, machine learning-based approaches, which exhibit superior performance to rule-based and statistical methods, are ineffective as they correct each character regardless of its appropriateness. In this work, we propose a novel detector-purificator-corrector framework based on denoising transformers by addressing previous issues. Moreover, we present a method for large-scale corpus creation from scratch which in turn resolves the resource limitation problem of any left-to-right scripted language. The empirical outcomes demonstrate the effectiveness of our approach that outperforms previous state-of-the-art methods by a significant margin for Bangla spelling error correction. The models and corpus are publicly available at https://tinyurl.com/DPCSpell.