最近的参数效率语言模型调整（PELT）方法可以使微调的性能与较少的可训练参数相匹配，并且在训练数据受到限制时尤其表现良好。但是，不同的PELT方法在相同的任务上的性能可能会有所不同，因此为特定任务选择最合适的方法是不平凡的，尤其是考虑到快速增长的新PELT方法和任务。鉴于模型多样性和模型选择的难度，我们提出了一个统一的框架Unipelt，该框架将不同的毛皮方法纳入了子模型，并学会了激活最适合当前数据或通过门控机制设置的方法。在胶水基准上，与最佳的单个毛皮方法相比，UniPelt始终达到1〜4％的增长，而其融合甚至超过了不同设置下的微调。此外，UniPelt通常超过上限，该上限在每个任务上单独使用的所有子模型的最佳性能，表明多种PELT方法的混合物可能本质上比单个方法更有效。

微调下游任务的大型预训练语言模型已成为NLP中的事实上学习范式。然而，常规方法微调预先训练模型的所有参数，这变得越来越稳定，因为模型尺寸和增长的任务数量。最近的工作提出了各种参数有效的转移学习方法，只需微调少数（额外）参数以获得强大的性能。虽然有效，但各种方法中的成功和联系的关键成分尚不清楚。在本文中，我们分解了最先进的参数有效的传输学习方法的设计，并提出了一个在它们之间建立连接的统一框架。具体而言，我们将它们重新框架作为预先训练的模型对特定隐藏状态的修改，并定义了一组设计尺寸，不同的方法变化，例如计算修改的功能和应用修改的位置。通过跨机翻译的全面实证研究，文本摘要，语言理解和文本分类基准，我们利用统一的视图来确定以前的方法中的重要设计选择。此外，我们的统一框架使得能够在不同的方法中传输设计元素，因此我们能够实例化新的参数高效的微调方法，该方法比以前的方法更加有效，而是更有效，实现可比的结果在所有四个任务上调整所有参数。

通过微调将大规模的预训练语言模型适应下游任务是实现NLP基准测试最先进性能的标准方法。然而，微调具有数百万或数十亿个参数的所有重量模型是对低资源设置中不稳定的采样低效，并且浪费，因为它需要为每个任务存储模型的单独副本。最近的工作已经开发了参数高效的微调方法，但这些方法仍然需要相对大量的参数或表现不足标准微调。在这项工作中，我们提出了一种特殊调整大型语言模型的方法，其在任务性能和比率参数之间具有更好的权衡的方法，而不是比上事先工作。 Compacter通过构建适配器，低级优化和参数化超复分乘法层的思想之上来实现这一目标。具体地，Compacter将特定于特定的权重矩阵插入到预估计模型的权重中，这些权重被有效地计算为共享的“慢速”权重和“快速”等级 - 每个Compacter层定义的矩阵之间的矩阵产品的总和。仅通过培训0.047％的预磨料模型的参数，Compacter会在胶水上标准微调和胜过标准微调的标准微调和低资源设置。我们的代码在〜\ url {https://github.com/rabeehk/compacter}上公开使用。

Recently, a large number of tuning strategies have been proposed to adapt pre-trained language models to downstream tasks. In this paper, we perform an extensive empirical evaluation of various tuning strategies for multilingual learning, particularly in the context of text summarization. Specifically, we explore the relative advantages of three families of multilingual tuning strategies (a total of five models) and empirically evaluate them for summarization over 45 languages. Experimentally, we not only established a new state-of-the-art on the XL-Sum dataset but also derive a series of observations that hopefully can provide hints for future research on the design of multilingual tuning strategies.

Conventional fine-tuning encounters increasing difficulties given the size of current Pre-trained Language Models, which makes parameter-efficient tuning become the focal point of frontier research. Previous methods in this field add tunable adapters into MHA or/and FFN of Transformer blocks to enable PLMs achieve transferability. However, as an important part of Transformer architecture, the power of layer normalization for parameter-efficent tuning is ignored. In this paper, we first propose LN-tuning, by tuning the gain and bias term of Layer Normalization module with only 0.03\% parameters, which is of high time-efficency and significantly superior to baselines which are less than 0.1\% tunable parameters. Further, we study the unified framework of combining LN-tuning with previous ones and we find that: (1) the unified framework of combining prefix-tuning, the adapter-based method working on MHA, and LN-tuning achieves SOTA performance. (2) unified framework which tunes MHA and LayerNorm simultaneously can get performance improvement but those which tune FFN and LayerNorm simultaneous will cause performance decrease. Ablation study validates LN-tuning is of no abundant parameters and gives a further understanding of it.

激活功能可以对降低输入数据的拓扑复杂性产生重大影响，从而提高模型的性能。选择合适的激活函数是神经模型设计中的重要步骤。但是，在基于变压器的语言模型中很少讨论或探索激活功能的选择。事先选择它们的激活功能，然后从预训练中固定到微调。结果，在这个漫长的生命周期中，无法调整它们对模型的电感偏见。此外，随后开发的模型（例如Roberta，Bart和GPT-3）经常跟进先前的工作（例如BERT），以使用相同的激活函数而无需合理。在本文中，我们研究了变压器体系结构中使用理性激活函数（RAF）（RAF）的有效性。与常规，预定义的激活功能相反，RAF可以根据输入数据自适应地学习最佳激活功能。我们的实验表明，基于RAF的变压器（RAFT）比具有GELU函数的香草BERT的验证性更低。我们进一步评估了低和全数据设置中下游任务的筏。我们的结果表明，筏在大多数任务和设置上都优于对应模型。例如，在低数据表情况下（有100个训练示例），木筏在胶水基准上的表现平均高出5.71点，在全数据设置的小队中，平均得分为2.05分。对学到的RAF的形状的分析进一步揭示了它们在预训练模型的不同层之间有很大的变化，并且看起来与常规激活函数大多不同。 RAFT为根据学习的激活功能打开了一个新的研究方向，用于分析和解释预训练的模型。

Fine-tuning a Pre-trained Language Model (PLM) on a specific downstream task has been a well-known paradigm in Natural Language Processing. However, with the ever-growing size of PLMs, training the entire model on several downstream tasks becomes very expensive and resource-hungry. Recently, different Parameter Efficient Tuning (PET) techniques are proposed to improve the efficiency of fine-tuning PLMs. One popular category of PET methods is the low-rank adaptation methods which insert learnable truncated SVD modules into the original model either sequentially or in parallel. However, low-rank decomposition suffers from limited representation power. In this work, we address this problem using the Kronecker product instead of the low-rank representation. We introduce KronA, a Kronecker product-based adapter module for efficient fine-tuning of Transformer-based PLMs. We apply the proposed methods for fine-tuning T5 on the GLUE benchmark to show that incorporating the Kronecker-based modules can outperform state-of-the-art PET methods.

Parameter-efficient methods (like Prompt or Adapters) for adapting pre-trained language models to downstream tasks have been popular recently. However, hindrances still prevent these methods from reaching their full potential. For example, two significant challenges are few-shot adaptation and cross-task generalization ability. To tackle these issues, we propose a general framework to enhance the few-shot adaptation and cross-domain generalization ability of parameter-efficient methods. In our framework, we prime the self-supervised model for parameter-efficient methods to rapidly adapt to various downstream few-shot tasks. To evaluate the authentic generalization ability of these parameter-efficient methods, we conduct experiments on a few-shot cross-domain benchmark containing 160 diverse NLP tasks. The experiment result reveals that priming by tuning PLM only with extra training tasks leads to the best performance. Also, we perform a comprehensive analysis of various parameter-efficient methods under few-shot cross-domain scenarios.

大多数NER方法都依赖于广泛的标记数据进行模型培训，这些数据在低资源场景中挣扎，培训数据有限。与资源丰富的源域相比，现有的主要方法通常会遇到目标域具有不同标签集的挑战，该标签集可以作为类传输和域转移得出的结论。在本文中，我们通过可拔出的提示（Lightner）提出了一个轻巧的调整范式，用于低资源。具体而言，我们构建了实体类别的统一可学习的语言器，以生成实体跨度序列和实体类别，而无需任何标签特定的分类器，从而解决了类转移问题。我们通过将可学习的参数纳入自我发言层作为指导，进一步提出了一个可插入的指导模块，该参数可以重新调节注意力并调整预训练的权重。请注意，我们仅通过修复了预训练的语言模型的整个参数来调整那些插入的模块，从而使我们的方法轻巧且灵活地适合低资源场景，并且可以更好地跨域传输知识。实验结果表明，Lightner可以在标准监督环境中获得可比的性能，并且在低资源设置中优于强大基线。代码在https://github.com/zjunlp/deepke/tree/main/main/example/ner/few-shot中。

最近，在大型文本语料库上预先培训的微调语言模型已经为Vision-and Langual（V＆L）任务以及纯语言任务提供了巨大的改进。但是，微调预训练模型的整个参数集变得不切实际，因为模型大小正在快速增长。因此，在本文中，我们将基于适配器的参数高效转移学习技术引入VL-BART和VL-T5等V＆L型号。我们在四个不同V＆L任务的统一多任务设置中评估我们的方法：VQAV2，GQA，NLVR2和MSCOCO图像标题。通过仔细的培训和彻底的实验，我们将三种流行的基于适配器的方法（适配器，Hyperformer，Compacter）基准，抵御标准的全部微调和最近提出的及时调整方法。我们还通过分享其权重以获得跨任务的知识来增强适配器的效率和性能。我们的结果表明，使用权重共享技术（总参数的4.4％）培训适配器可以匹配微调整个模型的性能。最后，我们提出了一个全面的分析，包括适配器和任务特定提示的组合以及V＆L对适配器进行培训的影响。我们的代码可用于：https://github.com/ylsung/vl_adapter。

This work introduces a new multi-task, parameter-efficient language model (LM) tuning method that learns to transfer knowledge across different tasks via a mixture of soft prompts-small prefix embedding vectors pre-trained for different tasks. Our method, called ATTEMPT (ATTEntional Mixtures of Prompt Tuning), obtains source prompts as encodings of large-scale source tasks into a small number of parameters and trains an attention module to interpolate the source prompts and a newly initialized target prompt for every instance in the target task. During training, only the target task prompt and the attention weights, which are shared between tasks in multi-task training, are updated, while the original LM and source prompts are intact. ATTEMPT is highly parameter-efficient (e.g., updates 2,300 times fewer parameters than full fine-tuning) while achieving high task performance using knowledge from high-resource tasks. Moreover, it is modular using pre-trained soft prompts, and can flexibly add or remove source prompts for effective knowledge transfer. Our experimental results across 21 diverse NLP datasets show that ATTEMPT significantly outperforms prompt tuning and outperforms or matches fully fine-tuned or other parameter-efficient tuning approaches that use over ten times more parameters. Finally, ATTEMPT outperforms previous work in few-shot learning settings.

Adapter Tuning, which freezes the pretrained language models (PLMs) and only fine-tunes a few extra modules, becomes an appealing efficient alternative to the full model fine-tuning. Although computationally efficient, the recent Adapters often increase parameters (e.g. bottleneck dimension) for matching the performance of full model fine-tuning, which we argue goes against their original intention. In this work, we re-examine the parameter-efficiency of Adapters through the lens of network pruning (we name such plug-in concept as \texttt{SparseAdapter}) and find that SparseAdapter can achieve comparable or better performance than standard Adapters when the sparse ratio reaches up to 80\%. Based on our findings, we introduce an easy but effective setting ``\textit{Large-Sparse}'' to improve the model capacity of Adapters under the same parameter budget. Experiments on five competitive Adapters upon three advanced PLMs show that with proper sparse method (e.g. SNIP) and ratio (e.g. 40\%) SparseAdapter can consistently outperform their corresponding counterpart. Encouragingly, with the \textit{Large-Sparse} setting, we can obtain further appealing gains, even outperforming the full fine-tuning by a large margin. Our code will be released at: https://github.com/Shwai-He/SparseAdapter.

具有数百万参数的基于变压器的预训练模型需要大量存储。最近的方法通过培训适配器解决了这一缺点，但是这些方法仍然需要相对较大的参数。在这项研究中，提出了一种令人惊讶的简单但有效的适配器体系结构的Adapterbias。AdapterBias向变压器层的隐藏输出添加了代币依赖性转移，以适应仅使用向量和线性层的下游任务。进行了广泛的实验，以证明适配性的有效性。实验表明，与先前的作品相比，我们提出的方法可以大大减少可训练的参数，而任务性能与微调的预训练模型相比最小。我们进一步发现，适应性比亚斯自动学习以将更重要的表示形式分配给与任务相关的代币转移。

本文探讨了时间视频接地（TVG）的任务，在该任务中，给定未修剪的视频和查询句子，目标是在提供的自然语言查询描述的视频中识别和确定动作实例的时间界。最近的作品通过使用大型预训练的语言模型（PLM）直接编码查询来解决此任务。但是，很难隔离改进的语言表示的影响，因为这些作品还提出了视觉输入的改进。此外，这些PLM大大增加了训练TVG模型的计算成本。因此，本文研究了PLM在TVG任务中的影响，并根据适配器评估了NLP参数效率培训替代方案的适用性。我们将流行的PLM与选择现有方法和测试不同的适配器相结合，以减少其他参数的影响。我们在三个具有挑战性的数据集上的结果表明，当TVG模型对该任务进行微调时，可以从PLM中受益匪浅，并且适配器是完全微调的有效替代方法，即使它们并不适合我们的任务。具体而言，适配器有助于节省计算成本，从而使PLM集成在较大的TVG模型中，并提供与最先进模型相当的结果。最后，通过对TVG中不同类型的适配器进行基准测试，我们的结果阐明了哪种适配器最适合每个研究的情况。

In computer vision, it has achieved great transfer learning performance via adapting large-scale pretrained vision models (e.g., vision transformers) to downstream tasks. Common approaches for model adaptation either update all model parameters or leverage linear probes. In this paper, we aim to study parameter-efficient model adaptation strategies for vision transformers on the image classification task. We formulate efficient model adaptation as a subspace training problem and perform a comprehensive benchmarking over different efficient adaptation methods. We conduct an empirical study on each efficient model adaptation method focusing on its performance alongside parameter cost. Furthermore, we propose a parameter-efficient model adaptation framework, which first selects submodules by measuring local intrinsic dimensions and then projects them into subspace for further decomposition via a novel Kronecker Adaptation (KAdaptation) method. We analyze and compare our method with a diverse set of baseline model adaptation methods (including state-of-the-art methods for pretrained language models). Our method performs the best in terms of the tradeoff between accuracy and parameter efficiency across 20 image classification datasets under the few-shot setting and 7 image classification datasets under the full-shot setting.

Parameter-efficient fine-tuning (PEFT) methods can adapt large language models to downstream tasks by training a small amount of newly added parameters. In multi-task settings, PEFT adapters typically train on each task independently, inhibiting transfer across tasks, or on the concatenation of all tasks, which can lead to negative interference. To address this, Polytropon (Ponti et al.) jointly learns an inventory of PEFT adapters and a routing function to share variable-size sets of adapters across tasks. Subsequently, adapters can be re-combined and fine-tuned on novel tasks even with limited data. In this paper, we investigate to what extent the ability to control which adapters are active for each task leads to sample-efficient generalization. Thus, we propose less expressive variants where we perform weighted averaging of the adapters before few-shot adaptation (Poly-mu) instead of learning a routing function. Moreover, we introduce more expressive variants where finer-grained task-adapter allocation is learned through a multi-head routing function (Poly-S). We test these variants on three separate benchmarks for multi-task learning. We find that Poly-S achieves gains on all three (up to 5.3 points on average) over strong baselines, while incurring a negligible additional cost in parameter count. In particular, we find that instruction tuning, where models are fully fine-tuned on natural language instructions for each task, is inferior to modular methods such as Polytropon and our proposed variants.

Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task. On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.

神经桌面到文本的生成方法是渴望数据的，限制了它们对低资源现实世界应用的适应性。先前的工作主要诉诸于训练的语言模型（PLM），以生成表格的表格摘要。但是，由于PLM的性质不受控制，它们通常包含幻觉内容。此外，很少研究表和序列之间的拓扑差异。最后但并非最不重要的一点是，在PLM上进行少量实例进行微调可能会导致过度贴合和灾难性的遗忘。为了减轻这些问题，我们提出了一种基于及时的方法，前缀控制的发电机（即PCG），用于几乎没有表格到文本的生成。我们为PLM的特定于任务的前缀预备，以使表结构更适合预训练的输入。此外，我们生成一个特定于输入的前缀，以控制生成的文本的事实内容和单词顺序。对Wikibio数据集的不同领域（人类，书籍和歌曲）的自动评估和人类评估都显示出对基线方法的实质性改进。

当前的Modus Operandi在改编预训练的模型中涉及更新所有骨干参数，即，完整的微调。本文介绍了视觉及时调整（VPT），作为视觉中大规模变压器模型的全面微调的有效替代方案。VPT从最近有效地调整大型语言模型的最新进展中汲取灵感，在输入空间中仅引入了少量的可训练参数（少于模型参数），同时保持模型骨架冻结。通过对各种下游识别任务的广泛实验，我们表明VPT与其他参数有效调整协议相比获得了显着的性能增长。最重要的是，在许多情况下，VPT甚至在模型能力和培训数据量表的许多情况下都胜过全面的微调，同时降低了每任务的存储成本。

With increasing privacy concerns on data, recent studies have made significant progress using federated learning (FL) on privacy-sensitive natural language processing (NLP) tasks. Much literature suggests fully fine-tuning pre-trained language models (PLMs) in the FL paradigm can mitigate the data heterogeneity problem and close the performance gap with centralized training. However, large PLMs bring the curse of prohibitive communication overhead and local model adaptation costs for the FL system. To this end, we introduce various parameter-efficient tuning (PETuning) methods into federated learning. Specifically, we provide a holistic empirical study of representative PLMs tuning methods in FL. The experimental results cover the analysis of data heterogeneity levels, data scales, and different FL scenarios. Overall communication overhead can be significantly reduced by locally tuning and globally aggregating lightweight model parameters while maintaining acceptable performance in various FL settings. To facilitate the research of PETuning in FL, we also develop a federated tuning framework FedPETuning, which allows practitioners to exploit different PETuning methods under the FL training paradigm conveniently. The source code is available at \url{https://github.com/iezhuozhuo/FedETuning/tree/deltaTuning}.