基于蒸馏的压缩网络的性能受蒸馏质量的管辖。大型网络（教师）到较小网络（学生）的次优蒸馏的原因主要归因于给定教师与学生对的学习能力中的差距。虽然很难蒸馏所有教师的知识，但可以在很大程度上控制蒸馏质量以实现更好的性能。我们的实验表明，蒸馏品质主要受教师响应的质量来限制，这反过来又受到其反应中存在相似信息的影响。训练有素的大容量老师在学习细粒度辨别性质的过程中丢失了类别之间的相似性信息。没有相似性信息导致蒸馏过程从一个例子 - 许多阶级学习减少到一个示例 - 一类学习，从而限制了教师的不同知识的流程。由于隐式假设只能蒸馏出灌输所知，而不是仅关注知识蒸馏过程，我们仔细审查了知识序列过程。我们认为，对于给定的教师 - 学生对，通过在训练老师的同时找到批量大小和时代数量之间的甜蜜点，可以提高蒸馏品。我们讨论了找到这种甜蜜点以便更好地蒸馏的步骤。我们还提出了蒸馏假设，以区分知识蒸馏和正则化效果之间的蒸馏过程的行为。我们在三个不同的数据集中进行我们的所有实验。

Knowledge distillation (KD) has gained a lot of attention in the field of model compression for edge devices thanks to its effectiveness in compressing large powerful networks into smaller lower-capacity models. Online distillation, in which both the teacher and the student are learning collaboratively, has also gained much interest due to its ability to improve on the performance of the networks involved. The Kullback-Leibler (KL) divergence ensures the proper knowledge transfer between the teacher and student. However, most online KD techniques present some bottlenecks under the network capacity gap. By cooperatively and simultaneously training, the models the KL distance becomes incapable of properly minimizing the teacher's and student's distributions. Alongside accuracy, critical edge device applications are in need of well-calibrated compact networks. Confidence calibration provides a sensible way of getting trustworthy predictions. We propose BD-KD: Balancing of Divergences for online Knowledge Distillation. We show that adaptively balancing between the reverse and forward divergences shifts the focus of the training strategy to the compact student network without limiting the teacher network's learning process. We demonstrate that, by performing this balancing design at the level of the student distillation loss, we improve upon both performance accuracy and calibration of the compact student network. We conducted extensive experiments using a variety of network architectures and show improvements on multiple datasets including CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet. We illustrate the effectiveness of our approach through comprehensive comparisons and ablations with current state-of-the-art online and offline KD techniques.

Figure 1. An illustration of standard knowledge distillation. Despite widespread use, an understanding of when the student can learn from the teacher is missing.

机器学习中的知识蒸馏是将知识从名为教师的大型模型转移到一个名为“学生”的较小模型的过程。知识蒸馏是将大型网络（教师）压缩到较小网络（学生）的技术之一，该网络可以部署在手机等小型设备中。当教师和学生之间的网络规模差距增加时，学生网络的表现就会下降。为了解决这个问题，在教师模型和名为助教模型的学生模型之间采用了中间模型，这反过来弥补了教师与学生之间的差距。在这项研究中，我们已经表明，使用多个助教模型，可以进一步改进学生模型（较小的模型）。我们使用加权集合学习将这些多个助教模型组合在一起，我们使用了差异评估优化算法来生成权重值。

尽管深层神经网络在各种任务中取得了巨大的成功，但它们不断增加的规模也为部署带来了重要的开销。为了压缩这些模型，提出了知识蒸馏将知识从笨拙（教师）网络转移到轻量级（学生）网络中。但是，老师的指导并不总是改善学生的概括，尤其是当学生和老师之间的差距很大时。以前的作品认为，这是由于老师的高确定性，导致更难适应的标签。为了软化这些标签，我们提出了一种修剪方法，称为预测不确定性扩大（PRUE），以简化教师。具体而言，我们的方法旨在减少教师对数据的确定性，从而为学生产生软预测。我们从经验上研究了提出的方法通过在CIFAR-10/100，Tiny-Imagenet和Imagenet上实验的实验的有效性。结果表明，接受稀疏教师培训的学生网络取得更好的表现。此外，我们的方法允许研究人员从更深的网络中提取知识，以进一步改善学生。我们的代码公开：\ url {https://github.com/wangshaopu/prue}。

最初引入了知识蒸馏，以利用来自单一教师模型的额外监督为学生模型培训。为了提高学生表现，最近的一些变体试图利用多个教师利用不同的知识来源。然而，现有研究主要通过对多种教师预测的平均或将它们与其他无标签策略相结合，将知识集成在多种来源中，可能在可能存在低质量的教师预测存在中误导学生。为了解决这个问题，我们提出了信心感知的多教师知识蒸馏（CA-MKD），该知识蒸馏（CA-MKD）在地面真理标签的帮助下，适用于每个教师预测的样本明智的可靠性，与那些接近单热的教师预测标签分配了大量的重量。此外，CA-MKD包含中间层，以进一步提高学生表现。广泛的实验表明，我们的CA-MKD始终如一地优于各种教师学生架构的所有最先进的方法。

Knowledge Distillation (KD) aims to distill the knowledge of a cumbersome teacher model into a lightweight student model. Its success is generally attributed to the privileged information on similarities among categories provided by the teacher model, and in this sense, only strong teacher models are deployed to teach weaker students in practice. In this work, we challenge this common belief by following experimental observations: 1) beyond the acknowledgment that the teacher can improve the student, the student can also enhance the teacher significantly by reversing the KD procedure; 2) a poorly-trained teacher with much lower accuracy than the student can still improve the latter significantly. To explain these observations, we provide a theoretical analysis of the relationships between KD and label smoothing regularization. We prove that 1) KD is a type of learned label smoothing regularization and 2) label smoothing regularization provides a virtual teacher model for KD. From these results, we argue that the success of KD is not fully due to the similarity information between categories from teachers, but also to the regularization of soft targets, which is equally or even more important.Based on these analyses, we further propose a novel Teacher-free Knowledge Distillation (Tf-KD) framework, where a student model learns from itself or manuallydesigned regularization distribution. The Tf-KD achieves comparable performance with normal KD from a superior teacher, which is well applied when a stronger teacher model is unavailable. Meanwhile, Tf-KD is generic and can be directly deployed for training deep neural networks. Without any extra computation cost, Tf-KD achieves up to 0.65% improvement on ImageNet over well-established baseline models, which is superior to label smoothing regularization.

Mixup is a popular data augmentation technique based on creating new samples by linear interpolation between two given data samples, to improve both the generalization and robustness of the trained model. Knowledge distillation (KD), on the other hand, is widely used for model compression and transfer learning, which involves using a larger network's implicit knowledge to guide the learning of a smaller network. At first glance, these two techniques seem very different, however, we found that ``smoothness" is the connecting link between the two and is also a crucial attribute in understanding KD's interplay with mixup. Although many mixup variants and distillation methods have been proposed, much remains to be understood regarding the role of a mixup in knowledge distillation. In this paper, we present a detailed empirical study on various important dimensions of compatibility between mixup and knowledge distillation. We also scrutinize the behavior of the networks trained with a mixup in the light of knowledge distillation through extensive analysis, visualizations, and comprehensive experiments on image classification. Finally, based on our findings, we suggest improved strategies to guide the student network to enhance its effectiveness. Additionally, the findings of this study provide insightful suggestions to researchers and practitioners that commonly use techniques from KD. Our code is available at https://github.com/hchoi71/MIX-KD.

这项工作研究了标签平滑（LS）和知识蒸馏（KD）之间的兼容性。解决这一论文陈述的当代发现采取二分法的观点：Muller等。 （2019）和Shen等。 （2021b）。至关重要的是，没有努力理解和解决这些矛盾的发现，留下了原始问题 - 顺利还是不平稳教师网络？ - 未得到答复。我们工作的主要贡献是对系统扩散的发现，分析和验证是缺失的概念，这在理解和解决这些矛盾的发现方面具有重要作用。这种系统的扩散基本上削减了从LS训练的老师蒸馏的好处，从而使KD在升高的温度无效时使KD呈现。我们的发现得到了大规模实验，分析和案例研究的全面支持，包括图像分类，神经机器翻译和紧凑的学生蒸馏任务，这些任务跨越了多个数据集和教师 - 学生架构。根据我们的分析，我们建议从业者使用具有低温转移的LS训练的老师来实现高性能学生。代码和型号可在https://keshik6.github.io/revisiting-ls-kd-compatibility/

Knowledge Distillation (KD) has been extensively used for natural language understanding (NLU) tasks to improve a small model's (a student) generalization by transferring the knowledge from a larger model (a teacher). Although KD methods achieve state-of-the-art performance in numerous settings, they suffer from several problems limiting their performance. It is shown in the literature that the capacity gap between the teacher and the student networks can make KD ineffective. Additionally, existing KD techniques do not mitigate the noise in the teacher's output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features. We propose a new KD method that addresses these problems and facilitates the training compared to previous techniques. Inspired by continuation optimization, we design a training procedure that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds. Our method (Continuation-KD) achieves state-of-the-art performance across various compact architectures on NLU (GLUE benchmark) and computer vision tasks (CIFAR-10 and CIFAR-100).

Despite the fact that deep neural networks are powerful models and achieve appealing results on many tasks, they are too large to be deployed on edge devices like smartphones or embedded sensor nodes. There have been efforts to compress these networks, and a popular method is knowledge distillation, where a large (teacher) pre-trained network is used to train a smaller (student) network. However, in this paper, we show that the student network performance degrades when the gap between student and teacher is large. Given a fixed student network, one cannot employ an arbitrarily large teacher, or in other words, a teacher can effectively transfer its knowledge to students up to a certain size, not smaller. To alleviate this shortcoming, we introduce multi-step knowledge distillation, which employs an intermediate-sized network (teacher assistant) to bridge the gap between the student and the teacher. Moreover, we study the effect of teacher assistant size and extend the framework to multi-step distillation. Theoretical analysis and extensive experiments on CIFAR-10,100 and ImageNet datasets and on CNN and ResNet architectures substantiate the effectiveness of our proposed approach.

近年来，知识蒸馏有显着改善，可以为更好的效率产生紧凑的学生模型，同时保留教师模型的模型效果。以前的研究发现：由于能力不匹配，更准确的教师对更好的教师无需。在本文中，我们旨在通过模型校准的角度分析现象。我们发现较大的教师模型可能过于过度自信，因此学生模型无法有效地模仿。虽然，在教师模型的简单模型校准之后，教师模型的大小与学生模型的性能具有正相关。

知识蒸馏（KD）已广泛发展并增强了各种任务。经典的KD方法将KD损失添加到原始的跨熵（CE）损失中。我们尝试分解KD损失，以探索其与CE损失的关系。令人惊讶的是，我们发现它可以被视为CE损失和额外损失的组合，其形式与CE损失相同。但是，我们注意到额外的损失迫使学生学习教师绝对概率的相对可能性。此外，这两个概率的总和是不同的，因此很难优化。为了解决这个问题，我们修改了配方并提出分布式损失。此外，我们将教师的目标输出作为软目标，提出软损失。结合软损失和分布式损失，我们提出了新的KD损失（NKD）。此外，我们将学生的目标输出稳定，将其视为无需教师的培训的软目标，并提出了无教师的新KD损失（TF-NKD）。我们的方法在CIFAR-100和Imagenet上实现了最先进的性能。例如，以Resnet-34为老师，我们将Imagenet TOP-1的RESNET18的TOP-1精度从69.90％提高到71.96％。在没有教师的培训中，Mobilenet，Resnet-18和Swintransformer-tiny的培训占70.04％，70.76％和81.48％，分别比基线高0.83％，0.86％和0.30％。该代码可在https://github.com/yzd-v/cls_kd上找到。

在多种方式知识蒸馏研究的背景下，现有方法主要集中在唯一的学习教师最终产出问题。因此，教师网络与学生网络之间存在深处。有必要强制学生网络来学习教师网络的模态关系信息。为了有效利用从教师转移到学生的知识，采用了一种新的模型关系蒸馏范式，通过建模不同的模态之间的关系信息，即学习教师模级克矩阵。

在线知识蒸馏（OKD）通过相互利用教师和学生之间的差异来改善所涉及的模型。它们之间的差距上有几个关键的瓶颈 - 例如，为什么以及何时以及何时损害表现，尤其是对学生的表现？如何量化教师和学生之间的差距？ - 接受了有限的正式研究。在本文中，我们提出了可切换的在线知识蒸馏（Switokd），以回答这些问题。 Switokd的核心思想不是专注于测试阶段的准确性差距，而是通过两种模式之间的切换策略来适应训练阶段的差距，即蒸馏差距 - 专家模式（暂停老师，同时暂停教师保持学生学习）和学习模式（重新启动老师）。为了拥有适当的蒸馏差距，我们进一步设计了一个自适应开关阈值，该阈值提供了有关何时切换到学习模式或专家模式的正式标准，从而改善了学生的表现。同时，老师从我们的自适应切换阈值中受益，并基本上与其他在线艺术保持同步。我们进一步将Switokd扩展到具有两个基础拓扑的多个网络。最后，广泛的实验和分析验证了Switokd在最新面前的分类的优点。我们的代码可在https://github.com/hfutqian/switokd上找到。

While depth tends to improve network performances, it also makes gradient-based training more difficult since deeper networks tend to be more non-linear. The recently proposed knowledge distillation approach is aimed at obtaining small and fast-to-execute models, and it has shown that a student network could imitate the soft output of a larger teacher network or ensemble of networks. In this paper, we extend this idea to allow the training of a student that is deeper and thinner than the teacher, using not only the outputs but also the intermediate representations learned by the teacher as hints to improve the training process and final performance of the student. Because the student intermediate hidden layer will generally be smaller than the teacher's intermediate hidden layer, additional parameters are introduced to map the student hidden layer to the prediction of the teacher hidden layer. This allows one to train deeper students that can generalize better or run faster, a trade-off that is controlled by the chosen student capacity. For example, on CIFAR-10, a deep student network with almost 10.4 times less parameters outperforms a larger, state-of-the-art teacher network.

知识蒸馏（KD）是一种有效的方法，可以将知识从大型“教师”网络转移到较小的“学生”网络。传统的KD方法需要大量标记的培训样本和白盒老师（可以访问参数）才能培训好学生。但是，这些资源并不总是在现实世界应用中获得。蒸馏过程通常发生在我们无法访问大量数据的外部政党方面，并且由于安全性和隐私问题，教师没有披露其参数。为了克服这些挑战，我们提出了一种黑盒子少的KD方法，以培训学生很少的未标记培训样本和一个黑盒老师。我们的主要思想是通过使用混合和有条件的变异自动编码器生成一组不同的分布合成图像来扩展训练集。这些合成图像及其从老师获得的标签用于培训学生。我们进行了广泛的实验，以表明我们的方法在图像分类任务上明显优于最近的SOTA/零射击KD方法。代码和型号可在以下网址找到：https：//github.com/nphdang/fs-bbt

知识蒸馏（KD）是压缩边缘设备深层分类模型的有效工具。但是，KD的表现受教师和学生网络之间较大容量差距的影响。最近的方法已诉诸KD的多个教师助手（TA）设置，该设置依次降低了教师模型的大小，以相对弥合这些模型之间的尺寸差距。本文提出了一种称为“知识蒸馏”课程专家选择的新技术，以有效地增强在容量差距问题下对紧凑型学生的学习。该技术建立在以下假设的基础上：学生网络应逐渐使用分层的教学课程来逐步指导，因为它可以从较低（较高的）容量教师网络中更好地学习（硬）数据样本。具体而言，我们的方法是一种基于TA的逐渐的KD技术，它每个输入图像选择单个教师，该课程是基于通过对图像进行分类的难度驱动的课程的。在这项工作中，我们凭经验验证了我们的假设，并对CIFAR-10，CIFAR-100，CINIC-10和Imagenet数据集进行了严格的实验，并在类似VGG的模型，Resnets和WideresNets架构上显示出提高的准确性。

知识蒸馏是通过知识转移模型压缩的有效稳定的方法。传统知识蒸馏（KD）是将来自大型和训练有素的教师网络的知识转移到小型学生网络，这是一种单向过程。最近，已经提出了深度相互学习（DML）来帮助学生网络协同和同时学习。然而，据我们所知，KD和DML从未在统一的框架中共同探索，以解决知识蒸馏问题。在本文中，我们调查教师模型在KD中支持更值得信赖的监督信号，而学生则在DML中捕获教师的类似行为。基于这些观察，我们首先建议将KD与DML联合在统一的框架中。此外，我们提出了一个半球知识蒸馏（SOKD）方法，有效提高了学生和教师的表现。在这种方法中，我们在DML中介绍了同伴教学培训时尚，以缓解学生的模仿困难，并利用KD训练有素的教师提供的监督信号。此外，我们还显示我们的框架可以轻松扩展到基于功能的蒸馏方法。在CiFAR-100和Imagenet数据集上的广泛实验证明了所提出的方法实现了最先进的性能。

深度神经网络是参数化的数千或数百万个参数，并且在许多分类问题中表现出巨大的成功。然而，大量参数使得难以将这些模型集成到智能手机和可穿戴设备的边缘设备中。为了解决这个问题，知识蒸馏（KD）已被广泛采用，它使用预先训练的高容量网络来培训更小的网络，适用于边缘设备。本文首次研究了使用KD用于可穿戴设备的时间序列数据的适用性和挑战。 KD的成功应用需要在培训期间需要具体的数据增强方法。然而，如果在KD期间存在用于选择增强方法的相干策略，则尚不清楚。在本文中，我们报告了详细研究的结果，这些研究比较和对比基于KD的人类活动分析中的各种常见选择和一些混合数据增强策略。该领域的研究通常是有限的，因为公共领域没有可穿戴设备的全面数据库。我们的研究将数据库视为公共规模的数据库，以源于大规模介入研究的人类活动和久坐行为。我们发现，在KD期间的数据增强技术的选择具有对最终性能的可变影响程度，并发现最佳网络选择以及数据增强策略特定于手头的数据集。但是，我们还通过一系列关于数据库提供强大基线表现的一般建议。