知识蒸馏是将“知识”从大型模型（教师）转移到更紧凑的（学生）的过程，通常在模型压缩的背景下使用。当两个模型都具有相同的体系结构时，此过程称为自distillation。几项轶事表明，一个自灭的学生可以在持有的数据上胜过老师的表现。在这项工作中，我们系统地研究了许多设置。我们首先表明，即使有一个高度准确的老师，自我介绍也使学生在所有情况下都可以超越老师。其次，我们重新审视了（自我）蒸馏的现有理论解释，并确定矛盾的例子，揭示了这些解释的可能缺点。最后，我们通过损失景观几何形状的镜头为自我鉴定的动态提供了另一种解释。我们进行了广泛的实验，以表明自我验证会导致最小化的最小值，从而导致更好的概括。

Figure 1. An illustration of standard knowledge distillation. Despite widespread use, an understanding of when the student can learn from the teacher is missing.

知识蒸馏是一种培训小型学生网络的流行技术，以模仿更大的教师模型，例如网络的集合。我们表明，虽然知识蒸馏可以改善学生泛化，但它通常不得如此普遍地工作：虽然在教师和学生的预测分布之间，甚至在学生容量的情况下，通常仍然存在令人惊讶的差异完美地匹配老师。我们认为优化的困难是为什么学生无法与老师匹配的关键原因。我们还展示了用于蒸馏的数据集的细节如何在学生与老师匹配的紧密关系中发挥作用 - 以及教师矛盾的教师并不总是导致更好的学生泛化。

Knowledge Distillation (KD) consists of transferring "knowledge" from one machine learning model (the teacher) to another (the student). Commonly, the teacher is a high-capacity model with formidable performance, while the student is more compact. By transferring knowledge, one hopes to benefit from the student's compactness, without sacrificing too much performance. We study KD from a new perspective: rather than compressing models, we train students parameterized identically to their teachers. Surprisingly, these Born-Again Networks (BANs), outperform their teachers significantly, both on computer vision and language modeling tasks. Our experiments with BANs based on DenseNets demonstrate state-of-the-art performance on the CIFAR-10 (3.5%) and CIFAR-100 (15.5%) datasets, by validation error. Additional experiments explore two distillation objectives: (i) Confidence-Weighted by Teacher Max (CWTM) and (ii) Dark Knowledge with Permuted Predictions (DKPP). Both methods elucidate the essential components of KD, demonstrating the effect of the teacher outputs on both predicted and nonpredicted classes.

机器学习中的知识蒸馏是将知识从名为教师的大型模型转移到一个名为“学生”的较小模型的过程。知识蒸馏是将大型网络（教师）压缩到较小网络（学生）的技术之一，该网络可以部署在手机等小型设备中。当教师和学生之间的网络规模差距增加时，学生网络的表现就会下降。为了解决这个问题，在教师模型和名为助教模型的学生模型之间采用了中间模型，这反过来弥补了教师与学生之间的差距。在这项研究中，我们已经表明，使用多个助教模型，可以进一步改进学生模型（较小的模型）。我们使用加权集合学习将这些多个助教模型组合在一起，我们使用了差异评估优化算法来生成权重值。

Despite the fact that deep neural networks are powerful models and achieve appealing results on many tasks, they are too large to be deployed on edge devices like smartphones or embedded sensor nodes. There have been efforts to compress these networks, and a popular method is knowledge distillation, where a large (teacher) pre-trained network is used to train a smaller (student) network. However, in this paper, we show that the student network performance degrades when the gap between student and teacher is large. Given a fixed student network, one cannot employ an arbitrarily large teacher, or in other words, a teacher can effectively transfer its knowledge to students up to a certain size, not smaller. To alleviate this shortcoming, we introduce multi-step knowledge distillation, which employs an intermediate-sized network (teacher assistant) to bridge the gap between the student and the teacher. Moreover, we study the effect of teacher assistant size and extend the framework to multi-step distillation. Theoretical analysis and extensive experiments on CIFAR-10,100 and ImageNet datasets and on CNN and ResNet architectures substantiate the effectiveness of our proposed approach.

Knowledge Distillation (KD) aims to distill the knowledge of a cumbersome teacher model into a lightweight student model. Its success is generally attributed to the privileged information on similarities among categories provided by the teacher model, and in this sense, only strong teacher models are deployed to teach weaker students in practice. In this work, we challenge this common belief by following experimental observations: 1) beyond the acknowledgment that the teacher can improve the student, the student can also enhance the teacher significantly by reversing the KD procedure; 2) a poorly-trained teacher with much lower accuracy than the student can still improve the latter significantly. To explain these observations, we provide a theoretical analysis of the relationships between KD and label smoothing regularization. We prove that 1) KD is a type of learned label smoothing regularization and 2) label smoothing regularization provides a virtual teacher model for KD. From these results, we argue that the success of KD is not fully due to the similarity information between categories from teachers, but also to the regularization of soft targets, which is equally or even more important.Based on these analyses, we further propose a novel Teacher-free Knowledge Distillation (Tf-KD) framework, where a student model learns from itself or manuallydesigned regularization distribution. The Tf-KD achieves comparable performance with normal KD from a superior teacher, which is well applied when a stronger teacher model is unavailable. Meanwhile, Tf-KD is generic and can be directly deployed for training deep neural networks. Without any extra computation cost, Tf-KD achieves up to 0.65% improvement on ImageNet over well-established baseline models, which is superior to label smoothing regularization.

最初引入了知识蒸馏，以利用来自单一教师模型的额外监督为学生模型培训。为了提高学生表现，最近的一些变体试图利用多个教师利用不同的知识来源。然而，现有研究主要通过对多种教师预测的平均或将它们与其他无标签策略相结合，将知识集成在多种来源中，可能在可能存在低质量的教师预测存在中误导学生。为了解决这个问题，我们提出了信心感知的多教师知识蒸馏（CA-MKD），该知识蒸馏（CA-MKD）在地面真理标签的帮助下，适用于每个教师预测的样本明智的可靠性，与那些接近单热的教师预测标签分配了大量的重量。此外，CA-MKD包含中间层，以进一步提高学生表现。广泛的实验表明，我们的CA-MKD始终如一地优于各种教师学生架构的所有最先进的方法。

Knowledge distillation aims at transferring knowledge acquired in one model (a teacher) to another model (a student) that is typically smaller. Previous approaches can be expressed as a form of training the student to mimic output activations of individual data examples represented by the teacher. We introduce a novel approach, dubbed relational knowledge distillation (RKD), that transfers mutual relations of data examples instead. For concrete realizations of RKD, we propose distance-wise and angle-wise distillation losses that penalize structural differences in relations. Experiments conducted on different tasks show that the proposed method improves educated student models with a significant margin. In particular for metric learning, it allows students to outperform their teachers' performance, achieving the state of the arts on standard benchmark datasets.

Mixup is a popular data augmentation technique based on creating new samples by linear interpolation between two given data samples, to improve both the generalization and robustness of the trained model. Knowledge distillation (KD), on the other hand, is widely used for model compression and transfer learning, which involves using a larger network's implicit knowledge to guide the learning of a smaller network. At first glance, these two techniques seem very different, however, we found that ``smoothness" is the connecting link between the two and is also a crucial attribute in understanding KD's interplay with mixup. Although many mixup variants and distillation methods have been proposed, much remains to be understood regarding the role of a mixup in knowledge distillation. In this paper, we present a detailed empirical study on various important dimensions of compatibility between mixup and knowledge distillation. We also scrutinize the behavior of the networks trained with a mixup in the light of knowledge distillation through extensive analysis, visualizations, and comprehensive experiments on image classification. Finally, based on our findings, we suggest improved strategies to guide the student network to enhance its effectiveness. Additionally, the findings of this study provide insightful suggestions to researchers and practitioners that commonly use techniques from KD. Our code is available at https://github.com/hchoi71/MIX-KD.

除了使用硬标签的标准监督学习外，通常在许多监督学习设置中使用辅助损失来改善模型的概括。例如，知识蒸馏增加了第二个教师模仿模型训练的损失，在该培训中，教师可能是一个验证的模型，可以输出比标签更丰富的分布。同样，在标记数据有限的设置中，弱标记信息以标签函数的形式使用。此处引入辅助损失来对抗标签函数，这些功能可能是基于嘈杂的规则的真实标签近似值。我们解决了学习以原则性方式结合这些损失的问题。我们介绍AMAL，该AMAL使用元学习在验证度量上学习实例特定的权重，以实现损失的最佳混合。在许多知识蒸馏和规则降解域中进行的实验表明，Amal在这些领域中对竞争基准的增长可显着。我们通过经验分析我们的方法，并分享有关其提供性能提升的机制的见解。

Convolutional neural networks have been widely deployed in various application scenarios. In order to extend the applications' boundaries to some accuracy-crucial domains, researchers have been investigating approaches to boost accuracy through either deeper or wider network structures, which brings with them the exponential increment of the computational and storage cost, delaying the responding time.In this paper, we propose a general training framework named self distillation, which notably enhances the performance (accuracy) of convolutional neural networks through shrinking the size of the network rather than aggrandizing it. Different from traditional knowledge distillation -a knowledge transformation methodology among networks, which forces student neural networks to approximate the softmax layer outputs of pre-trained teacher neural networks, the proposed self distillation framework distills knowledge within network itself. The networks are firstly divided into several sections. Then the knowledge in the deeper portion of the networks is squeezed into the shallow ones. Experiments further prove the generalization of the proposed self distillation framework: enhancement of accuracy at average level is 2.65%, varying from 0.61% in ResNeXt as minimum to 4.07% in VGG19 as maximum. In addition, it can also provide flexibility of depth-wise scalable inference on resource-limited edge devices. Our codes will be released on github soon.

在实现最先进的性能和在实际应用中负担得起的大型模型之间，计算机视觉的差异越来越大。在本文中，我们解决了这个问题，并显着弥合了这两种模型之间的差距。在我们的实证研究中，我们不一定要提出一种新方法，而是要努力确定一个可靠的有效食谱，以使最先进的大型模型在实践中负担得起。我们证明，当正确执行时，知识蒸馏可以成为减少大型尺寸而不损害其性能的强大工具。特别是，我们发现存在某些隐式设计选择，这可能会严重影响蒸馏的有效性。我们的关键贡献是对这些设计选择的明确识别，这些选择以前在文献中尚未阐明。我们通过一项全面的实证研究备份了我们的发现，在广泛的视觉数据集上展示了令人信服的结果，尤其是获得了最先进的Imagenet Resnet-50模型，该模型可实现82.8％的Top-1准确性。 。

Knowledge distillation (KD) has been actively studied for image classification tasks in deep learning, aiming to improve the performance of a student based on the knowledge from a teacher. However, applying KD in image regression with a scalar response variable has been rarely studied, and there exists no KD method applicable to both classification and regression tasks yet. Moreover, existing KD methods often require a practitioner to carefully select or adjust the teacher and student architectures, making these methods less flexible in practice. To address the above problems in a unified way, we propose a comprehensive KD framework based on cGANs, termed cGAN-KD. Fundamentally different from existing KD methods, cGAN-KD distills and transfers knowledge from a teacher model to a student model via cGAN-generated samples. This novel mechanism makes cGAN-KD suitable for both classification and regression tasks, compatible with other KD methods, and insensitive to the teacher and student architectures. An error bound for a student model trained in the cGAN-KD framework is derived in this work, providing a theory for why cGAN-KD is effective as well as guiding the practical implementation of cGAN-KD. Extensive experiments on CIFAR-100 and ImageNet-100 show that we can combine state of the art KD methods with the cGAN-KD framework to yield a new state of the art. Moreover, experiments on Steering Angle and UTKFace demonstrate the effectiveness of cGAN-KD in image regression tasks, where existing KD methods are inapplicable.

尽管深层神经网络在各种任务中取得了巨大的成功，但它们不断增加的规模也为部署带来了重要的开销。为了压缩这些模型，提出了知识蒸馏将知识从笨拙（教师）网络转移到轻量级（学生）网络中。但是，老师的指导并不总是改善学生的概括，尤其是当学生和老师之间的差距很大时。以前的作品认为，这是由于老师的高确定性，导致更难适应的标签。为了软化这些标签，我们提出了一种修剪方法，称为预测不确定性扩大（PRUE），以简化教师。具体而言，我们的方法旨在减少教师对数据的确定性，从而为学生产生软预测。我们从经验上研究了提出的方法通过在CIFAR-10/100，Tiny-Imagenet和Imagenet上实验的实验的有效性。结果表明，接受稀疏教师培训的学生网络取得更好的表现。此外，我们的方法允许研究人员从更深的网络中提取知识，以进一步改善学生。我们的代码公开：\ url {https://github.com/wangshaopu/prue}。

Often we wish to transfer representational knowledge from one neural network to another. Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator. Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network. We demonstrate that this objective ignores important structural knowledge of the teacher network. This motivates an alternative objective by which we train a student to capture significantly more information in the teacher's representation of the data. We formulate this objective as contrastive learning. Experiments demonstrate that our resulting new objective outperforms knowledge distillation and other cutting-edge distillers on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer. Our method sets a new state-of-the-art in many transfer tasks, and sometimes even outperforms the teacher network when combined with knowledge distillation.

Knowledge Distillation (KD) has been extensively used for natural language understanding (NLU) tasks to improve a small model's (a student) generalization by transferring the knowledge from a larger model (a teacher). Although KD methods achieve state-of-the-art performance in numerous settings, they suffer from several problems limiting their performance. It is shown in the literature that the capacity gap between the teacher and the student networks can make KD ineffective. Additionally, existing KD techniques do not mitigate the noise in the teacher's output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features. We propose a new KD method that addresses these problems and facilitates the training compared to previous techniques. Inspired by continuation optimization, we design a training procedure that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds. Our method (Continuation-KD) achieves state-of-the-art performance across various compact architectures on NLU (GLUE benchmark) and computer vision tasks (CIFAR-10 and CIFAR-100).

Knowledge distillation (KD) has gained a lot of attention in the field of model compression for edge devices thanks to its effectiveness in compressing large powerful networks into smaller lower-capacity models. Online distillation, in which both the teacher and the student are learning collaboratively, has also gained much interest due to its ability to improve on the performance of the networks involved. The Kullback-Leibler (KL) divergence ensures the proper knowledge transfer between the teacher and student. However, most online KD techniques present some bottlenecks under the network capacity gap. By cooperatively and simultaneously training, the models the KL distance becomes incapable of properly minimizing the teacher's and student's distributions. Alongside accuracy, critical edge device applications are in need of well-calibrated compact networks. Confidence calibration provides a sensible way of getting trustworthy predictions. We propose BD-KD: Balancing of Divergences for online Knowledge Distillation. We show that adaptively balancing between the reverse and forward divergences shifts the focus of the training strategy to the compact student network without limiting the teacher network's learning process. We demonstrate that, by performing this balancing design at the level of the student distillation loss, we improve upon both performance accuracy and calibration of the compact student network. We conducted extensive experiments using a variety of network architectures and show improvements on multiple datasets including CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet. We illustrate the effectiveness of our approach through comprehensive comparisons and ablations with current state-of-the-art online and offline KD techniques.

知识蒸馏（KD）已广泛发展并增强了各种任务。经典的KD方法将KD损失添加到原始的跨熵（CE）损失中。我们尝试分解KD损失，以探索其与CE损失的关系。令人惊讶的是，我们发现它可以被视为CE损失和额外损失的组合，其形式与CE损失相同。但是，我们注意到额外的损失迫使学生学习教师绝对概率的相对可能性。此外，这两个概率的总和是不同的，因此很难优化。为了解决这个问题，我们修改了配方并提出分布式损失。此外，我们将教师的目标输出作为软目标，提出软损失。结合软损失和分布式损失，我们提出了新的KD损失（NKD）。此外，我们将学生的目标输出稳定，将其视为无需教师的培训的软目标，并提出了无教师的新KD损失（TF-NKD）。我们的方法在CIFAR-100和Imagenet上实现了最先进的性能。例如，以Resnet-34为老师，我们将Imagenet TOP-1的RESNET18的TOP-1精度从69.90％提高到71.96％。在没有教师的培训中，Mobilenet，Resnet-18和Swintransformer-tiny的培训占70.04％，70.76％和81.48％，分别比基线高0.83％，0.86％和0.30％。该代码可在https://github.com/yzd-v/cls_kd上找到。

知识蒸馏（KD）是压缩边缘设备深层分类模型的有效工具。但是，KD的表现受教师和学生网络之间较大容量差距的影响。最近的方法已诉诸KD的多个教师助手（TA）设置，该设置依次降低了教师模型的大小，以相对弥合这些模型之间的尺寸差距。本文提出了一种称为“知识蒸馏”课程专家选择的新技术，以有效地增强在容量差距问题下对紧凑型学生的学习。该技术建立在以下假设的基础上：学生网络应逐渐使用分层的教学课程来逐步指导，因为它可以从较低（较高的）容量教师网络中更好地学习（硬）数据样本。具体而言，我们的方法是一种基于TA的逐渐的KD技术，它每个输入图像选择单个教师，该课程是基于通过对图像进行分类的难度驱动的课程的。在这项工作中，我们凭经验验证了我们的假设，并对CIFAR-10，CIFAR-100，CINIC-10和Imagenet数据集进行了严格的实验，并在类似VGG的模型，Resnets和WideresNets架构上显示出提高的准确性。