最先进的蒸馏方法主要基于中间层的深层特征，而logit蒸馏的重要性被极大地忽略了。为了提供研究逻辑蒸馏的新观点，我们将经典的KD损失重新分为两个部分，即目标类知识蒸馏（TCKD）和非目标类知识蒸馏（NCKD）。我们凭经验研究并证明了这两个部分的影响：TCKD转移有关训练样本“难度”的知识，而NCKD是Logit蒸馏起作用的重要原因。更重要的是，我们揭示了经典的KD损失是一种耦合的配方，该配方抑制了NCKD的有效性，并且（2）限制了平衡这两个部分的灵活性。为了解决这些问题，我们提出了脱钩的知识蒸馏（DKD），使TCKD和NCKD能够更有效，更灵活地发挥其角色。与基于功能的复杂方法相比，我们的DKD可相当甚至更好的结果，并且在CIFAR-100，ImageNet和MS-Coco数据集上具有更好的培训效率，用于图像分类和对象检测任务。本文证明了Logit蒸馏的巨大潜力，我们希望它对未来的研究有所帮助。该代码可从https://github.com/megvii-research/mdistiller获得。

Most existing distillation methods ignore the flexible role of the temperature in the loss function and fix it as a hyper-parameter that can be decided by an inefficient grid search. In general, the temperature controls the discrepancy between two distributions and can faithfully determine the difficulty level of the distillation task. Keeping a constant temperature, i.e., a fixed level of task difficulty, is usually sub-optimal for a growing student during its progressive learning stages. In this paper, we propose a simple curriculum-based technique, termed Curriculum Temperature for Knowledge Distillation (CTKD), which controls the task difficulty level during the student's learning career through a dynamic and learnable temperature. Specifically, following an easy-to-hard curriculum, we gradually increase the distillation loss w.r.t. the temperature, leading to increased distillation difficulty in an adversarial manner. As an easy-to-use plug-in technique, CTKD can be seamlessly integrated into existing knowledge distillation frameworks and brings general improvements at a negligible additional computation cost. Extensive experiments on CIFAR-100, ImageNet-2012, and MS-COCO demonstrate the effectiveness of our method. Our code is available at https://github.com/zhengli97/CTKD.

知识蒸馏（KD）已广泛发展并增强了各种任务。经典的KD方法将KD损失添加到原始的跨熵（CE）损失中。我们尝试分解KD损失，以探索其与CE损失的关系。令人惊讶的是，我们发现它可以被视为CE损失和额外损失的组合，其形式与CE损失相同。但是，我们注意到额外的损失迫使学生学习教师绝对概率的相对可能性。此外，这两个概率的总和是不同的，因此很难优化。为了解决这个问题，我们修改了配方并提出分布式损失。此外，我们将教师的目标输出作为软目标，提出软损失。结合软损失和分布式损失，我们提出了新的KD损失（NKD）。此外，我们将学生的目标输出稳定，将其视为无需教师的培训的软目标，并提出了无教师的新KD损失（TF-NKD）。我们的方法在CIFAR-100和Imagenet上实现了最先进的性能。例如，以Resnet-34为老师，我们将Imagenet TOP-1的RESNET18的TOP-1精度从69.90％提高到71.96％。在没有教师的培训中，Mobilenet，Resnet-18和Swintransformer-tiny的培训占70.04％，70.76％和81.48％，分别比基线高0.83％，0.86％和0.30％。该代码可在https://github.com/yzd-v/cls_kd上找到。

知识蒸馏（KD）在将学习表征从大型模型（教师）转移到小型模型（学生）方面表现出非常有希望的能力。但是，随着学生和教师之间的容量差距变得更大，现有的KD方法无法获得更好的结果。我们的工作表明，“先验知识”对KD至关重要，尤其是在应用大型老师时。特别是，我们提出了动态的先验知识（DPK），该知识将教师特征的一部分作为特征蒸馏之前的先验知识。这意味着我们的方法还将教师的功能视为“输入”，而不仅仅是``目标''。此外，我们根据特征差距动态调整训练阶段的先验知识比率，从而引导学生在适当的困难中。为了评估所提出的方法，我们对两个图像分类基准（即CIFAR100和Imagenet）和一个对象检测基准（即MS Coco）进行了广泛的实验。结果表明，在不同的设置下，我们方法在性能方面具有优势。更重要的是，我们的DPK使学生模型的表现与教师模型的表现呈正相关，这意味着我们可以通过应用更大的教师进一步提高学生的准确性。我们的代码将公开用于可重复性。

Knowledge distillation (KD) has gained a lot of attention in the field of model compression for edge devices thanks to its effectiveness in compressing large powerful networks into smaller lower-capacity models. Online distillation, in which both the teacher and the student are learning collaboratively, has also gained much interest due to its ability to improve on the performance of the networks involved. The Kullback-Leibler (KL) divergence ensures the proper knowledge transfer between the teacher and student. However, most online KD techniques present some bottlenecks under the network capacity gap. By cooperatively and simultaneously training, the models the KL distance becomes incapable of properly minimizing the teacher's and student's distributions. Alongside accuracy, critical edge device applications are in need of well-calibrated compact networks. Confidence calibration provides a sensible way of getting trustworthy predictions. We propose BD-KD: Balancing of Divergences for online Knowledge Distillation. We show that adaptively balancing between the reverse and forward divergences shifts the focus of the training strategy to the compact student network without limiting the teacher network's learning process. We demonstrate that, by performing this balancing design at the level of the student distillation loss, we improve upon both performance accuracy and calibration of the compact student network. We conducted extensive experiments using a variety of network architectures and show improvements on multiple datasets including CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet. We illustrate the effectiveness of our approach through comprehensive comparisons and ablations with current state-of-the-art online and offline KD techniques.

Often we wish to transfer representational knowledge from one neural network to another. Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator. Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network. We demonstrate that this objective ignores important structural knowledge of the teacher network. This motivates an alternative objective by which we train a student to capture significantly more information in the teacher's representation of the data. We formulate this objective as contrastive learning. Experiments demonstrate that our resulting new objective outperforms knowledge distillation and other cutting-edge distillers on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer. Our method sets a new state-of-the-art in many transfer tasks, and sometimes even outperforms the teacher network when combined with knowledge distillation.

In real teaching scenarios, an excellent teacher always teaches what he (or she) is good at but the student is not. This method gives the student the best assistance in making up for his (or her) weaknesses and becoming a good one overall. Enlightened by this, we introduce the approach to the knowledge distillation framework and propose a data-based distillation method named ``Teaching what you Should Teach (TST)''. To be specific, TST contains a neural network-based data augmentation module with the priori bias, which can assist in finding what the teacher is good at while the student are not by learning magnitudes and probabilities to generate suitable samples. By training the data augmentation module and the generalized distillation paradigm in turn, a student model that has excellent generalization ability can be created. To verify the effectiveness of TST, we conducted extensive comparative experiments on object recognition (CIFAR-100 and ImageNet-1k), detection (MS-COCO), and segmentation (Cityscapes) tasks. As experimentally demonstrated, TST achieves state-of-the-art performance on almost all teacher-student pairs. Furthermore, we conduct intriguing studies of TST, including how to solve the performance degradation caused by the stronger teacher and what magnitudes and probabilities are needed for the distillation framework.

在线知识蒸馏（OKD）通过相互利用教师和学生之间的差异来改善所涉及的模型。它们之间的差距上有几个关键的瓶颈 - 例如，为什么以及何时以及何时损害表现，尤其是对学生的表现？如何量化教师和学生之间的差距？ - 接受了有限的正式研究。在本文中，我们提出了可切换的在线知识蒸馏（Switokd），以回答这些问题。 Switokd的核心思想不是专注于测试阶段的准确性差距，而是通过两种模式之间的切换策略来适应训练阶段的差距，即蒸馏差距 - 专家模式（暂停老师，同时暂停教师保持学生学习）和学习模式（重新启动老师）。为了拥有适当的蒸馏差距，我们进一步设计了一个自适应开关阈值，该阈值提供了有关何时切换到学习模式或专家模式的正式标准，从而改善了学生的表现。同时，老师从我们的自适应切换阈值中受益，并基本上与其他在线艺术保持同步。我们进一步将Switokd扩展到具有两个基础拓扑的多个网络。最后，广泛的实验和分析验证了Switokd在最新面前的分类的优点。我们的代码可在https://github.com/hfutqian/switokd上找到。

无教师的在线知识蒸馏（KD）旨在培训多个学生模型的合奏，并彼此提炼知识。尽管现有的在线KD方法实现了理想的性能，但它们通常专注于阶级概率作为核心知识类型，而忽略了宝贵的特征代表性信息。我们为在线KD提供了一个相互的对比学习（MCL）框架。 MCL的核心思想是以在线方式进行对比分布的相互交互和对比度分布的转移。我们的MCL可以汇总跨网络嵌入信息，并最大化两个网络之间的相互信息的下限。这使每个网络能够从他人那里学习额外的对比知识，从而提供更好的特征表示形式，从而提高视觉识别任务的性能。除最后一层外，我们还将MCL扩展到辅助特征细化模块辅助的几个中间层。这进一步增强了在线KD的表示能力。关于图像分类和转移学习到视觉识别任务的实验表明，MCL可以针对最新的在线KD方法带来一致的性能提高。优势表明，MCL可以指导网络生成更好的特征表示。我们的代码可在https://github.com/winycg/mcl上公开获取。

知识蒸馏是通过知识转移模型压缩的有效稳定的方法。传统知识蒸馏（KD）是将来自大型和训练有素的教师网络的知识转移到小型学生网络，这是一种单向过程。最近，已经提出了深度相互学习（DML）来帮助学生网络协同和同时学习。然而，据我们所知，KD和DML从未在统一的框架中共同探索，以解决知识蒸馏问题。在本文中，我们调查教师模型在KD中支持更值得信赖的监督信号，而学生则在DML中捕获教师的类似行为。基于这些观察，我们首先建议将KD与DML联合在统一的框架中。此外，我们提出了一个半球知识蒸馏（SOKD）方法，有效提高了学生和教师的表现。在这种方法中，我们在DML中介绍了同伴教学培训时尚，以缓解学生的模仿困难，并利用KD训练有素的教师提供的监督信号。此外，我们还显示我们的框架可以轻松扩展到基于功能的蒸馏方法。在CiFAR-100和Imagenet数据集上的广泛实验证明了所提出的方法实现了最先进的性能。

Unlike existing knowledge distillation methods focus on the baseline settings, where the teacher models and training strategies are not that strong and competing as state-of-the-art approaches, this paper presents a method dubbed DIST to distill better from a stronger teacher. We empirically find that the discrepancy of predictions between the student and a stronger teacher may tend to be fairly severer. As a result, the exact match of predictions in KL divergence would disturb the training and make existing methods perform poorly. In this paper, we show that simply preserving the relations between the predictions of teacher and student would suffice, and propose a correlation-based loss to capture the intrinsic inter-class relations from the teacher explicitly. Besides, considering that different instances have different semantic similarities to each class, we also extend this relational match to the intra-class level. Our method is simple yet practical, and extensive experiments demonstrate that it adapts well to various architectures, model sizes and training strategies, and can achieve state-of-the-art performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at: https://github.com/hunto/DIST_KD .

最近对知识蒸馏的研究发现，组合来自多位教师或学生的“黑暗知识”是有助于为培训创造更好的软目标，但以更大的计算和/或参数的成本为本。在这项工作中，我们通过在同一批量中传播和集合其他样本的知识来提供批处理知识合奏（烘焙）以生产用于锚固图像的精细柔软目标。具体地，对于每个感兴趣的样本，根据采样间的亲和力加权知识的传播，其与当前网络一起估计。然后可以集合传播的知识以形成更好的蒸馏靶。通过这种方式，我们的烘焙框架只通过单个网络跨多个样本进行在线知识。与现有知识合并方法相比，它需要最小的计算和内存开销。广泛的实验表明，轻质但有效的烘烤始终如一地提升多个数据集上各种架构的分类性能，例如，在想象网上的显着+ 0.7％的VINE-T的增益，只有+ 1.5％计算开销和零附加参数。烘焙不仅改善了Vanilla基线，还超越了所有基准的单一网络最先进。

最初引入了知识蒸馏，以利用来自单一教师模型的额外监督为学生模型培训。为了提高学生表现，最近的一些变体试图利用多个教师利用不同的知识来源。然而，现有研究主要通过对多种教师预测的平均或将它们与其他无标签策略相结合，将知识集成在多种来源中，可能在可能存在低质量的教师预测存在中误导学生。为了解决这个问题，我们提出了信心感知的多教师知识蒸馏（CA-MKD），该知识蒸馏（CA-MKD）在地面真理标签的帮助下，适用于每个教师预测的样本明智的可靠性，与那些接近单热的教师预测标签分配了大量的重量。此外，CA-MKD包含中间层，以进一步提高学生表现。广泛的实验表明，我们的CA-MKD始终如一地优于各种教师学生架构的所有最先进的方法。

知识蒸馏已成为获得紧凑又有效模型的重要方法。为实现这一目标，培训小型学生模型以利用大型训练有素的教师模型的知识。然而，由于教师和学生之间的能力差距，学生的表现很难达到老师的水平。关于这个问题，现有方法建议通过代理方式减少教师知识的难度。我们认为这些基于代理的方法忽视了教师的知识损失，这可能导致学生遇到容量瓶颈。在本文中，我们从新的角度来缓解能力差距问题，以避免知识损失的目的。我们建议通过对抗性协作学习建立一个更有力的学生，而不是牺牲教师的知识。为此，我们进一步提出了一种逆势协作知识蒸馏（ACKD）方法，有效提高了知识蒸馏的性能。具体来说，我们用多个辅助学习者构建学生模型。同时，我们设计了对抗的对抗性协作模块（ACM），引入注意机制和对抗的学习，以提高学生的能力。四个分类任务的广泛实验显示了拟议的Ackd的优越性。

Figure 1. An illustration of standard knowledge distillation. Despite widespread use, an understanding of when the student can learn from the teacher is missing.

自我介绍在训练过程中利用自身的非均匀软监管，并在没有任何运行时成本的情况下提高性能。但是，在训练过程中的开销经常被忽略，但是在巨型模型的时代，培训期间的时间和记忆开销越来越重要。本文提出了一种名为ZIPF标签平滑（ZIPF的LS）的有效自我验证方法，该方法使用网络的直立预测来生成软监管，该软监管在不使用任何对比样本或辅助参数的情况下符合ZIPF分布。我们的想法来自经验观察，即当对网络进行适当训练时，在按样品的大小和平均分类后，应遵循分布的分布，让人联想到ZIPF的自然语言频率统计信息，这是在按样品中的大小和平均值进行排序之后进行的。 。通过在样本级别和整个培训期内强制执行此属性，我们发现预测准确性可以大大提高。使用INAT21细粒分类数据集上的RESNET50，与香草基线相比，我们的技术获得了 +3.61％的准确性增长，而与先前的标签平滑或自我验证策略相比，增益增加了0.88％。该实现可在https://github.com/megvii-research/zipfls上公开获得。

尽管深层神经网络在各种任务中取得了巨大的成功，但它们不断增加的规模也为部署带来了重要的开销。为了压缩这些模型，提出了知识蒸馏将知识从笨拙（教师）网络转移到轻量级（学生）网络中。但是，老师的指导并不总是改善学生的概括，尤其是当学生和老师之间的差距很大时。以前的作品认为，这是由于老师的高确定性，导致更难适应的标签。为了软化这些标签，我们提出了一种修剪方法，称为预测不确定性扩大（PRUE），以简化教师。具体而言，我们的方法旨在减少教师对数据的确定性，从而为学生产生软预测。我们从经验上研究了提出的方法通过在CIFAR-10/100，Tiny-Imagenet和Imagenet上实验的实验的有效性。结果表明，接受稀疏教师培训的学生网络取得更好的表现。此外，我们的方法允许研究人员从更深的网络中提取知识，以进一步改善学生。我们的代码公开：\ url {https://github.com/wangshaopu/prue}。

知识蒸馏已成功地应用于各种任务。当前的蒸馏算法通常通过模仿教师的产出来改善学生的表现。本文表明，教师还可以通过指导学生的功能恢复来提高学生的代表权。从这个角度来看，我们提出了掩盖的生成蒸馏（MGD），这很简单：我们掩盖了学生功能的随机像素，并强迫它通过简单的块生成教师的完整功能。 MGD是一种真正的基于特征的蒸馏方法，可用于各种任务，包括图像分类，对象检测，语义分割和实例分割。我们在具有广泛数据集的不同模型上进行了实验，结果表明所有学生都取得了出色的改进。值得注意的是，我们将RESNET-18从69.90％提高到71.69％的Imagenet Top-1精度，带有Resnet-50骨架的视网膜从37.4到41.0界盒映射，基于Resnet-50的独奏从33.1到33.1至36.2 Mask Map和DeepLabV3， 18从73.20到76.02 miou。我们的代码可在https://github.com/yzd-v/mgd上找到。

Knowledge Distillation (KD) aims to distill the knowledge of a cumbersome teacher model into a lightweight student model. Its success is generally attributed to the privileged information on similarities among categories provided by the teacher model, and in this sense, only strong teacher models are deployed to teach weaker students in practice. In this work, we challenge this common belief by following experimental observations: 1) beyond the acknowledgment that the teacher can improve the student, the student can also enhance the teacher significantly by reversing the KD procedure; 2) a poorly-trained teacher with much lower accuracy than the student can still improve the latter significantly. To explain these observations, we provide a theoretical analysis of the relationships between KD and label smoothing regularization. We prove that 1) KD is a type of learned label smoothing regularization and 2) label smoothing regularization provides a virtual teacher model for KD. From these results, we argue that the success of KD is not fully due to the similarity information between categories from teachers, but also to the regularization of soft targets, which is equally or even more important.Based on these analyses, we further propose a novel Teacher-free Knowledge Distillation (Tf-KD) framework, where a student model learns from itself or manuallydesigned regularization distribution. The Tf-KD achieves comparable performance with normal KD from a superior teacher, which is well applied when a stronger teacher model is unavailable. Meanwhile, Tf-KD is generic and can be directly deployed for training deep neural networks. Without any extra computation cost, Tf-KD achieves up to 0.65% improvement on ImageNet over well-established baseline models, which is superior to label smoothing regularization.

One of the most efficient methods for model compression is hint distillation, where the student model is injected with information (hints) from several different layers of the teacher model. Although the selection of hint points can drastically alter the compression performance, conventional distillation approaches overlook this fact and use the same hint points as in the early studies. Therefore, we propose a clustering based hint selection methodology, where the layers of teacher model are clustered with respect to several metrics and the cluster centers are used as the hint points. Our method is applicable for any student network, once it is applied on a chosen teacher network. The proposed approach is validated in CIFAR-100 and ImageNet datasets, using various teacher-student pairs and numerous hint distillation methods. Our results show that hint points selected by our algorithm results in superior compression performance compared to state-of-the-art knowledge distillation algorithms on the same student models and datasets.