The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels. Smoothing the labels in this way prevents the network from becoming over-confident and label smoothing has been used in many state-of-the-art models, including image classification, language translation and speech recognition. Despite its widespread use, label smoothing is still poorly understood. Here we show empirically that in addition to improving generalization, label smoothing improves model calibration which can significantly improve beam-search. However, we also observe that if a teacher network is trained with label smoothing, knowledge distillation into a student network is much less effective. To explain these observations, we visualize how label smoothing changes the representations learned by the penultimate layer of the network. We show that label smoothing encourages the representations of training examples from the same class to group in tight clusters. This results in loss of information in the logits about resemblances between instances of different classes, which is necessary for distillation, but does not hurt generalization or calibration of the model's predictions.

这项工作研究了标签平滑（LS）和知识蒸馏（KD）之间的兼容性。解决这一论文陈述的当代发现采取二分法的观点：Muller等。 （2019）和Shen等。 （2021b）。至关重要的是，没有努力理解和解决这些矛盾的发现，留下了原始问题 - 顺利还是不平稳教师网络？ - 未得到答复。我们工作的主要贡献是对系统扩散的发现，分析和验证是缺失的概念，这在理解和解决这些矛盾的发现方面具有重要作用。这种系统的扩散基本上削减了从LS训练的老师蒸馏的好处，从而使KD在升高的温度无效时使KD呈现。我们的发现得到了大规模实验，分析和案例研究的全面支持，包括图像分类，神经机器翻译和紧凑的学生蒸馏任务，这些任务跨越了多个数据集和教师 - 学生架构。根据我们的分析，我们建议从业者使用具有低温转移的LS训练的老师来实现高性能学生。代码和型号可在https://keshik6.github.io/revisiting-ls-kd-compatibility/

以前的工作提出了许多新的损失函数和常规程序，可提高图像分类任务的测试准确性。但是，目前尚不清楚这些损失函数是否了解下游任务的更好表示。本文研究了培训目标的选择如何影响卷积神经网络隐藏表示的可转移性，训练在想象中。我们展示了许多目标在Vanilla Softmax交叉熵上导致想象的精度有统计学意义的改进，但由此产生的固定特征提取器转移到下游任务基本较差，并且当网络完全微调时，损失的选择几乎没有效果新任务。使用居中内核对齐来测量网络隐藏表示之间的相似性，我们发现损失函数之间的差异仅在网络的最后几层中都很明显。我们深入了解倒数第二层的陈述，发现不同的目标和近奇计的组合导致大幅不同的类别分离。具有较高类别分离的表示可以在原始任务上获得更高的准确性，但它们的功能对于下游任务不太有用。我们的结果表明，用于原始任务的学习不变功能与传输任务相关的功能之间存在权衡。

尽管深层神经网络在各种任务中取得了巨大的成功，但它们不断增加的规模也为部署带来了重要的开销。为了压缩这些模型，提出了知识蒸馏将知识从笨拙（教师）网络转移到轻量级（学生）网络中。但是，老师的指导并不总是改善学生的概括，尤其是当学生和老师之间的差距很大时。以前的作品认为，这是由于老师的高确定性，导致更难适应的标签。为了软化这些标签，我们提出了一种修剪方法，称为预测不确定性扩大（PRUE），以简化教师。具体而言，我们的方法旨在减少教师对数据的确定性，从而为学生产生软预测。我们从经验上研究了提出的方法通过在CIFAR-10/100，Tiny-Imagenet和Imagenet上实验的实验的有效性。结果表明，接受稀疏教师培训的学生网络取得更好的表现。此外，我们的方法允许研究人员从更深的网络中提取知识，以进一步改善学生。我们的代码公开：\ url {https://github.com/wangshaopu/prue}。

尽管深神经网络的占优势性能，但最近的作品表明它们校准不佳，导致过度自信的预测。由于培训期间的跨熵最小化，因此可以通过过度化来加剧错误烫伤，因为它促进了预测的Softmax概率来匹配单热标签分配。这产生了正确的类别的Pre-SoftMax激活，该类别明显大于剩余的激活。来自文献的最近证据表明，损失函数嵌入隐含或明确最大化的预测熵会产生最先进的校准性能。我们提供了当前最先进的校准损耗的统一约束优化视角。具体地，这些损失可以被视为在Logit距离上施加平等约束的线性惩罚（或拉格朗日）的近似值。这指出了这种潜在的平等约束的一个重要限制，其随后的梯度不断推动非信息解决方案，这可能会阻止在基于梯度的优化期间模型的辨别性能和校准之间的最佳妥协。在我们的观察之后，我们提出了一种基于不平等约束的简单灵活的泛化，这在Logit距离上强加了可控裕度。关于各种图像分类，语义分割和NLP基准的综合实验表明，我们的方法在网络校准方面对这些任务设置了新的最先进的结果，而不会影响辨别性能。代码可在https://github.com/by-liu/mbls上获得。

Knowledge distillation (KD) has gained a lot of attention in the field of model compression for edge devices thanks to its effectiveness in compressing large powerful networks into smaller lower-capacity models. Online distillation, in which both the teacher and the student are learning collaboratively, has also gained much interest due to its ability to improve on the performance of the networks involved. The Kullback-Leibler (KL) divergence ensures the proper knowledge transfer between the teacher and student. However, most online KD techniques present some bottlenecks under the network capacity gap. By cooperatively and simultaneously training, the models the KL distance becomes incapable of properly minimizing the teacher's and student's distributions. Alongside accuracy, critical edge device applications are in need of well-calibrated compact networks. Confidence calibration provides a sensible way of getting trustworthy predictions. We propose BD-KD: Balancing of Divergences for online Knowledge Distillation. We show that adaptively balancing between the reverse and forward divergences shifts the focus of the training strategy to the compact student network without limiting the teacher network's learning process. We demonstrate that, by performing this balancing design at the level of the student distillation loss, we improve upon both performance accuracy and calibration of the compact student network. We conducted extensive experiments using a variety of network architectures and show improvements on multiple datasets including CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet. We illustrate the effectiveness of our approach through comprehensive comparisons and ablations with current state-of-the-art online and offline KD techniques.

知识蒸馏是一种培训小型学生网络的流行技术，以模仿更大的教师模型，例如网络的集合。我们表明，虽然知识蒸馏可以改善学生泛化，但它通常不得如此普遍地工作：虽然在教师和学生的预测分布之间，甚至在学生容量的情况下，通常仍然存在令人惊讶的差异完美地匹配老师。我们认为优化的困难是为什么学生无法与老师匹配的关键原因。我们还展示了用于蒸馏的数据集的细节如何在学生与老师匹配的紧密关系中发挥作用 - 以及教师矛盾的教师并不总是导致更好的学生泛化。

Mixup is a popular data augmentation technique based on creating new samples by linear interpolation between two given data samples, to improve both the generalization and robustness of the trained model. Knowledge distillation (KD), on the other hand, is widely used for model compression and transfer learning, which involves using a larger network's implicit knowledge to guide the learning of a smaller network. At first glance, these two techniques seem very different, however, we found that ``smoothness" is the connecting link between the two and is also a crucial attribute in understanding KD's interplay with mixup. Although many mixup variants and distillation methods have been proposed, much remains to be understood regarding the role of a mixup in knowledge distillation. In this paper, we present a detailed empirical study on various important dimensions of compatibility between mixup and knowledge distillation. We also scrutinize the behavior of the networks trained with a mixup in the light of knowledge distillation through extensive analysis, visualizations, and comprehensive experiments on image classification. Finally, based on our findings, we suggest improved strategies to guide the student network to enhance its effectiveness. Additionally, the findings of this study provide insightful suggestions to researchers and practitioners that commonly use techniques from KD. Our code is available at https://github.com/hchoi71/MIX-KD.

基于蒸馏的压缩网络的性能受蒸馏质量的管辖。大型网络（教师）到较小网络（学生）的次优蒸馏的原因主要归因于给定教师与学生对的学习能力中的差距。虽然很难蒸馏所有教师的知识，但可以在很大程度上控制蒸馏质量以实现更好的性能。我们的实验表明，蒸馏品质主要受教师响应的质量来限制，这反过来又受到其反应中存在相似信息的影响。训练有素的大容量老师在学习细粒度辨别性质的过程中丢失了类别之间的相似性信息。没有相似性信息导致蒸馏过程从一个例子 - 许多阶级学习减少到一个示例 - 一类学习，从而限制了教师的不同知识的流程。由于隐式假设只能蒸馏出灌输所知，而不是仅关注知识蒸馏过程，我们仔细审查了知识序列过程。我们认为，对于给定的教师 - 学生对，通过在训练老师的同时找到批量大小和时代数量之间的甜蜜点，可以提高蒸馏品。我们讨论了找到这种甜蜜点以便更好地蒸馏的步骤。我们还提出了蒸馏假设，以区分知识蒸馏和正则化效果之间的蒸馏过程的行为。我们在三个不同的数据集中进行我们的所有实验。

Despite the fact that deep neural networks are powerful models and achieve appealing results on many tasks, they are too large to be deployed on edge devices like smartphones or embedded sensor nodes. There have been efforts to compress these networks, and a popular method is knowledge distillation, where a large (teacher) pre-trained network is used to train a smaller (student) network. However, in this paper, we show that the student network performance degrades when the gap between student and teacher is large. Given a fixed student network, one cannot employ an arbitrarily large teacher, or in other words, a teacher can effectively transfer its knowledge to students up to a certain size, not smaller. To alleviate this shortcoming, we introduce multi-step knowledge distillation, which employs an intermediate-sized network (teacher assistant) to bridge the gap between the student and the teacher. Moreover, we study the effect of teacher assistant size and extend the framework to multi-step distillation. Theoretical analysis and extensive experiments on CIFAR-10,100 and ImageNet datasets and on CNN and ResNet architectures substantiate the effectiveness of our proposed approach.

Figure 1. An illustration of standard knowledge distillation. Despite widespread use, an understanding of when the student can learn from the teacher is missing.

Often we wish to transfer representational knowledge from one neural network to another. Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator. Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network. We demonstrate that this objective ignores important structural knowledge of the teacher network. This motivates an alternative objective by which we train a student to capture significantly more information in the teacher's representation of the data. We formulate this objective as contrastive learning. Experiments demonstrate that our resulting new objective outperforms knowledge distillation and other cutting-edge distillers on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer. Our method sets a new state-of-the-art in many transfer tasks, and sometimes even outperforms the teacher network when combined with knowledge distillation.

Confidence calibration -the problem of predicting probability estimates representative of the true correctness likelihood -is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-ofthe-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -a singleparameter variant of Platt Scaling -is surprisingly effective at calibrating predictions.

近年来，知识蒸馏有显着改善，可以为更好的效率产生紧凑的学生模型，同时保留教师模型的模型效果。以前的研究发现：由于能力不匹配，更准确的教师对更好的教师无需。在本文中，我们旨在通过模型校准的角度分析现象。我们发现较大的教师模型可能过于过度自信，因此学生模型无法有效地模仿。虽然，在教师模型的简单模型校准之后，教师模型的大小与学生模型的性能具有正相关。

知识蒸馏是将“知识”从大型模型（教师）转移到更紧凑的（学生）的过程，通常在模型压缩的背景下使用。当两个模型都具有相同的体系结构时，此过程称为自distillation。几项轶事表明，一个自灭的学生可以在持有的数据上胜过老师的表现。在这项工作中，我们系统地研究了许多设置。我们首先表明，即使有一个高度准确的老师，自我介绍也使学生在所有情况下都可以超越老师。其次，我们重新审视了（自我）蒸馏的现有理论解释，并确定矛盾的例子，揭示了这些解释的可能缺点。最后，我们通过损失景观几何形状的镜头为自我鉴定的动态提供了另一种解释。我们进行了广泛的实验，以表明自我验证会导致最小化的最小值，从而导致更好的概括。

自我介绍在训练过程中利用自身的非均匀软监管，并在没有任何运行时成本的情况下提高性能。但是，在训练过程中的开销经常被忽略，但是在巨型模型的时代，培训期间的时间和记忆开销越来越重要。本文提出了一种名为ZIPF标签平滑（ZIPF的LS）的有效自我验证方法，该方法使用网络的直立预测来生成软监管，该软监管在不使用任何对比样本或辅助参数的情况下符合ZIPF分布。我们的想法来自经验观察，即当对网络进行适当训练时，在按样品的大小和平均分类后，应遵循分布的分布，让人联想到ZIPF的自然语言频率统计信息，这是在按样品中的大小和平均值进行排序之后进行的。 。通过在样本级别和整个培训期内强制执行此属性，我们发现预测准确性可以大大提高。使用INAT21细粒分类数据集上的RESNET50，与香草基线相比，我们的技术获得了 +3.61％的准确性增长，而与先前的标签平滑或自我验证策略相比，增益增加了0.88％。该实现可在https://github.com/megvii-research/zipfls上公开获得。

机器学习中的知识蒸馏是将知识从名为教师的大型模型转移到一个名为“学生”的较小模型的过程。知识蒸馏是将大型网络（教师）压缩到较小网络（学生）的技术之一，该网络可以部署在手机等小型设备中。当教师和学生之间的网络规模差距增加时，学生网络的表现就会下降。为了解决这个问题，在教师模型和名为助教模型的学生模型之间采用了中间模型，这反过来弥补了教师与学生之间的差距。在这项研究中，我们已经表明，使用多个助教模型，可以进一步改进学生模型（较小的模型）。我们使用加权集合学习将这些多个助教模型组合在一起，我们使用了差异评估优化算法来生成权重值。

本文描述了机器学习中“一般周期性训练”的原则，在该原理中，培训以“易于训练”开始和结束，而“硬训练”发生在中间时期。我们提出了几种训练神经网络的表现，包括算法示例（通过超参数和损失功能），基于数据的示例和基于模型的示例。具体而言，我们介绍了几种新技术：周期性重量衰减，周期性批量尺寸，周期性局灶性损失，周期性软度温度，周期性数据增强，周期性梯度剪辑和周期性的半监督学习。此外，我们证明了周期性的重量衰减，周期性软度温度和周期性梯度剪辑（作为该原理的三个示例）对训练有素的模型的测试准确性性能有益。此外，我们从一般周期性培训的角度讨论了基于模型的示例（例如预处理和知识蒸馏），并建议对典型培训方法进行一些更改。总而言之，本文定义了一般的周期性培训概念，并讨论了该概念可以应用于训练神经网络的几种特定方式。本着可重复性的精神，我们的实验中使用的代码可在\ url {https://github.com/lnsmith54/cfl}上获得。

Recent studies have revealed that, beyond conventional accuracy, calibration should also be considered for training modern deep neural networks. To address miscalibration during learning, some methods have explored different penalty functions as part of the learning objective, alongside a standard classification loss, with a hyper-parameter controlling the relative contribution of each term. Nevertheless, these methods share two major drawbacks: 1) the scalar balancing weight is the same for all classes, hindering the ability to address different intrinsic difficulties or imbalance among classes; and 2) the balancing weight is usually fixed without an adaptive strategy, which may prevent from reaching the best compromise between accuracy and calibration, and requires hyper-parameter search for each application. We propose Class Adaptive Label Smoothing (CALS) for calibrating deep networks, which allows to learn class-wise multipliers during training, yielding a powerful alternative to common label smoothing penalties. Our method builds on a general Augmented Lagrangian approach, a well-established technique in constrained optimization, but we introduce several modifications to tailor it for large-scale, class-adaptive training. Comprehensive evaluation and multiple comparisons on a variety of benchmarks, including standard and long-tailed image classification, semantic segmentation, and text classification, demonstrate the superiority of the proposed method. The code is available at https://github.com/by-liu/CALS.

The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change only once per epoch, Temporal Ensembling becomes unwieldy when learning large datasets. To overcome this problem, we propose Mean Teacher, a method that averages model weights instead of label predictions. As an additional benefit, Mean Teacher improves test accuracy and enables training with fewer labels than Temporal Ensembling. Without changing the network architecture, Mean Teacher achieves an error rate of 4.35% on SVHN with 250 labels, outperforming Temporal Ensembling trained with 1000 labels. We also show that a good network architecture is crucial to performance. Combining Mean Teacher and Residual Networks, we improve the state of the art on CIFAR-10 with 4000 labels from 10.55% to 6.28%, and on ImageNet 2012 with 10% of the labels from 35.24% to 9.11%.