Despite significant accuracy improvement in convolutional neural networks (CNN) based object detectors, they often require prohibitive runtimes to process an image for real-time applications. State-of-the-art models often use very deep networks with a large number of floating point operations. Efforts such as model compression learn compact models with fewer number of parameters, but with much reduced accuracy. In this work, we propose a new framework to learn compact and fast object detection networks with improved accuracy using knowledge distillation [20] and hint learning [34]. Although knowledge distillation has demonstrated excellent improvements for simpler classification setups, the complexity of detection poses new challenges in the form of regression, region proposals and less voluminous labels. We address this through several innovations such as a weighted cross-entropy loss to address class imbalance, a teacher bounded loss to handle the regression component and adaptation layers to better learn from intermediate teacher distributions. We conduct comprehensive empirical evaluation with different distillation configurations over multiple datasets including PASCAL, KITTI, ILSVRC and MS-COCO. Our results show consistent improvement in accuracy-speed trade-offs for modern multi-class detection models.

In recent years, Siamese network based trackers have significantly advanced the state-of-the-art in real-time tracking. Despite their success, Siamese trackers tend to suffer from high memory costs, which restrict their applicability to mobile devices with tight memory budgets. To address this issue, we propose a distilled Siamese tracking framework to learn small, fast and accurate trackers (students), which capture critical knowledge from large Siamese trackers (teachers) by a teacher-students knowledge distillation model. This model is intuitively inspired by the one teacher vs. multiple students learning method typically employed in schools. In particular, our model contains a single teacher-student distillation module and a student-student knowledge sharing mechanism. The former is designed using a tracking-specific distillation strategy to transfer knowledge from a teacher to students. The latter is utilized for mutual learning between students to enable in-depth knowledge understanding. Extensive empirical evaluations on several popular Siamese trackers demonstrate the generality and effectiveness of our framework. Moreover, the results on five tracking benchmarks show that the proposed distilled trackers achieve compression rates of up to 18$\times$ and frame-rates of $265$ FPS, while obtaining comparable tracking accuracy compared to base models.

在真实世界的环境中，可以通过对象检测器连续遇到来自新类的对象实例。当现有的对象探测器应用于这种情况时，它们在旧课程上的性能显着恶化。据报道，一些努力解决了这个限制，所有这些限制适用于知识蒸馏的变体，以避免灾难性的遗忘。我们注意到虽然蒸馏有助于保留以前的学习，但它阻碍了对新任务的快速适应性，这是增量学习的关键要求。在这种追求中，我们提出了一种学习方法，可以学习重塑模型梯度，使得跨增量任务的信息是最佳的共享。这可通过META学习梯度预处理来确保无缝信息传输，可最大限度地减少遗忘并最大化知识传输。与现有的元学习方法相比，我们的方法是任务不可知，允许将新类的增量添加到对象检测的高容量模型中。我们在Pascal-VOC和MS Coco Datasets上定义的各种增量学习设置中评估了我们的方法，我们的方法对最先进的方法进行了好评。

具有更多参数数量的深卷积神经网络在自然图像上的对象检测任务中提高了精度，其中感兴趣的对象用水平边界框注释。从鸟类视角捕获的航空图像上，这些对模型架构和更深卷积层的改进也可以提高定向对象检测任务的性能。但是，很难直接在设备上使用有限的计算资源应用那些最先进的对象探测器，这需要通过模型压缩来实现轻量级模型。为了解决此问题，我们提出了一种模型压缩方法，用于通过知识蒸馏（即KD-RNET）在空中图像上旋转对象检测。凭借具有大量参数的训练有素的以教师为导向的对象探测器，获得的对象类别和位置信息都通过协作培训策略转移到KD-RNET的紧凑型学生网络中。传输类别信息是通过对预测概率分布的知识蒸馏来实现的，并且在处理位置信息传输中的位移时采用了软回归损失。大规模空中对象检测数据集（DOTA）的实验结果表明，提出的KD-RNET模型可以通过减少参数数量来提高均值平均精度（MAP），同时kd-rnet促进性能增强性能在提供高质量检测的情况下，与地面截然注释的重叠更高。

主流对象检测器通常由两个子任务组成，包括由两个并行头部实现的分类和回归任务。这种经典的设计范式不可避免地会导致分类得分和本地化质量（IOU）之间的空间分布不一致。因此，本文从知识蒸馏的角度来减轻这种错位。首先，我们观察到，与轻量级学生相比，庞大的老师获得的和谐预测比例更高。基于这个有趣的观察，设计了一种新颖的和谐评分（HS），以估计分类和回归质量的一致性。 HS对两个子任务之间的关系进行建模，并被视为先验知识，以促进学生的和谐预测。其次，这种空间未对准将在提炼特征时会导致选择性区域的选择。为了减轻这个问题，通过灵活平衡分类和回归任务的贡献，提出了一种新颖的任务功能蒸馏（TFD）。最终，HD和TFD构成了所提出的方法，称为任务均衡蒸馏（TBD）。广泛的实验证明了该方法的巨大潜力和概括。具体而言，当配备TBD时，带有Resnet-50的视网膜在可可基准下获得41.0地图，表现优于最近的FGD和FRS。

几次射击对象检测的大多数现有方法都遵循微调范式，该范式可能假设可以通过众多样本的基本类别学习并将其隐式转移到具有限量样本的新颖类中，从而将类别的概括性知识隐含地转移到有限的类别中。舞台培训策略。但是，这不一定是正确的，因为对象检测器几乎无法在没有明确的建模的情况下自动区分类别不合时宜的知识和特定于类的知识。在这项工作中，我们建议在基础和新颖类之间学习三种类型的类不足的共同点：与识别相关的语义共同点，与定位相关的语义共同点和分布共同点。我们基于内存库设计了一个统一的蒸馏框架，该框架能够共同有效地进行所有三种类型的共同点。广泛的实验表明，我们的方法可以很容易地集成到大多数现有的基于微调的方法中，并始终如一地通过大幅度提高性能。

知识蒸馏已成功应用于图像分类。然而，物体检测更复杂，大多数知识蒸馏方法都失败了。在本文中，我们指出，在物体检测中，教师和学生的特征在不同的区域变化，特别是在前景和背景中。如果我们同样蒸馏它们，则特征图之间的不均匀差异会对蒸馏产生负面影响。因此，我们提出了焦点和全球蒸馏（FGD）。焦蒸馏分离前景和背景，强迫学生专注于教师的临界像素和渠道。全球蒸馏重建了不同像素之间的关系，并将其从教师转移给学生，弥补了局灶性蒸馏中缺失的全球信息。由于我们的方法仅需要计算特征图上的损失，因此FGD可以应用于各种探测器。我们在不同骨干网上进行各种探测器，结果表明，学生探测器实现了优异的地图改进。例如，基于Reset-50基于RecinAnet，更快的RCNN，Reppoints和Mask RCNN，Coco2017上达到40.7％，42.0％，42.0％和42.1％地图，3.3,3.6,3.4和2.9高于基线，分别。我们的代码可在https://github.com/yzd-v/fgd获得。

Due to object detection's close relationship with video analysis and image understanding, it has attracted much research attention in recent years. Traditional object detection methods are built on handcrafted features and shallow trainable architectures. Their performance easily stagnates by constructing complex ensembles which combine multiple low-level image features with high-level context from object detectors and scene classifiers. With the rapid development in deep learning, more powerful tools, which are able to learn semantic, high-level, deeper features, are introduced to address the problems existing in traditional architectures. These models behave differently in network architecture, training strategy and optimization function, etc. In this paper, we provide a review on deep learning based object detection frameworks. Our review begins with a brief introduction on the history of deep learning and its representative tool, namely Convolutional Neural Network (CNN). Then we focus on typical generic object detection architectures along with some modifications and useful tricks to improve detection performance further. As distinct specific detection tasks exhibit different characteristics, we also briefly survey several specific tasks, including salient object detection, face detection and pedestrian detection. Experimental analyses are also provided to compare various methods and draw some meaningful conclusions. Finally, several promising directions and tasks are provided to serve as guidelines for future work in both object detection and relevant neural network based learning systems.

机器学习中的知识蒸馏是将知识从名为教师的大型模型转移到一个名为“学生”的较小模型的过程。知识蒸馏是将大型网络（教师）压缩到较小网络（学生）的技术之一，该网络可以部署在手机等小型设备中。当教师和学生之间的网络规模差距增加时，学生网络的表现就会下降。为了解决这个问题，在教师模型和名为助教模型的学生模型之间采用了中间模型，这反过来弥补了教师与学生之间的差距。在这项研究中，我们已经表明，使用多个助教模型，可以进一步改进学生模型（较小的模型）。我们使用加权集合学习将这些多个助教模型组合在一起，我们使用了差异评估优化算法来生成权重值。

Domain adaptive object detection (DAOD) aims to alleviate transfer performance degradation caused by the cross-domain discrepancy. However, most existing DAOD methods are dominated by computationally intensive two-stage detectors, which are not the first choice for industrial applications. In this paper, we propose a novel semi-supervised domain adaptive YOLO (SSDA-YOLO) based method to improve cross-domain detection performance by integrating the compact one-stage detector YOLOv5 with domain adaptation. Specifically, we adapt the knowledge distillation framework with the Mean Teacher model to assist the student model in obtaining instance-level features of the unlabeled target domain. We also utilize the scene style transfer to cross-generate pseudo images in different domains for remedying image-level differences. In addition, an intuitive consistency loss is proposed to further align cross-domain predictions. We evaluate our proposed SSDA-YOLO on public benchmarks including PascalVOC, Clipart1k, Cityscapes, and Foggy Cityscapes. Moreover, to verify its generalization, we conduct experiments on yawning detection datasets collected from various classrooms. The results show considerable improvements of our method in these DAOD tasks. Our code is available on \url{https://github.com/hnuzhy/SSDA-YOLO}.

近年来，大规模的深层模型取得了巨大的成功，但巨大的计算复杂性和大规模的存储要求使其在资源限制设备中部署它们是一个巨大的挑战。作为模型压缩和加速度方法，知识蒸馏通过从教师探测器转移黑暗知识有效提高了小型模型的性能。然而，大多数基于蒸馏的检测方法主要模仿近边界盒附近的特征，这遭受了两个限制。首先，它们忽略边界盒外面的有益特征。其次，这些方法模仿一些特征，这些特征被教师探测器被错误地被视为背景。为了解决上述问题，我们提出了一种新颖的特征性 - 丰富的评分（FRS）方法，可以选择改善蒸馏过程中的广义可检测性的重要特征。所提出的方法有效地检索边界盒外面的重要特征，并消除边界盒内的有害特征。广泛的实验表明，我们的方法在基于锚和无锚探测器上实现了出色的性能。例如，具有Reset-50的RetinAnet在Coco2017数据集上达到39.7％，甚至超过基于Reset-101的教师检测器38.9％甚至超过0.8％。

知识蒸馏在分类中取得了巨大的成功，但是，仍然有挑战性。在用于检测的典型图像中，来自不同位置的表示可能对检测目标具有不同的贡献，使蒸馏难以平衡。在本文中，我们提出了一种有条件的蒸馏框架来蒸馏出所需的知识，即关于每个例子的分类和本地化有益的知识。该框架引入了一种可学习的条件解码模块，其将每个目标实例检索为查询的信息。具体而言，我们将条件信息编码为查询并使用教师的表示作为键。查询和键之间的注意用于测量不同特征的贡献，由本地化识别敏感辅助任务指导。广泛的实验表明了我们的方法的功效：我们在各种环境下观察到令人印象深刻的改进。值得注意的是，在1倍计划下，我们将通过37.4至40.7地图（+3.3）与Reset-50骨架的Restinetet提升。代码已在https://github.com/megvii-research/icd上发布。

One of the most efficient methods for model compression is hint distillation, where the student model is injected with information (hints) from several different layers of the teacher model. Although the selection of hint points can drastically alter the compression performance, conventional distillation approaches overlook this fact and use the same hint points as in the early studies. Therefore, we propose a clustering based hint selection methodology, where the layers of teacher model are clustered with respect to several metrics and the cluster centers are used as the hint points. Our method is applicable for any student network, once it is applied on a chosen teacher network. The proposed approach is validated in CIFAR-100 and ImageNet datasets, using various teacher-student pairs and numerous hint distillation methods. Our results show that hint points selected by our algorithm results in superior compression performance compared to state-of-the-art knowledge distillation algorithms on the same student models and datasets.

Previous knowledge distillation (KD) methods for object detection mostly focus on feature imitation instead of mimicking the prediction logits due to its inefficiency in distilling the localization information. In this paper, we investigate whether logit mimicking always lags behind feature imitation. Towards this goal, we first present a novel localization distillation (LD) method which can efficiently transfer the localization knowledge from the teacher to the student. Second, we introduce the concept of valuable localization region that can aid to selectively distill the classification and localization knowledge for a certain region. Combining these two new components, for the first time, we show that logit mimicking can outperform feature imitation and the absence of localization distillation is a critical reason for why logit mimicking underperforms for years. The thorough studies exhibit the great potential of logit mimicking that can significantly alleviate the localization ambiguity, learn robust feature representation, and ease the training difficulty in the early stage. We also provide the theoretical connection between the proposed LD and the classification KD, that they share the equivalent optimization effect. Our distillation scheme is simple as well as effective and can be easily applied to both dense horizontal object detectors and rotated object detectors. Extensive experiments on the MS COCO, PASCAL VOC, and DOTA benchmarks demonstrate that our method can achieve considerable AP improvement without any sacrifice on the inference speed. Our source code and pretrained models are publicly available at https://github.com/HikariTJU/LD.

While depth tends to improve network performances, it also makes gradient-based training more difficult since deeper networks tend to be more non-linear. The recently proposed knowledge distillation approach is aimed at obtaining small and fast-to-execute models, and it has shown that a student network could imitate the soft output of a larger teacher network or ensemble of networks. In this paper, we extend this idea to allow the training of a student that is deeper and thinner than the teacher, using not only the outputs but also the intermediate representations learned by the teacher as hints to improve the training process and final performance of the student. Because the student intermediate hidden layer will generally be smaller than the teacher's intermediate hidden layer, additional parameters are introduced to map the student hidden layer to the prediction of the teacher hidden layer. This allows one to train deeper students that can generalize better or run faster, a trade-off that is controlled by the chosen student capacity. For example, on CIFAR-10, a deep student network with almost 10.4 times less parameters outperforms a larger, state-of-the-art teacher network.

深度学习的巨大成功主要是由于大规模的网络架构和高质量的培训数据。但是，在具有有限的内存和成像能力的便携式设备上部署最近的深层模型仍然挑战。一些现有的作品通过知识蒸馏进行了压缩模型。不幸的是，这些方法不能处理具有缩小图像质量的图像，例如低分辨率（LR）图像。为此，我们采取了开创性的努力，从高分辨率（HR）图像到达将处理LR图像的紧凑型网络模型中学习的繁重网络模型中蒸馏有用的知识，从而推动了新颖的像素蒸馏的当前知识蒸馏技术。为实现这一目标，我们提出了一名教师助理 - 学生（TAS）框架，将知识蒸馏分解为模型压缩阶段和高分辨率表示转移阶段。通过装备新颖的特点超分辨率（FSR）模块，我们的方法可以学习轻量级网络模型，可以实现与重型教师模型相似的准确性，但参数更少，推理速度和较低分辨率的输入。在三个广泛使用的基准，\即，幼崽200-2011，Pascal VOC 2007和ImageNetsub上的综合实验证明了我们方法的有效性。

Scaling object taxonomies is one of the important steps toward a robust real-world deployment of recognition systems. We have faced remarkable progress in images since the introduction of the LVIS benchmark. To continue this success in videos, a new video benchmark, TAO, was recently presented. Given the recent encouraging results from both detection and tracking communities, we are interested in marrying those two advances and building a strong large vocabulary video tracker. However, supervisions in LVIS and TAO are inherently sparse or even missing, posing two new challenges for training the large vocabulary trackers. First, no tracking supervisions are in LVIS, which leads to inconsistent learning of detection (with LVIS and TAO) and tracking (only with TAO). Second, the detection supervisions in TAO are partial, which results in catastrophic forgetting of absent LVIS categories during video fine-tuning. To resolve these challenges, we present a simple but effective learning framework that takes full advantage of all available training data to learn detection and tracking while not losing any LVIS categories to recognize. With this new learning scheme, we show that consistent improvements of various large vocabulary trackers are capable, setting strong baseline results on the challenging TAO benchmarks.

知识蒸馏（KD）是一种广泛使用的技术，将繁琐的教师模型继承到紧凑的学生模型，从而实现模型压缩和加速度。与图像分类相比，对象检测是一个更复杂的任务，设计特定的KD方法用于对象检测是非微小的。在这项工作中，我们精心研究教师和学生检测模型之间的行为差​​异，并获得了两个有趣的观察：首先，教师和学生对其检测到的候选盒子相得益彰，这导致了它们的精确差异。其次，教师和学生之间的特征响应差异和预测差异之间存在相当大的差距，表明同样模仿老师的所有特征映射是提高学生准确性的次优选。基于这两个观察，我们提出了用于分别蒸馏单级探测器的测量模拟（RM）和预测引导的特征模仿（PFI）。 RM从教师那里夺取候选人盒的等级作为一种新的知识形式，蒸馏，这始终如一地优于传统的软标签蒸馏。 PFI试图将特征差异与预测差异相关，使特征模仿直接有助于提高学生的准确性。在MS Coco和Pascal VOC基准测试中，广泛的实验在不同骨干的各种探测器上进行，以验证我们方法的有效性。具体而言，具有Reset50的RetinAnet在MS Coco中实现了40.4％的图，比其基线高3.5％，并且还优于先前的KD方法。

深度学习方法需要大量的注释数据以优化参数。例如，附加具有准确边界框注释的数据集对于现代对象检测任务至关重要。但是，具有这样的像素准确性的标签是费力且耗时的，并且精心制作的标记程序对于降低人造噪声是必不可少的，涉及注释审查和接受测试。在本文中，我们关注嘈杂的位置注释对对象检测方法的性能的影响，并旨在减少噪声的不利影响。首先，当将噪声引入边界框注释中时，一阶段和两阶段检测器都会在实验上观察到明显的性能降解。例如，我们的合成噪声导致可可测试分裂的FCO探测器的性能从38.9％的AP降低到33.6％的AP，对于更快的R-CNN而言，COCO检测器的性能从38.9％的AP下降到37.8％的AP和33.7％的AP。其次，提出了一种基于贝叶斯过滤器进行预测合奏的自我纠正技术，以更好地利用教师学习范式后的嘈杂位置注释。合成和现实世界情景的实验始终证明了我们方法的有效性，例如，我们的方法将FCOS检测器的降解性能从33.6％的AP提高到可可的35.6％AP。

胃肠道（GI）癌症的患病率每年令人震惊，导致死亡率大幅上升。内窥镜检测提供了至关重要的诊断支持，但是，上胃肠道中的细微病变很难检测到，并引起大量的错过检测。在这项工作中，我们利用深度学习来开发一个框架，以改善难以检测病变的本地化并最大程度地减少遗漏的检测率。我们提出了一个端到端的学生教师学习设置，其中使用较大数据集的一个班级训练有素的教师模型的班级概率用于惩罚多级学生网络。我们的模型在两种内窥镜疾病检测（EDD2020）挑战和Kvasir-SEG数据集上，在平均平均精度（MAP）方面达到了更高的性能。此外，我们表明，使用这样的学习范式，我们的模型可以推广到看不见的测试集，从而为临床上关键的肿瘤和息肉类别提供更高的APS