We present a strong object detector with encoder-decoder pretraining and finetuning. Our method, called Group DETR v2, is built upon a vision transformer encoder ViT-Huge~\cite{dosovitskiy2020image}, a DETR variant DINO~\cite{zhang2022dino}, and an efficient DETR training method Group DETR~\cite{chen2022group}. The training process consists of self-supervised pretraining and finetuning a ViT-Huge encoder on ImageNet-1K, pretraining the detector on Object365, and finally finetuning it on COCO. Group DETR v2 achieves $\textbf{64.5}$ mAP on COCO test-dev, and establishes a new SoTA on the COCO leaderboard https://paperswithcode.com/sota/object-detection-on-coco

我们将Dino（\ textbf {d} etr与\ textbf {i} mpred de \ textbf {n} oising hand \ textbf {o} r boxes），一种最先进的端到端对象检测器。 ％ 在本文中。 Dino通过使用一种对比度方法来降级训练，一种用于锚定初始化的混合查询选择方法以及对盒子预测的两次方案，通过使用对比的方式来改善性能和效率的模型。 Dino在$ 12 $时代获得$ 49.4 $ ap，$ 12.3 $ ap in Coco $ 24 $时期，带有Resnet-50骨干和多尺度功能，可显着改善$ \ textbf {+6.0} $ \ textbf {ap}和ap {ap}和ap}和$ \ textbf {+2.7} $ \ textbf {ap}与以前的最佳detr样模型相比，分别是dn-detr。 Dino在模型大小和数据大小方面都很好地缩放。没有铃铛和哨子，在对objects365数据集进行了swinl骨架的预训练后，Dino在两个Coco \ texttt {val2017}（$ \ textbf {63.2} $ \ textbf {ap ap}）和\ testtt { -dev}（\ textbf {$ \ textbf {63.3} $ ap}）。与排行榜上的其他模型相比，Dino大大降低了其模型大小和预训练数据大小，同时实现了更好的结果。我们的代码将在\ url {https://github.com/ideacvr/dino}提供。

已经提出了各种模型来执行对象检测。但是，大多数人都需要许多手工设计的组件，例如锚和非最大抑制（NMS），以表现出良好的性能。为了减轻这些问题，建议了基于变压器的DETR及其变体可变形DETR。这些解决了为对象检测模型设计头部时的许多复杂问题。但是，当将基于变压器的模型视为其他模型的对象检测中的最新方法时，仍然存在对性能的疑问，这取决于锚定和NMS，揭示了更好的结果。此外，目前尚不清楚是否可以仅与注意模块结合使用端到端管道，因为Detr适应的变压器方法使用卷积神经网络（CNN）作为骨干身体。在这项研究中，我们建议将几个注意力模块与我们的新任务特异性分裂变压器（TSST）相结合是一种有力的方法，可以在没有传统手工设计的组件的情况下生成可可结果上最先进的性能。通过将通用注意模块分为两个分开的目标注意模块，该方法允许设计简单的对象检测模型。对可可基准的广泛实验证明了我们方法的有效性。代码可在https://github.com/navervision/tsst上获得

检测变压器（DETR）依赖于一对一的标签分配，即仅分配一个地面真相（GT）对象作为一个阳性对象查询，用于端到端对象检测，并且缺乏利用多个积极查询的能力。我们提出了一种新颖的DETR训练方法，称为{\ em grout detr}，以支持多个积极查询。具体来说，我们将阳性分解为多个独立组，并在每个组中只保留一个阳性对象。我们在培训期间进行了简单的修改：（i）采用$ k $ of Absock Queries； （ii）对具有相同参数的每组对象查询进行解码器自我注意； （iii）为每个组执行一对一的标签分配，从而为每个GT对象提供$ K $阳性对象查询。在推论中，我们只使用一组对象查询，对架构和过程没有任何修改。我们验证了提出的方法对DITR变体的有效性，包括条件DITR，DAB-DER，DN-DEN和DINO。

一对一的匹配是DETR建立其端到端功能的关键设计，因此对象检测不需要手工制作的NMS（非最大抑制）方法来删除重复检测。这种端到端的签名对于DETR的多功能性很重要，并且已将其推广到广泛的视觉问题，包括实例/语义分割，人体姿势估计以及基于点云/多视图的检测，但是，我们注意到，由于分配为正样本的查询太少，因此一对一的匹配显着降低了阳性样品的训练效率。本文提出了一种基于混合匹配方案的简单而有效的方法，该方法将原始的一对一匹配分支与辅助查询结合在一起，这些查询在训练过程中使用一对一的匹配损失。该混合策略已被证明可显着提高训练效率并提高准确性。在推断中，仅使用原始的一对一匹配分支，从而维持端到端的优点和相同的DETR推断效率。该方法命名为$ \ MATHCAL {H} $ - DETR，它表明可以在各种视觉任务中始终如一地改进各种代表性的DITR方法，包括可变形，3DDER/PETRV2，PETR和TRANDRACK， ，其他。代码将在以下网址提供：https：//github.com/hdetr

最近对物体检测的自我监督预防方法在很大程度上专注于预先绘制物体探测器的骨干，忽略了检测架构的关键部分。相反，我们介绍了DetReg，这是一种新的自我监督方法，用于预先列出整个对象检测网络，包括对象本地化和嵌入组件。在预先绘制期间，DetReg预测对象本地化以与无监督区域提议生成器匹配本地化，并同时将相应的特征嵌入与自我监控图像编码器的嵌入式对齐。我们使用DETR系列探测器实施DetReg，并显示它在Coco，Pascal VOC和空中客车船基准上的Fineetuned时改善了竞争性基线。在低数据制度中，包括半监督和几秒钟学习设置，DetReg建立了许多最先进的结果，例如，在Coco上，我们看到10次检测和+3.5的AP改进A +6.0 AP改进当培训只有1％的标签时。对于代码和预用模型，请访问https://amirbar.net/detreg的项目页面

Detection Transformer (DETR) directly transforms queries to unique objects by using one-to-one bipartite matching during training and enables end-to-end object detection. Recently, these models have surpassed traditional detectors on COCO with undeniable elegance. However, they differ from traditional detectors in multiple designs, including model architecture and training schedules, and thus the effectiveness of one-to-one matching is not fully understood. In this work, we conduct a strict comparison between the one-to-one Hungarian matching in DETRs and the one-to-many label assignments in traditional detectors with non-maximum supervision (NMS). Surprisingly, we observe one-to-many assignments with NMS consistently outperform standard one-to-one matching under the same setting, with a significant gain of up to 2.5 mAP. Our detector that trains Deformable-DETR with traditional IoU-based label assignment achieved 50.2 COCO mAP within 12 epochs (1x schedule) with ResNet50 backbone, outperforming all existing traditional or transformer-based detectors in this setting. On multiple datasets, schedules, and architectures, we consistently show bipartite matching is unnecessary for performant detection transformers. Furthermore, we attribute the success of detection transformers to their expressive transformer architecture. Code is available at https://github.com/jozhang97/DETA.

We present in this paper a novel denoising training method to speedup DETR (DEtection TRansformer) training and offer a deepened understanding of the slow convergence issue of DETR-like methods. We show that the slow convergence results from the instability of bipartite graph matching which causes inconsistent optimization goals in early training stages. To address this issue, except for the Hungarian loss, our method additionally feeds ground-truth bounding boxes with noises into Transformer decoder and trains the model to reconstruct the original boxes, which effectively reduces the bipartite graph matching difficulty and leads to a faster convergence. Our method is universal and can be easily plugged into any DETR-like methods by adding dozens of lines of code to achieve a remarkable improvement. As a result, our DN-DETR results in a remarkable improvement ($+1.9$AP) under the same setting and achieves the best result (AP $43.4$ and $48.6$ with $12$ and $50$ epochs of training respectively) among DETR-like methods with ResNet-$50$ backbone. Compared with the baseline under the same setting, DN-DETR achieves comparable performance with $50\%$ training epochs. Code is available at \url{https://github.com/FengLi-ust/DN-DETR}.

Modern object detectors have taken the advantages of backbone networks pre-trained on large scale datasets. Except for the backbone networks, however, other components such as the detector head and the feature pyramid network (FPN) remain trained from scratch, which hinders fully tapping the potential of representation models. In this study, we propose to integrally migrate pre-trained transformer encoder-decoders (imTED) to a detector, constructing a feature extraction path which is ``fully pre-trained" so that detectors' generalization capacity is maximized. The essential differences between imTED with the baseline detector are twofold: (1) migrating the pre-trained transformer decoder to the detector head while removing the randomly initialized FPN from the feature extraction path; and (2) defining a multi-scale feature modulator (MFM) to enhance scale adaptability. Such designs not only reduce randomly initialized parameters significantly but also unify detector training with representation learning intendedly. Experiments on the MS COCO object detection dataset show that imTED consistently outperforms its counterparts by $\sim$2.4 AP. Without bells and whistles, imTED improves the state-of-the-art of few-shot object detection by up to 7.6 AP. Code is available at https://github.com/LiewFeng/imTED.

Object detection with transformers (DETR) reaches competitive performance with Faster R-CNN via a transformer encoder-decoder architecture. Inspired by the great success of pre-training transformers in natural language processing, we propose a pretext task named random query patch detection to Unsupervisedly Pre-train DETR (UP-DETR) for object detection. Specifically, we randomly crop patches from the given image and then feed them as queries to the decoder. The model is pre-trained to detect these query patches from the original image. During the pre-training, we address two critical issues: multi-task learning and multi-query localization. (1) To trade off classification and localization preferences in the pretext task, we freeze the CNN backbone and propose a patch feature reconstruction branch which is jointly optimized with patch detection.(2) To perform multi-query localization, we introduce UP-DETR from single-query patch and extend it to multiquery patches with object query shuffle and attention mask. In our experiments, UP-DETR significantly boosts the performance of DETR with faster convergence and higher average precision on object detection, one-shot detection and panoptic segmentation. Code and pre-training models: https://github.com/dddzg/up-detr.

In this paper we present Mask DINO, a unified object detection and segmentation framework. Mask DINO extends DINO (DETR with Improved Denoising Anchor Boxes) by adding a mask prediction branch which supports all image segmentation tasks (instance, panoptic, and semantic). It makes use of the query embeddings from DINO to dot-product a high-resolution pixel embedding map to predict a set of binary masks. Some key components in DINO are extended for segmentation through a shared architecture and training process. Mask DINO is simple, efficient, and scalable, and it can benefit from joint large-scale detection and segmentation datasets. Our experiments show that Mask DINO significantly outperforms all existing specialized segmentation methods, both on a ResNet-50 backbone and a pre-trained model with SwinL backbone. Notably, Mask DINO establishes the best results to date on instance segmentation (54.5 AP on COCO), panoptic segmentation (59.4 PQ on COCO), and semantic segmentation (60.8 mIoU on ADE20K) among models under one billion parameters. Code is available at \url{https://github.com/IDEACVR/MaskDINO}.

在本文中，我们对检测变压器（DETR）感兴趣，这是一种基于变压器编码器编码器架构的端到端对象检测方法，而无需手工制作的后处理，例如NMS。受到有条件的Detr的启发，这是一种具有快速训练收敛性的改进的DETR，对内部解码器层提出了盒子查询（最初称为空间查询），我们将对象查询重新将对象查询重新布置为盒子查询的格式，该格式是参考参考嵌入的组成点和框相对于参考点的转换。该重新制定表明在更快地使用R-CNN中广泛研究的DETR中的对象查询与锚固框之间的联系。此外，我们从图像内容中学习了盒子查询，从而进一步提高了通过快速训练收敛的有条件DETR的检测质量。此外，我们采用轴向自我注意的想法来节省内存成本并加速编码器。所得的检测器（称为条件DETR V2）取得比条件DETR更好的结果，可节省内存成本并更有效地运行。例如，对于DC $ 5 $ -Resnet- $ 50 $骨干，我们的方法在可可$ Val $ set上获得了$ 44.8 $ ap，$ 16.4 $ fps和有条件的detr相比，它运行了$ 1.6 \ tims $ $ $ $ $，节省$ 74 $ \ \ \ \ \ \ \ \ \ \ \ \ \ $ 74美元总体内存成本的百分比，并提高$ 1.0 $ ap得分。

最近，基于合成数据的实例分割已成为一种极其有利的优化范式，因为它利用模拟渲染和物理学来生成高质量的图像宣传对。在本文中，我们提出了一个并行预训练的变压器（PPT）框架，以完成基于合成数据的实例分割任务。具体而言，我们利用现成的预训练的视觉变压器来减轻自然数据和合成数据之间的差距，这有助于在下游合成数据场景中提供良好的概括，几乎没有样本。基于SWIN-B基的CBNET V2，基于SWINL的CBNET V2和SWIN-L基统一器用于并行特征学习，并且这三个模型的结果由像素级非最大最大抑制（NMS）算法融合来获得更强大的结果。实验结果表明，PPT在CVPR2022 AVA可访问性视觉和自主性挑战中排名第一，地图为65.155％。

弱监督的对象检测（WSOD）使对象检测器能够使用图像级类标签训练对象检测器。但是，当前WSOD模型的实际应用是有限的，因为它们在小规模上运行，需要进行广泛的培训和精致。我们提出了弱监督的检测变压器，该变压器可以有效地从大规模预处理数据集到数百个新物体的WSOD列表有效地转移。我们利用预处理的知识来改善WSOD中使用的多个实例学习框架，并且实验表明我们的方法的表现优于数据集上的最新方法，其新颖类是本文的两倍。

We launch EVA, a vision-centric foundation model to explore the limits of visual representation at scale using only publicly accessible data. EVA is a vanilla ViT pre-trained to reconstruct the masked out image-text aligned vision features conditioned on visible image patches. Via this pretext task, we can efficiently scale up EVA to one billion parameters, and sets new records on a broad range of representative vision downstream tasks, such as image recognition, video action recognition, object detection, instance segmentation and semantic segmentation without heavy supervised training. Moreover, we observe quantitative changes in scaling EVA result in qualitative changes in transfer learning performance that are not present in other models. For instance, EVA takes a great leap in the challenging large vocabulary instance segmentation task: our model achieves almost the same state-of-the-art performance on LVISv1.0 dataset with over a thousand categories and COCO dataset with only eighty categories. Beyond a pure vision encoder, EVA can also serve as a vision-centric, multi-modal pivot to connect images and text. We find initializing the vision tower of a giant CLIP from EVA can greatly stabilize the training and outperform the training from scratch counterpart with much fewer samples and less compute, providing a new direction for scaling up and accelerating the costly training of multi-modal foundation models. To facilitate future research, we release all the code and models at https://github.com/baaivision/EVA.

多尺度功能已被证明在对象检测方面非常有效，大多数基于Convnet的对象检测器采用特征金字塔网络（FPN）作为利用多尺度功能的基本组件。但是，对于最近提出的基于变压器的对象探测器，直接结合多尺度功能会导致由于处理高分辨率特征的注意机制的高复杂性，因此导致了高度的计算开销。本文介绍了迭代多尺度特征聚合（IMFA） - 一种通用范式，可有效利用基于变压器的对象检测器中的多尺度特征。核心想法是从仅几个关键位置利用稀疏的多尺度特征，并且通过两种新颖的设计实现了稀疏的特征。首先，IMFA重新安排变压器编码器数据管道，因此可以根据检测预测进行迭代更新编码的功能。其次，在先前检测预测的指导下，IMFA稀疏的量表自适应特征可从几个关键点位置进行精制检测。结果，采样的多尺度特征稀疏，但仍然对对象检测非常有益。广泛的实验表明，提出的IMFA在略有计算开销的情况下显着提高了基于变压器的对象检测器的性能。项目页面：https：//github.com/zhanggongjie/imfa。

在这项工作中，我们研究了对象检测模型的自我监督预审计的不同方法。我们首先设计一个通用框架，通过随机采样和投射框来学习从图像中学习空间一致的密集表示，并将其投影到每个增强视图，并最大程度地提高相应的盒子功能之间的相似性。我们研究文献中的现有设计选择，例如盒子生成，功能提取策略，并使用其在实例级图像表示学习技术上获得成功启发的多种视图。我们的结果表明，该方法对超参数的不同选择是可靠的，并且使用多个视图不如实例级图像表示学习所显示的那样有效。我们还设计了两个辅助任务，以通过（1）通过使用对比度损失从采样设置中预测盒子中的一个视图中的框来预测框，并且（2）使用变压器预测盒子坐标，这可能会受益。下游对象检测任务。我们发现，在标记数据上预审计的模型时，这些任务不会导致更好的对象检测性能。

最近的端到端多对象检测器通过删除手工制作的过程（例如使用非最大最大抑制（NMS））删除手工制作的过程来简化推理管道。但是，在训练中，他们需要两分匹配来计算检测器输出的损失。与端到端学习的核心的方向性相反，双方匹配使端到端探测器复杂，启发式和依赖的培训。在本文中，我们提出了一种训练端到端多对象探测器而无需匹配的方法。为此，我们使用混合模型将端到端多对象检测作为密度估计问题。我们提出的检测器，称为稀疏混合物密度检测器（稀疏MDOD），使用混合模型估算边界盒的分布。稀疏MDOD是通过最大程度地减少负对数似然性和我们提出的正则化项，最大成分最大化（MCM）损失来训练的，从而阻止了重复的预测。在训练过程中，不需要其他过程，例如两分匹配，并且损失是直接从网络输出中计算出来的。此外，我们的稀疏MDOD优于MS-Coco上的现有检测器，MS-Coco是一种著名的多对象检测基准。

The DETR object detection approach applies the transformer encoder and decoder architecture to detect objects and achieves promising performance. In this paper, we present a simple approach to address the main problem of DETR, the slow convergence, by using representation learning technique. In this approach, we detect an object bounding box as a pair of keypoints, the top-left corner and the center, using two decoders. By detecting objects as paired keypoints, the model builds up a joint classification and pair association on the output queries from two decoders. For the pair association we propose utilizing contrastive self-supervised learning algorithm without requiring specialized architecture. Experimental results on MS COCO dataset show that Pair DETR can converge at least 10x faster than original DETR and 1.5x faster than Conditional DETR during training, while having consistently higher Average Precision scores.

DETR是使用变压器编码器 - 解码器架构的第一端到端对象检测器，并在高分辨率特征映射上展示竞争性能但低计算效率。随后的工作变形Detr，通过更换可变形的关注来提高DEDR的效率，这实现了10倍的收敛性和改进的性能。可变形DETR使用多尺度特征来改善性能，但是，与DETR相比，编码器令牌的数量增加了20倍，编码器注意的计算成本仍然是瓶颈。在我们的初步实验中，我们观察到，即使只更新了编码器令牌的一部分，检测性能也几乎没有恶化。灵感来自该观察，我们提出了稀疏的DETR，其仅选择性更新预期的解码器预期的令牌，从而有效地检测模型。此外，我们表明在编码器中的所选令牌上应用辅助检测丢失可以提高性能，同时最小化计算开销。即使在Coco数据集上只有10％的编码器令牌，我们验证稀疏DETR也可以比可变形DETR实现更好的性能。尽管只有编码器令牌稀疏，但总计算成本减少了38％，与可变形的Detr相比，每秒帧（FPS）增加42％。代码可在https://github.com/kakaobrain/sparse-dett