尽管有不同的相关框架，已经通过不同和专门的框架解决了语义，实例和Panoptic分段。本文为这些基本相似的任务提供了统一，简单，有效的框架。该框架，名为K-Net，段段由一组被学习内核持续一致，其中每个内核负责为潜在实例或填充类生成掩码。要解决区分各种实例的困难，我们提出了一个内核更新策略，使每个内核动态和条件在输入图像中的有意义的组上。 K-NET可以以结尾的方式培训，具有二分匹配，其培训和推论是自然的NMS和无框。没有钟声和口哨，K-Net超越了先前发表的全面的全面的单一模型，在ADE20K Val上的MS Coco Test-Dev分割和语义分割上分别与55.2％PQ和54.3％Miou分裂。其实例分割性能也与MS COCO上的级联掩模R-CNN相同，具有60％-90％的推理速度。代码和模型将在https://github.com/zwwwayne/k-net/发布。

图像分割是关于使用不同语义的分组像素，例如类别或实例成员身份，其中每个语义选择定义任务。虽然只有每个任务的语义不同，但目前的研究侧重于为每项任务设计专业架构。我们提出了蒙面关注掩模变压器（Mask2Former），这是一种能够寻址任何图像分段任务（Panoptic，实例或语义）的新架构。其关键部件包括屏蔽注意，通过限制预测掩模区域内的横向提取局部特征。除了将研究工作减少三次之外，它还优于四个流行的数据集中的最佳专业架构。最值得注意的是，Mask2Former为Panoptic semonation（Coco 57.8 PQ）设置了新的最先进的，实例分段（Coco上50.1 AP）和语义分割（ADE20K上的57.7 miou）。

Panoptic semonation涉及联合语义分割和实例分割的组合，其中图像内容分为两种类型：事物和东西。我们展示了Panoptic SegFormer，是与变压器的Panoptic Semonation的一般框架。它包含三个创新组件：高效的深度监督掩模解码器，查询解耦策略以及改进的后处理方法。我们还使用可变形的DETR来有效地处理多尺度功能，这是一种快速高效的DETR版本。具体而言，我们以层式方式监督掩模解码器中的注意模块。这种深度监督策略让注意模块快速关注有意义的语义区域。与可变形的DETR相比，它可以提高性能并将所需培训纪元的数量减少一半。我们的查询解耦策略对查询集的职责解耦并避免了事物和东西之间的相互干扰。此外，我们的后处理策略通过联合考虑分类和分割质量来解决突出的面具重叠而没有额外成本的情况。我们的方法会在基线DETR模型上增加6.2 \％PQ。 Panoptic SegFormer通过56.2 \％PQ实现最先进的结果。它还显示出对现有方法的更强大的零射鲁布利。代码释放\ url {https://github.com/zhiqi-li/panoptic-segformer}。

现代方法通常将语义分割标记为每个像素分类任务，而使用替代掩码分类处理实例级分割。我们的主要洞察力：掩码分类是足够的一般，可以使用完全相同的模型，丢失和培训过程来解决语义和实例级分段任务。在此观察之后，我们提出了一个简单的掩模分类模型，该模型预测了一组二进制掩码，每个模型与单个全局类标签预测相关联。总的来说，所提出的基于掩模分类的方法简化了语义和Panoptic分割任务的有效方法的景观，并显示出优异的经验结果。特别是，当类的数量大时，我们观察到掩码形成器优于每个像素分类基线。我们的面具基于分类的方法优于当前最先进的语义（ADE20K上的55.6 miou）和Panoptic Seation（Coco）模型的Panoptic Seationation（52.7 PQ）。

In this paper we present Mask DINO, a unified object detection and segmentation framework. Mask DINO extends DINO (DETR with Improved Denoising Anchor Boxes) by adding a mask prediction branch which supports all image segmentation tasks (instance, panoptic, and semantic). It makes use of the query embeddings from DINO to dot-product a high-resolution pixel embedding map to predict a set of binary masks. Some key components in DINO are extended for segmentation through a shared architecture and training process. Mask DINO is simple, efficient, and scalable, and it can benefit from joint large-scale detection and segmentation datasets. Our experiments show that Mask DINO significantly outperforms all existing specialized segmentation methods, both on a ResNet-50 backbone and a pre-trained model with SwinL backbone. Notably, Mask DINO establishes the best results to date on instance segmentation (54.5 AP on COCO), panoptic segmentation (59.4 PQ on COCO), and semantic segmentation (60.8 mIoU on ADE20K) among models under one billion parameters. Code is available at \url{https://github.com/IDEACVR/MaskDINO}.

Panoptic Part Segmentation (PPS) unifies panoptic segmentation and part segmentation into one task. Previous works utilize separated approaches to handle thing, stuff, and part predictions without shared computation and task association. We aim to unify these tasks at the architectural level, designing the first end-to-end unified framework named Panoptic-PartFormer. Moreover, we find the previous metric PartPQ biases to PQ. To handle both issues, we make the following contributions: Firstly, we design a meta-architecture that decouples part feature and things/stuff feature, respectively. We model things, stuff, and parts as object queries and directly learn to optimize all three forms of prediction as a unified mask prediction and classification problem. We term our model as Panoptic-PartFormer. Secondly, we propose a new metric Part-Whole Quality (PWQ) to better measure such task from both pixel-region and part-whole perspectives. It can also decouple the error for part segmentation and panoptic segmentation. Thirdly, inspired by Mask2Former, based on our meta-architecture, we propose Panoptic-PartFormer++ and design a new part-whole cross attention scheme to further boost part segmentation qualities. We design a new part-whole interaction method using masked cross attention. Finally, the extensive ablation studies and analysis demonstrate the effectiveness of both Panoptic-PartFormer and Panoptic-PartFormer++. Compared with previous Panoptic-PartFormer, our Panoptic-PartFormer++ achieves 2% PartPQ and 3% PWQ improvements on the Cityscapes PPS dataset and 5% PartPQ on the Pascal Context PPS dataset. On both datasets, Panoptic-PartFormer++ achieves new state-of-the-art results with a significant cost drop of 70% on GFlops and 50% on parameters. Our models can serve as a strong baseline and aid future research in PPS. Code will be available.

We present a new, embarrassingly simple approach to instance segmentation. Compared to many other dense prediction tasks, e.g., semantic segmentation, it is the arbitrary number of instances that have made instance segmentation much more challenging. In order to predict a mask for each instance, mainstream approaches either follow the "detect-then-segment" strategy (e.g., Mask R-CNN), or predict embedding vectors first then use clustering techniques to group pixels into individual instances. We view the task of instance segmentation from a completely new perspective by introducing the notion of "instance categories", which assigns categories to each pixel within an instance according to the instance's location and size, thus nicely converting instance segmentation into a single-shot classification-solvable problem. We demonstrate a much simpler and flexible instance segmentation framework with strong performance, achieving on par accuracy with Mask R-CNN and outperforming recent single-shot instance segmenters in accuracy. We hope that this simple and strong framework can serve as a baseline for many instance-level recognition tasks besides instance segmentation. Code is available at https://git.io/AdelaiDet

Cascade is a classic yet powerful architecture that has boosted performance on various tasks. However, how to introduce cascade to instance segmentation remains an open question. A simple combination of Cascade R-CNN and Mask R-CNN only brings limited gain. In exploring a more effective approach, we find that the key to a successful instance segmentation cascade is to fully leverage the reciprocal relationship between detection and segmentation. In this work, we propose a new framework, Hybrid Task Cascade (HTC), which differs in two important aspects: (1) instead of performing cascaded refinement on these two tasks separately, it interweaves them for a joint multi-stage processing; (2) it adopts a fully convolutional branch to provide spatial context, which can help distinguishing hard foreground from cluttered background. Overall, this framework can learn more discriminative features progressively while integrating complementary features together in each stage. Without bells and whistles, a single HTC obtains 38.4% and 1.5% improvement over a strong Cascade Mask R-CNN baseline on MSCOCO dataset. Moreover, our overall system achieves 48.6 mask AP on the test-challenge split, ranking 1st in the COCO 2018 Challenge Object Detection Task. Code is available at: https://github.com/ open-mmlab/mmdetection.

本文介绍了端到端的实例分段框架，称为SOIT，该段具有实例感知变压器的段对象。灵感来自Detr〜\ Cite {carion2020end}，我们的方法视图实例分段为直接设置预测问题，有效地消除了对ROI裁剪，一对多标签分配等许多手工制作组件的需求，以及非最大抑制（ nms）。在SOIT中，通过在全局图像上下文下直接地将多个查询直接理解语义类别，边界框位置和像素 - WISE掩码的一组对象嵌入。类和边界盒可以通过固定长度的向量轻松嵌入。尤其是由一组参数嵌入像素方面的掩模以构建轻量级实例感知变压器。之后，实例感知变压器产生全分辨率掩码，而不涉及基于ROI的任何操作。总的来说，SOIT介绍了一个简单的单级实例分段框架，它是无乐和NMS的。 MS Coco DataSet上的实验结果表明，优于最先进的实例分割显着的优势。此外，在统一查询嵌入中的多个任务的联合学习还可以大大提高检测性能。代码可用于\ url {https://github.com/yuxiaodonghri/soit}。

The recently introduced panoptic segmentation task has renewed our community's interest in unifying the tasks of instance segmentation (for thing classes) and semantic segmentation (for stuff classes). However, current state-ofthe-art methods for this joint task use separate and dissimilar networks for instance and semantic segmentation, without performing any shared computation. In this work, we aim to unify these methods at the architectural level, designing a single network for both tasks. Our approach is to endow Mask R-CNN, a popular instance segmentation method, with a semantic segmentation branch using a shared Feature Pyramid Network (FPN) backbone. Surprisingly, this simple baseline not only remains effective for instance segmentation, but also yields a lightweight, topperforming method for semantic segmentation. In this work, we perform a detailed study of this minimally extended version of Mask R-CNN with FPN, which we refer to as Panoptic FPN, and show it is a robust and accurate baseline for both tasks. Given its effectiveness and conceptual simplicity, we hope our method can serve as a strong baseline and aid future research in panoptic segmentation.

全景部分分割（PPS）旨在将泛型分割和部分分割统一为一个任务。先前的工作主要利用分离的方法来处理事物，物品和部分预测，而无需执行任何共享的计算和任务关联。在这项工作中，我们旨在将这些任务统一在架构层面上，设计第一个名为Panoptic-Partformer的端到端统一方法。特别是，由于视觉变压器的最新进展，我们将事物，内容和部分建模为对象查询，并直接学会优化所有三个预测作为统一掩码的预测和分类问题。我们设计了一个脱钩的解码器，以分别生成零件功能和事物/东西功能。然后，我们建议利用所有查询和相应的特征共同执行推理。最终掩码可以通过查询和相应特征之间的内部产品获得。广泛的消融研究和分析证明了我们框架的有效性。我们的全景局势群体在CityScapes PPS和Pascal Context PPS数据集上实现了新的最新结果，至少有70％的GFLOPS和50％的参数降低。特别是，在Pascal上下文PPS数据集上采用SWIN Transformer后，我们可以通过RESNET50骨干链和10％的改进获得3.4％的相对改进。据我们所知，我们是第一个通过\ textit {统一和端到端变压器模型来解决PPS问题的人。鉴于其有效性和概念上的简单性，我们希望我们的全景贡献者能够充当良好的基准，并帮助未来的PPS统一研究。我们的代码和型号可在https://github.com/lxtgh/panoptic-partformer上找到。

最近提出的深度感知视频Panoptic分段（DVPS）旨在预测视频中的Panoptic分段结果和深度映射，这是一个具有挑战性的场景理解问题。在本文中，我们提供了多相变压器，揭示了DVPS任务下的所有子任务。我们的方法通过基于查询的学习探讨了深度估计与Panoptic分割的关系。特别是，我们设计三个不同的查询，包括查询，填写询问和深度查询的东西。然后我们建议通过门控融合来学习这些查询之间的相关性。从实验中，我们从深度估计和Panoptic分割方面证明了我们设计的好处。由于每个物品查询还对实例信息进行了编码，因此通过具有外观学习的裁剪实例掩码功能来执行跟踪是自然的。我们的方法在ICCV-2021 BMTT挑战视频+深度轨道上排名第一。据报道，消融研究表明我们如何提高性能。代码将在https://github.com/harboryuan/polyphonicformer提供。

现有的实例分割方法已经达到了令人印象深刻的表现，但仍遭受了共同的困境：一个实例推断出冗余表示（例如，多个框，网格和锚点），这导致了多个重复的预测。因此，主流方法通常依赖于手工设计的非最大抑制（NMS）后处理步骤来选择最佳预测结果，这会阻碍端到端训练。为了解决此问题，我们建议一个称为Uniinst的无盒和无端机实例分割框架，该框架仅对每个实例产生一个唯一的表示。具体而言，我们设计了一种实例意识到的一对一分配方案，即仅产生一个表示（Oyor），该方案根据预测和地面真相之间的匹配质量，动态地为每个实例动态分配一个独特的表示。然后，一种新颖的预测重新排列策略被优雅地集成到框架中，以解决分类评分和掩盖质量之间的错位，从而使学习的表示形式更具歧视性。借助这些技术，我们的Uniinst，第一个基于FCN的盒子和无NMS实例分段框架，实现竞争性能，例如，使用Resnet-50-FPN和40.2 mask AP使用Resnet-101-FPN，使用Resnet-50-FPN和40.2 mask AP，使用Resnet-101-FPN，对抗AP可可测试-DEV的主流方法。此外，提出的实例感知方法对于遮挡场景是可靠的，在重锁定的ochuman基准上，通过杰出的掩码AP优于公共基线。我们的代码将在出版后提供。

In this paper, we propose a unified panoptic segmentation network (UPSNet) for tackling the newly proposed panoptic segmentation task. On top of a single backbone residual network, we first design a deformable convolution based semantic segmentation head and a Mask R-CNN style instance segmentation head which solve these two subtasks simultaneously. More importantly, we introduce a parameter-free panoptic head which solves the panoptic segmentation via pixel-wise classification. It first leverages the logits from the previous two heads and then innovatively expands the representation for enabling prediction of an extra unknown class which helps better resolve the conflicts between semantic and instance segmentation. Additionally, it handles the challenge caused by the varying number of instances and permits back propagation to the bottom modules in an end-to-end manner. Extensive experimental results on Cityscapes, COCO and our internal dataset demonstrate that our UPSNet achieves stateof-the-art performance with much faster inference. Code has been made available at: https://github.com/ uber-research/UPSNet. * Equal contribution.† This work was done when Hengshuang Zhao was an intern at Uber ATG.

我们介绍了一个新的图像分段任务，称为实体分段（ES），该任务旨在在不预测其语义标签的情况下划分图像中的所有视觉实体（对象和填充）。通过删除类标签预测的需要，对此类任务培训的模型可以更多地关注提高分割质量。它具有许多实际应用，例如图像操纵和编辑，其中分割掩模的质量至关重要，但类标签不太重要。我们通过统一的方式调查第一次研究，以调查卷大中心的代表对分割事物和东西的可行性，并显示这种代表在es的背景下非常好。更具体地说，我们提出了一种类似的完全卷积的架构，具有两种新颖的模块，专门设计用于利用es的类无话和非重叠要求。实验表明，在分割质量方面设计和培训的模型显着优于流行的专用Panoptic分段模型。此外，可以在多个数据集的组合中容易地培训ES模型，而无需解决数据集合并中的标签冲突，并且在一个或多个数据集中培训的模型可以概括到未经看管域的其他测试数据集。代码已在https://github.com/dvlab-research/entity发布。

本文提出了一种用于对象和场景的高质量图像分割的新方法。灵感来自于形态学图像处理技术中的扩张和侵蚀操作，像素级图像分割问题被视为挤压对象边界。从这个角度来看，提出了一种新颖且有效的\ textBF {边界挤压}模块。该模块用于从内侧和外侧方向挤压对象边界，这有助于精确掩模表示。提出了双向基于流的翘曲过程来产生这种挤压特征表示，并且设计了两个特定的损耗信号以监控挤压过程。边界挤压模块可以通过构建一些现有方法构建作为即插即用模块，可以轻松应用于实例和语义分段任务。此外，所提出的模块是重量的，因此具有实际使用的潜力。实验结果表明，我们简单但有效的设计可以在几个不同的数据集中产生高质量的结果。此外，边界上的其他几个指标用于证明我们对以前的工作中的方法的有效性。我们的方法对实例和语义分割的具有利于Coco和CityCapes数据集来产生重大改进，并且在相同的设置下以前的最先进的速度优于先前的最先进的速度。代码和模型将在\ url {https:/github.com/lxtgh/bsseg}发布。

In contrast to fully supervised methods using pixel-wise mask labels, box-supervised instance segmentation takes advantage of simple box annotations, which has recently attracted increasing research attention. This paper presents a novel single-shot instance segmentation approach, namely Box2Mask, which integrates the classical level-set evolution model into deep neural network learning to achieve accurate mask prediction with only bounding box supervision. Specifically, both the input image and its deep features are employed to evolve the level-set curves implicitly, and a local consistency module based on a pixel affinity kernel is used to mine the local context and spatial relations. Two types of single-stage frameworks, i.e., CNN-based and transformer-based frameworks, are developed to empower the level-set evolution for box-supervised instance segmentation, and each framework consists of three essential components: instance-aware decoder, box-level matching assignment and level-set evolution. By minimizing the level-set energy function, the mask map of each instance can be iteratively optimized within its bounding box annotation. The experimental results on five challenging testbeds, covering general scenes, remote sensing, medical and scene text images, demonstrate the outstanding performance of our proposed Box2Mask approach for box-supervised instance segmentation. In particular, with the Swin-Transformer large backbone, our Box2Mask obtains 42.4% mask AP on COCO, which is on par with the recently developed fully mask-supervised methods. The code is available at: https://github.com/LiWentomng/boxlevelset.

两阶段和基于查询的实例分段方法取得了显着的结果。然而，他们的分段面具仍然非常粗糙。在本文中，我们呈现了用于高质量高效的实例分割的掩模转发器。我们的掩模转发器代替常规密集的张量，而不是在常规密集的张量上进行分解，并表示作为Quadtree的图像区域。我们基于变换器的方法仅处理检测到的错误易于树节点，并并行自我纠正其错误。虽然这些稀疏的像素仅构成总数的小比例，但它们对最终掩模质量至关重要。这允许掩模转换器以低计算成本预测高精度的实例掩模。广泛的实验表明，掩模转发器在三个流行的基准上优于当前实例分段方法，显着改善了COCO和BDD100K上的大型+3.0掩模AP的+3.0掩模AP的大余量和CityScapes上的+6.6边界AP。我们的代码和培训的型号将在http://vis.xyz/pub/transfiner提供。

我们提出了聚类蒙版变压器（CMT-DeepLab），这是一种基于变压器的框架，用于围绕聚类设计的泛型分割。它重新考虑了用于分割和检测的现有变压器架构；CMT-DeepLab认为对象查询是群集中心，该中心填充了应用于分割时将像素分组的作用。群集通过交替的过程计算，首先通过其功能亲和力将像素分配给簇，然后更新集群中心和像素功能。这些操作共同包含聚类蒙版变压器（CMT）层，该层产生了越野器的交叉注意，并且与最终的分割任务更加一致。CMT-DeepLab在可可Test-DEV集中实现了55.7％的PQ的新最先进的PQ，可显着提高先前ART的性能。

Universal Image Segmentation is not a new concept. Past attempts to unify image segmentation in the last decades include scene parsing, panoptic segmentation, and, more recently, new panoptic architectures. However, such panoptic architectures do not truly unify image segmentation because they need to be trained individually on the semantic, instance, or panoptic segmentation to achieve the best performance. Ideally, a truly universal framework should be trained only once and achieve SOTA performance across all three image segmentation tasks. To that end, we propose OneFormer, a universal image segmentation framework that unifies segmentation with a multi-task train-once design. We first propose a task-conditioned joint training strategy that enables training on ground truths of each domain (semantic, instance, and panoptic segmentation) within a single multi-task training process. Secondly, we introduce a task token to condition our model on the task at hand, making our model task-dynamic to support multi-task training and inference. Thirdly, we propose using a query-text contrastive loss during training to establish better inter-task and inter-class distinctions. Notably, our single OneFormer model outperforms specialized Mask2Former models across all three segmentation tasks on ADE20k, CityScapes, and COCO, despite the latter being trained on each of the three tasks individually with three times the resources. With new ConvNeXt and DiNAT backbones, we observe even more performance improvement. We believe OneFormer is a significant step towards making image segmentation more universal and accessible. To support further research, we open-source our code and models at https://github.com/SHI-Labs/OneFormer