We present a new, embarrassingly simple approach to instance segmentation. Compared to many other dense prediction tasks, e.g., semantic segmentation, it is the arbitrary number of instances that have made instance segmentation much more challenging. In order to predict a mask for each instance, mainstream approaches either follow the "detect-then-segment" strategy (e.g., Mask R-CNN), or predict embedding vectors first then use clustering techniques to group pixels into individual instances. We view the task of instance segmentation from a completely new perspective by introducing the notion of "instance categories", which assigns categories to each pixel within an instance according to the instance's location and size, thus nicely converting instance segmentation into a single-shot classification-solvable problem. We demonstrate a much simpler and flexible instance segmentation framework with strong performance, achieving on par accuracy with Mask R-CNN and outperforming recent single-shot instance segmenters in accuracy. We hope that this simple and strong framework can serve as a baseline for many instance-level recognition tasks besides instance segmentation. Code is available at https://git.io/AdelaiDet

We propose a simple yet effective instance segmentation framework, termed CondInst (conditional convolutions for instance segmentation). Top-performing instance segmentation methods such as Mask R-CNN rely on ROI operations (typically ROIPool or ROIAlign) to obtain the final instance masks. In contrast, we propose to solve instance segmentation from a new perspective. Instead of using instancewise ROIs as inputs to a network of fixed weights, we employ dynamic instance-aware networks, conditioned on instances. CondInst enjoys two advantages: 1) Instance segmentation is solved by a fully convolutional network, eliminating the need for ROI cropping and feature alignment.2) Due to the much improved capacity of dynamically-generated conditional convolutions, the mask head can be very compact (e.g., 3 conv. layers, each having only 8 channels), leading to significantly faster inference. We demonstrate a simpler instance segmentation method that can achieve improved performance in both accuracy and inference speed. On the COCO dataset, we outperform a few recent methods including welltuned Mask R-CNN baselines, without longer training schedules needed.

We present a conceptually simple, flexible, and general framework for object instance segmentation. Our approach efficiently detects objects in an image while simultaneously generating a high-quality segmentation mask for each instance. The method, called Mask R-CNN, extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. Mask R-CNN is simple to train and adds only a small overhead to Faster R-CNN, running at 5 fps. Moreover, Mask R-CNN is easy to generalize to other tasks, e.g., allowing us to estimate human poses in the same framework. We show top results in all three tracks of the COCO suite of challenges, including instance segmentation, bounding-box object detection, and person keypoint detection. Without tricks, Mask R-CNN outperforms all existing, single-model entries on every task, including the COCO 2016 challenge winners. We hope our simple and effective approach will serve as a solid baseline and help ease future research in instance-level recognition. Code will be made available.

现有的实例分割方法已经达到了令人印象深刻的表现，但仍遭受了共同的困境：一个实例推断出冗余表示（例如，多个框，网格和锚点），这导致了多个重复的预测。因此，主流方法通常依赖于手工设计的非最大抑制（NMS）后处理步骤来选择最佳预测结果，这会阻碍端到端训练。为了解决此问题，我们建议一个称为Uniinst的无盒和无端机实例分割框架，该框架仅对每个实例产生一个唯一的表示。具体而言，我们设计了一种实例意识到的一对一分配方案，即仅产生一个表示（Oyor），该方案根据预测和地面真相之间的匹配质量，动态地为每个实例动态分配一个独特的表示。然后，一种新颖的预测重新排列策略被优雅地集成到框架中，以解决分类评分和掩盖质量之间的错位，从而使学习的表示形式更具歧视性。借助这些技术，我们的Uniinst，第一个基于FCN的盒子和无NMS实例分段框架，实现竞争性能，例如，使用Resnet-50-FPN和40.2 mask AP使用Resnet-101-FPN，使用Resnet-50-FPN和40.2 mask AP，使用Resnet-101-FPN，对抗AP可可测试-DEV的主流方法。此外，提出的实例感知方法对于遮挡场景是可靠的，在重锁定的ochuman基准上，通过杰出的掩码AP优于公共基线。我们的代码将在出版后提供。

In this paper, we introduce an anchor-box free and single shot instance segmentation method, which is conceptually simple, fully convolutional and can be used by easily embedding it into most off-the-shelf detection methods. Our method, termed PolarMask, formulates the instance segmentation problem as predicting contour of instance through instance center classification and dense distance regression in a polar coordinate. Moreover, we propose two effective approaches to deal with sampling high-quality center examples and optimization for dense distance regression, respectively, which can significantly improve the performance and simplify the training process. Without any bells and whistles, PolarMask achieves 32.9% in mask mAP with single-model and single-scale training/testing on the challenging COCO dataset.For the first time, we show that the complexity of instance segmentation, in terms of both design and computation complexity, can be the same as bounding box object detection and this much simpler and flexible instance segmentation framework can achieve competitive accuracy. We hope that the proposed PolarMask framework can serve as a fundamental and strong baseline for single shot instance segmentation task. Code is available at: github.com/xieenze/PolarMask.

We propose a fully convolutional one-stage object detector (FCOS) to solve object detection in a per-pixel prediction fashion, analogue to semantic segmentation. Almost all state-of-the-art object detectors such as RetinaNet, SSD, YOLOv3, and Faster R-CNN rely on pre-defined anchor boxes. In contrast, our proposed detector FCOS is anchor box free, as well as proposal free. By eliminating the predefined set of anchor boxes, FCOS completely avoids the complicated computation related to anchor boxes such as calculating overlapping during training. More importantly, we also avoid all hyper-parameters related to anchor boxes, which are often very sensitive to the final detection performance. With the only post-processing non-maximum suppression (NMS), FCOS with ResNeXt-64x4d-101 achieves 44.7% in AP with single-model and single-scale testing, surpassing previous one-stage detectors with the advantage of being much simpler. For the first time, we demonstrate a much simpler and flexible detection framework achieving improved detection accuracy. We hope that the proposed FCOS framework can serve as a simple and strong alternative for many other instance-level tasks. Code is available at:tinyurl.com/FCOSv1

尽管有不同的相关框架，已经通过不同和专门的框架解决了语义，实例和Panoptic分段。本文为这些基本相似的任务提供了统一，简单，有效的框架。该框架，名为K-Net，段段由一组被学习内核持续一致，其中每个内核负责为潜在实例或填充类生成掩码。要解决区分各种实例的困难，我们提出了一个内核更新策略，使每个内核动态和条件在输入图像中的有意义的组上。 K-NET可以以结尾的方式培训，具有二分匹配，其培训和推论是自然的NMS和无框。没有钟声和口哨，K-Net超越了先前发表的全面的全面的单一模型，在ADE20K Val上的MS Coco Test-Dev分割和语义分割上分别与55.2％PQ和54.3％Miou分裂。其实例分割性能也与MS COCO上的级联掩模R-CNN相同，具有60％-90％的推理速度。代码和模型将在https://github.com/zwwwayne/k-net/发布。

Letting a deep network be aware of the quality of its own predictions is an interesting yet important problem. In the task of instance segmentation, the confidence of instance classification is used as mask quality score in most instance segmentation frameworks. However, the mask quality, quantified as the IoU between the instance mask and its ground truth, is usually not well correlated with classification score. In this paper, we study this problem and propose Mask Scoring R-CNN which contains a network block to learn the quality of the predicted instance masks. The proposed network block takes the instance feature and the corresponding predicted mask together to regress the mask IoU. The mask scoring strategy calibrates the misalignment between mask quality and mask score, and improves instance segmentation performance by prioritizing more accurate mask predictions during COCO AP evaluation. By extensive evaluations on the COCO dataset, Mask Scoring R-CNN brings consistent and noticeable gain with different models, and outperforms the state-of-the-art Mask R-CNN. We hope our simple and effective approach will provide a new direction for improving instance segmentation. The source code of our method is available at https:// github.com/zjhuang22/maskscoring_rcnn. * The work was done when Zhaojin Huang was an intern in Horizon Robotics Inc.

自动核细胞分割和分类在数字病理学中起着至关重要的作用。但是，以前的作品主要基于具有有限的多样性和小尺寸的数据构建，使得在实际下游任务中的结果可疑或误导。在本文中，我们的目标是建立一种可靠且强大的方法，能够处理“临床野生”中的数据。具体地，我们研究和设计一种同时检测，分段和分类来自血红素和曙红（H＆E）染色的组织病理学数据的新方法，并使用最近的最大数据集评估我们的方法：Pannuke。我们以新颖的语义关键点估计问题解决每个核的检测和分类，以确定每个核的中心点。接下来，使用动态实例分段获得核心点的相应类别 - 不可止液掩模。通过解耦两个同步具有挑战性的任务，我们的方法可以从类别感知的检测和类别不可知的细分中受益，从而导致显着的性能提升。我们展示了我们提出的核细胞分割和分类方法的卓越性能，跨越19种不同的组织类型，提供了新的基准结果。

Recent state-of-the-art one-stage instance segmentation model SOLO divides the input image into a grid and directly predicts per grid cell object masks with fully-convolutional networks, yielding comparably good performance as traditional two-stage Mask R-CNN yet enjoying much simpler architecture and higher efficiency. We observe SOLO generates similar masks for an object at nearby grid cells, and these neighboring predictions can complement each other as some may better segment certain object part, most of which are however directly discarded by non-maximum-suppression. Motivated by the observed gap, we develop a novel learning-based aggregation method that improves upon SOLO by leveraging the rich neighboring information while maintaining the architectural efficiency. The resulting model is named SODAR. Unlike the original per grid cell object masks, SODAR is implicitly supervised to learn mask representations that encode geometric structure of nearby objects and complement adjacent representations with context. The aggregation method further includes two novel designs: 1) a mask interpolation mechanism that enables the model to generate much fewer mask representations by sharing neighboring representations among nearby grid cells, and thus saves computation and memory; 2) a deformable neighbour sampling mechanism that allows the model to adaptively adjust neighbor sampling locations thus gathering mask representations with more relevant context and achieving higher performance. SODAR significantly improves the instance segmentation performance, e.g., it outperforms a SOLO model with ResNet-101 backbone by 2.2 AP on COCO \texttt{test} set, with only about 3\% additional computation. We further show consistent performance gain with the SOLOv2 model.

本文提出了一种用于对象和场景的高质量图像分割的新方法。灵感来自于形态学图像处理技术中的扩张和侵蚀操作，像素级图像分割问题被视为挤压对象边界。从这个角度来看，提出了一种新颖且有效的\ textBF {边界挤压}模块。该模块用于从内侧和外侧方向挤压对象边界，这有助于精确掩模表示。提出了双向基于流的翘曲过程来产生这种挤压特征表示，并且设计了两个特定的损耗信号以监控挤压过程。边界挤压模块可以通过构建一些现有方法构建作为即插即用模块，可以轻松应用于实例和语义分段任务。此外，所提出的模块是重量的，因此具有实际使用的潜力。实验结果表明，我们简单但有效的设计可以在几个不同的数据集中产生高质量的结果。此外，边界上的其他几个指标用于证明我们对以前的工作中的方法的有效性。我们的方法对实例和语义分割的具有利于Coco和CityCapes数据集来产生重大改进，并且在相同的设置下以前的最先进的速度优于先前的最先进的速度。代码和模型将在\ url {https:/github.com/lxtgh/bsseg}发布。

Feature pyramids are a basic component in recognition systems for detecting objects at different scales. But recent deep learning object detectors have avoided pyramid representations, in part because they are compute and memory intensive. In this paper, we exploit the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A topdown architecture with lateral connections is developed for building high-level semantic feature maps at all scales. This architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using FPN in a basic Faster R-CNN system, our method achieves state-of-the-art singlemodel results on the COCO detection benchmark without bells and whistles, surpassing all existing single-model entries including those from the COCO 2016 challenge winners. In addition, our method can run at 6 FPS on a GPU and thus is a practical and accurate solution to multi-scale object detection. Code will be made publicly available.

We present a simple, fully-convolutional model for realtime instance segmentation that achieves 29.8 mAP on MS COCO at 33.5 fps evaluated on a single Titan Xp, which is significantly faster than any previous competitive approach. Moreover, we obtain this result after training on only one GPU. We accomplish this by breaking instance segmentation into two parallel subtasks: (1) generating a set of prototype masks and (2) predicting per-instance mask coefficients. Then we produce instance masks by linearly combining the prototypes with the mask coefficients. We find that because this process doesn't depend on repooling, this approach produces very high-quality masks and exhibits temporal stability for free. Furthermore, we analyze the emergent behavior of our prototypes and show they learn to localize instances on their own in a translation variant manner, despite being fully-convolutional. Finally, we also propose Fast NMS, a drop-in 12 ms faster replacement for standard NMS that only has a marginal performance penalty.

分割高度重叠的图像对象是具有挑战性的，因为图像上的真实对象轮廓和遮挡边界之间通常没有区别。与先前的实例分割方法不同，我们将图像形成模拟为两个重叠层的组成，并提出了双层卷积网络（BCNET），其中顶层检测到遮挡对象（遮挡器），而底层则渗透到部分闭塞实例（胶囊）。遮挡关系与双层结构的显式建模自然地将遮挡和遮挡实例的边界解散，并在掩模回归过程中考虑了它们之间的相互作用。我们使用两种流行的卷积网络设计（即完全卷积网络（FCN）和图形卷积网络（GCN））研究了双层结构的功效。此外，我们通过将图像中的实例表示为单独的可学习封闭器和封闭者查询，从而使用视觉变压器（VIT）制定双层解耦。使用一个/两个阶段和基于查询的对象探测器具有各种骨架和网络层选择验证双层解耦合的概括能力，如图像实例分段基准（可可，亲戚，可可）和视频所示实例分割基准（YTVIS，OVIS，BDD100K MOTS），特别是对于重闭塞病例。代码和数据可在https://github.com/lkeab/bcnet上找到。

In this paper, we propose a unified panoptic segmentation network (UPSNet) for tackling the newly proposed panoptic segmentation task. On top of a single backbone residual network, we first design a deformable convolution based semantic segmentation head and a Mask R-CNN style instance segmentation head which solve these two subtasks simultaneously. More importantly, we introduce a parameter-free panoptic head which solves the panoptic segmentation via pixel-wise classification. It first leverages the logits from the previous two heads and then innovatively expands the representation for enabling prediction of an extra unknown class which helps better resolve the conflicts between semantic and instance segmentation. Additionally, it handles the challenge caused by the varying number of instances and permits back propagation to the bottom modules in an end-to-end manner. Extensive experimental results on Cityscapes, COCO and our internal dataset demonstrate that our UPSNet achieves stateof-the-art performance with much faster inference. Code has been made available at: https://github.com/ uber-research/UPSNet. * Equal contribution.† This work was done when Hengshuang Zhao was an intern at Uber ATG.

In contrast to fully supervised methods using pixel-wise mask labels, box-supervised instance segmentation takes advantage of simple box annotations, which has recently attracted increasing research attention. This paper presents a novel single-shot instance segmentation approach, namely Box2Mask, which integrates the classical level-set evolution model into deep neural network learning to achieve accurate mask prediction with only bounding box supervision. Specifically, both the input image and its deep features are employed to evolve the level-set curves implicitly, and a local consistency module based on a pixel affinity kernel is used to mine the local context and spatial relations. Two types of single-stage frameworks, i.e., CNN-based and transformer-based frameworks, are developed to empower the level-set evolution for box-supervised instance segmentation, and each framework consists of three essential components: instance-aware decoder, box-level matching assignment and level-set evolution. By minimizing the level-set energy function, the mask map of each instance can be iteratively optimized within its bounding box annotation. The experimental results on five challenging testbeds, covering general scenes, remote sensing, medical and scene text images, demonstrate the outstanding performance of our proposed Box2Mask approach for box-supervised instance segmentation. In particular, with the Swin-Transformer large backbone, our Box2Mask obtains 42.4% mask AP on COCO, which is on par with the recently developed fully mask-supervised methods. The code is available at: https://github.com/LiWentomng/boxlevelset.

本文介绍了端到端的实例分段框架，称为SOIT，该段具有实例感知变压器的段对象。灵感来自Detr〜\ Cite {carion2020end}，我们的方法视图实例分段为直接设置预测问题，有效地消除了对ROI裁剪，一对多标签分配等许多手工制作组件的需求，以及非最大抑制（ nms）。在SOIT中，通过在全局图像上下文下直接地将多个查询直接理解语义类别，边界框位置和像素 - WISE掩码的一组对象嵌入。类和边界盒可以通过固定长度的向量轻松嵌入。尤其是由一组参数嵌入像素方面的掩模以构建轻量级实例感知变压器。之后，实例感知变压器产生全分辨率掩码，而不涉及基于ROI的任何操作。总的来说，SOIT介绍了一个简单的单级实例分段框架，它是无乐和NMS的。 MS Coco DataSet上的实验结果表明，优于最先进的实例分割显着的优势。此外，在统一查询嵌入中的多个任务的联合学习还可以大大提高检测性能。代码可用于\ url {https://github.com/yuxiaodonghri/soit}。

大多数最先进的实例级人类解析模型都采用了两阶段的基于锚的探测器，因此无法避免启发式锚盒设计和像素级别缺乏分析。为了解决这两个问题，我们设计了一个实例级人类解析网络，该网络在像素级别上无锚固且可解决。它由两个简单的子网络组成：一个用于边界框预测的无锚检测头和一个用于人体分割的边缘引导解析头。无锚探测器的头继承了像素样的优点，并有效地避免了对象检测应用中证明的超参数的敏感性。通过引入部分感知的边界线索，边缘引导的解析头能够将相邻的人类部分与彼此区分开，最多可在一个人类实例中，甚至重叠的实例。同时，利用了精炼的头部整合盒子级别的分数和部分分析质量，以提高解析结果的质量。在两个多个人类解析数据集（即CIHP和LV-MHP-V2.0）和一个视频实例级人类解析数据集（即VIP）上进行实验，表明我们的方法实现了超过全球级别和实例级别的性能最新的一阶段自上而下的替代方案。

We present a new method for efficient high-quality image segmentation of objects and scenes. By analogizing classical computer graphics methods for efficient rendering with over-and undersampling challenges faced in pixel labeling tasks, we develop a unique perspective of image segmentation as a rendering problem. From this vantage, we present the PointRend (Point-based Rendering) neural network module: a module that performs point-based segmentation predictions at adaptively selected locations based on an iterative subdivision algorithm. PointRend can be flexibly applied to both instance and semantic segmentation tasks by building on top of existing state-ofthe-art models. While many concrete implementations of the general idea are possible, we show that a simple design already achieves excellent results. Qualitatively, PointRend outputs crisp object boundaries in regions that are oversmoothed by previous methods. Quantitatively, PointRend yields significant gains on COCO and Cityscapes, for both instance and semantic segmentation. PointRend's efficiency enables output resolutions that are otherwise impractical in terms of memory or computation compared to existing approaches. Code has been made available at https:// github.com/facebookresearch/detectron2/ tree/master/projects/PointRend.

我们提出了一个令人尴尬的简单点注释方案，以收集弱监督，例如分割。除了边界框外，我们还收集了在每个边界框内均匀采样的一组点的二进制标签。我们表明，为完整的掩模监督开发的现有实例细分模型可以通过我们的方案收集基于点的监督而无缝培训。值得注意的是，接受了可可，Pascal VOC，CityScapes和LVI的面具R-CNN，每个物体只有10个带注释的随机点可实现94％ - 占其完全监督的性能的98％，为弱化的实例细分定下了强大的基线。新点注释方案的速度比注释完整的对象掩码快5倍，使高质量实例分割在实践中更容易访问。受基于点的注释形式的启发，我们提出了对Pointrend实例分割模块的修改。对于每个对象，称为隐式pointrend的新体系结构生成一个函数的参数，该函数可以使最终的点级掩码预测。隐式Pointrend更加简单，并使用单点级掩蔽丢失。我们的实验表明，新模块更适合基于点的监督。