The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.

Exploring dense matching between the current frame and past frames for long-range context modeling, memory-based methods have demonstrated impressive results in video object segmentation (VOS) recently. Nevertheless, due to the lack of instance understanding ability, the above approaches are oftentimes brittle to large appearance variations or viewpoint changes resulted from the movement of objects and cameras. In this paper, we argue that instance understanding matters in VOS, and integrating it with memory-based matching can enjoy the synergy, which is intuitively sensible from the definition of VOS task, \ie, identifying and segmenting object instances within the video. Towards this goal, we present a two-branch network for VOS, where the query-based instance segmentation (IS) branch delves into the instance details of the current frame and the VOS branch performs spatial-temporal matching with the memory bank. We employ the well-learned object queries from IS branch to inject instance-specific information into the query key, with which the instance-augmented matching is further performed. In addition, we introduce a multi-path fusion block to effectively combine the memory readout with multi-scale features from the instance segmentation decoder, which incorporates high-resolution instance-aware features to produce final segmentation results. Our method achieves state-of-the-art performance on DAVIS 2016/2017 val (92.6% and 87.1%), DAVIS 2017 test-dev (82.8%), and YouTube-VOS 2018/2019 val (86.3% and 86.3%), outperforming alternative methods by clear margins.

We introduce the MAsked Generative VIdeo Transformer, MAGVIT, to tackle various video synthesis tasks with a single model. We introduce a 3D tokenizer to quantize a video into spatial-temporal visual tokens and propose an embedding method for masked video token modeling to facilitate multi-task learning. We conduct extensive experiments to demonstrate the quality, efficiency, and flexibility of MAGVIT. Our experiments show that (i) MAGVIT performs favorably against state-of-the-art approaches and establishes the best-published FVD on three video generation benchmarks, including the challenging Kinetics-600. (ii) MAGVIT outperforms existing methods in inference time by two orders of magnitude against diffusion models and by 60x against autoregressive models. (iii) A single MAGVIT model supports ten diverse generation tasks and generalizes across videos from different visual domains. The source code and trained models will be released to the public at https://magvit.cs.cmu.edu.

Benefiting from masked visual modeling, self-supervised video representation learning has achieved remarkable progress. However, existing methods focus on learning representations from scratch through reconstructing low-level features like raw pixel RGB values. In this paper, we propose masked video distillation (MVD), a simple yet effective two-stage masked feature modeling framework for video representation learning: firstly we pretrain an image (or video) model by recovering low-level features of masked patches, then we use the resulting features as targets for masked feature modeling. For the choice of teacher models, we observe that students taught by video teachers perform better on temporally-heavy video tasks, while image teachers transfer stronger spatial representations for spatially-heavy video tasks. Visualization analysis also indicates different teachers produce different learned patterns for students. Motivated by this observation, to leverage the advantage of different teachers, we design a spatial-temporal co-teaching method for MVD. Specifically, we distill student models from both video teachers and image teachers by masked feature modeling. Extensive experimental results demonstrate that video transformers pretrained with spatial-temporal co-teaching outperform models distilled with a single teacher on a multitude of video datasets. Our MVD with vanilla ViT achieves state-of-the-art performance compared with previous supervised or self-supervised methods on several challenging video downstream tasks. For example, with the ViT-Large model, our MVD achieves 86.4% and 75.9% Top-1 accuracy on Kinetics-400 and Something-Something-v2, outperforming VideoMAE by 1.2% and 1.6% respectively. Code will be available at \url{https://github.com/ruiwang2021/mvd}.

Existing object detection methods are bounded in a fixed-set vocabulary by costly labeled data. When dealing with novel categories, the model has to be retrained with more bounding box annotations. Natural language supervision is an attractive alternative for its annotation-free attributes and broader object concepts. However, learning open-vocabulary object detection from language is challenging since image-text pairs do not contain fine-grained object-language alignments. Previous solutions rely on either expensive grounding annotations or distilling classification-oriented vision models. In this paper, we propose a novel open-vocabulary object detection framework directly learning from image-text pair data. We formulate object-language alignment as a set matching problem between a set of image region features and a set of word embeddings. It enables us to train an open-vocabulary object detector on image-text pairs in a much simple and effective way. Extensive experiments on two benchmark datasets, COCO and LVIS, demonstrate our superior performance over the competing approaches on novel categories, e.g. achieving 32.0% mAP on COCO and 21.7% mask mAP on LVIS. Code is available at: https://github.com/clin1223/VLDet.

In this report, we present a fast and accurate object detection method dubbed DAMO-YOLO, which achieves higher performance than the state-of-the-art YOLO series. DAMO-YOLO is extended from YOLO with some new technologies, including Neural Architecture Search (NAS), efficient Reparameterized Generalized-FPN (RepGFPN), a lightweight head with AlignedOTA label assignment, and distillation enhancement. In particular, we use MAE-NAS, a method guided by the principle of maximum entropy, to search our detection backbone under the constraints of low latency and high performance, producing ResNet-like / CSP-like structures with spatial pyramid pooling and focus modules. In the design of necks and heads, we follow the rule of "large neck, small head". We import Generalized-FPN with accelerated queen-fusion to build the detector neck and upgrade its CSPNet with efficient layer aggregation networks (ELAN) and reparameterization. Then we investigate how detector head size affects detection performance and find that a heavy neck with only one task projection layer would yield better results. In addition, AlignedOTA is proposed to solve the misalignment problem in label assignment. And a distillation schema is introduced to improve performance to a higher level. Based on these new techs, we build a suite of models at various scales to meet the needs of different scenarios, i.e., DAMO-YOLO-Tiny/Small/Medium. They can achieve 43.0/46.8/50.0 mAPs on COCO with the latency of 2.78/3.83/5.62 ms on T4 GPUs respectively. The code is available at https://github.com/tinyvision/damo-yolo.

在本文中，我们从经验上研究了如何充分利用低分辨率框架以进行有效的视频识别。现有方法主要集中于开发紧凑的网络或减轻视频输入的时间冗余以提高效率，而压缩框架分辨率很少被认为是有希望的解决方案。一个主要问题是低分辨率帧的识别准确性不佳。因此，我们首先分析低分辨率帧上性能降解的根本原因。我们的主要发现是，降级的主要原因不是在下采样过程中的信息丢失，而是网络体系结构和输入量表之间的不匹配。通过知识蒸馏（KD）的成功，我们建议通过跨分辨率KD（RESKD）弥合网络和输入大小之间的差距。我们的工作表明，RESKD是一种简单但有效的方法，可以提高低分辨率帧的识别精度。没有铃铛和哨子，RESKD在四个大规模基准数据集（即ActivityNet，FCVID，Mini-Kinetics，sopeings soseings ossings v2）上，就效率和准确性上的所有竞争方法都大大超过了所有竞争方法。此外，我们广泛地展示了其对最先进的体系结构（即3D-CNN和视频变压器）的有效性，以及对超低分辨率帧的可扩展性。结果表明，RESKD可以作为最先进视频识别的一般推理加速方法。我们的代码将在https://github.com/cvmi-lab/reskd上找到。

近年来，已取得了巨大进展，以通过半监督学习（SSL）来纳入未标记的数据来克服效率低下的监督问题。大多数最先进的模型是基于对未标记的数据追求一致的模型预测的想法，该模型被称为输入噪声，这称为一致性正则化。尽管如此，对其成功的原因缺乏理论上的见解。为了弥合理论和实际结果之间的差距，我们在本文中提出了SSL的最坏情况一致性正则化技术。具体而言，我们首先提出了针对SSL的概括，该概括由分别在标记和未标记的训练数据上观察到的经验损失项组成。在这种界限的激励下，我们得出了一个SSL目标，该目标可最大程度地减少原始未标记的样本与其多重增强变体之间最大的不一致性。然后，我们提供了一种简单但有效的算法来解决提出的最小问题，从理论上证明它会收敛到固定点。五个流行基准数据集的实验验证了我们提出的方法的有效性。

在许多分类模型中，数据被离散化以更好地估计其分布。现有的离散方法通常是针对最大化离散数据的判别能力的，同时忽略了分类中数据离散化的主要目标是改善概括性能。结果，数据往往会超出许多小型垃圾箱，因为数据没有离散化保留了最大判别信息。因此，我们提出了一个最大依赖性最差（MDMD）标准，该标准可最大程度地提高离散数据的判别信息和概括能力。更具体地说，最大依赖性标准可最大化离散数据和分类变量之间的统计依赖性，而最小差异标准则明确最大程度地减少了给定离散方案的训练数据与验证数据之间的JS差异。拟议的MDMD标准在技术上很有吸引力，但是很难可靠地估计属性的高阶联合分布和分类变量。因此，我们进一步提出了一个更实用的解决方案，最大值 - 差异 - 差异（MRMD）离散方案，其中每个属性通过同时最大化判别信息和离散数据的概括能力分别离散化。将提出的MRMD与45个机器学习基准数据集的Naive Bayes分类框架下的最新离散算法进行了比较。它大大优于大多数数据集上所有比较的方法。

尽管视觉问题答案取得了长足的进步（VQA），但当前的VQA模型严重依赖问题类型及其相应的频繁答案（即语言先验）之间的表面相关性来做出预测，而无需真正理解输入。在这项工作中，我们用相同的问题类型定义了培训实例，但与\ textit {表面上相似的实例}定义了不同的答案，并将语言先验归因于VQA模型在此类情况下的混淆。为了解决这个问题，我们提出了一个新颖的培训框架，该培训框架明确鼓励VQA模型区分表面上相似的实例。具体而言，对于每个培训实例，我们首先构建一个包含其表面上相似的对应物的集合。然后，我们利用所提出的区分模块增加了答案空间中实例及其对应物之间的距离。这样，VQA模型被迫进一步关注问题类型的输入的其他部分，这有助于克服语言先验。实验结果表明，我们的方法在VQA-CP V2上实现了最新性能。代码可在\ href {https://github.com/wyk-nku/distinguishing-vqa.git} {sickithing-vqa}中获得。