The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.

预先训练的图像文本模型（如剪辑）已经证明了从大规模的Web收集的图像文本数据中学到的视觉表示的强大力量。鉴于学习良好的视觉特征，一些现有的作品将图像表示转移到视频域并取得良好的结果。但是，如何利用图像语言预训练的模型（例如，剪辑）进行视频培训（后培训）仍在探索。在本文中，我们研究了两个问题：1）阻碍后期剪辑的因素是什么因素，以进一步提高视频语言任务的性能？ 2）如何减轻这些因素的影响？通过一系列比较实验和分析，我们发现语言源之间的数据量表和域间隙具有很大的影响。由这些动机，我们提出了一种配备了视频代理机制的Omnisource跨模式学习方法，即剪辑，即剪辑VIP。广泛的结果表明，我们的方法可以提高视频检索的剪辑的性能。我们的模型还可以在包括MSR-VTT，DIDEMO，LSMDC和ActivityNet在内的各种数据集上实现SOTA结果。我们在https://github.com/microsoft/xpretrain/tree/main/main/main/clip-vip上发布了代码和预训练的剪辑模型。

与传统方法相比，学到的图像压缩已在PSNR和MS-SSIM中取得了非凡的速率延伸性能。但是，它遭受了密集的计算，这对于现实世界的应用是无法忍受的，目前导致其工业应用有限。在本文中，我们将神经体系结构搜索（NAS）介绍到具有较低延迟的更有效网络，并利用量化以加速推理过程。同时，已经为提高效率而做出了工程努力。使用PSNR和MS-SSIM的混合损失以更好的视觉质量进行了优化，我们获得的MSSIM比JPEG，JPEG XL和AVIF在所有比特率上都高得多，而JPEG XL和AVIF之间的PSNR则获得了PSNR。与JPEG-Turbo相比，我们的LIC的软件实施实现了可比较甚至更快的推理速度，而多次比JPEG XL和AVIF快。此外，我们的LIC实施达到了145 fps的惊人吞吐量，用于编码为208 fps，用于在Tesla T4 GPU上解码1080p图像。在CPU上，我们实施的延迟与JPEG XL相当。

我们研究了联合视频和语言（VL）预培训，以实现跨模型学习和益处丰富的下游VL任务。现有的作品要么提取低质量的视频特征或学习有限的文本嵌入，但忽略了高分辨率视频和多样化的语义可以显着提高跨模型学习。在本文中，我们提出了一种新的高分辨率和多样化的视频 - 语言预训练模型（HD-VILA），用于许多可视任务。特别是，我们收集具有两个不同属性的大型数据集：1）第一个高分辨率数据集包括371.5k小时的720p视频，2）最多样化的数据集涵盖15个流行的YouTube类别。为了启用VL预培训，我们通过学习丰富的时空特征的混合变压器联合优化HD-VILA模型，以及多峰变压器，用于强制学习视频功能与多样化文本的交互。我们的预训练模式实现了新的最先进的导致10 VL了解任务和2个新颖的文本到视觉生成任务。例如，我们以零拍摄MSR-VTT文本到视频检索任务的相对增加38.5％R @ 1的相对增长，高分辨率数据集LSMDC为53.6％。学习的VL嵌入也有效地在文本到视觉操纵和超分辨率任务中产生视觉上令人愉悦和语义相关结果。

Vision-Language预培训（VLP）旨在从图像文本对中学习多模态表示，并以微调方式为下游视觉语言任务服务。主导VLP模型采用CNN变压器架构，该架构将图像与CNN嵌入，然后使用变压器对齐图像和文本。视觉内容之间的视觉关系在图像理解中发挥着重要作用，并且是模态对齐学习的基本。然而，由于局部接受领域在建模远程依赖性方面的弱点，CNNS具有局限性。因此，在相同的变压器网络中封装了学习视觉关系和模态对齐的两个目标。这种设计可能通过忽略每个目标的专用特性来限制变压器中的模态对准学习。为了解决这个问题，我们提出了一个完全变压器视觉嵌入VLP，以更好地学习视觉关系，进一步促进模态对齐。具体地，我们提出了一个名为Domank跨性流量的度量（IMF），以测量视觉和语言模态之间的交互（即，互别互别）。我们还设计了一种名为Massed Featuber Resollion（MFR）的新型屏蔽优化机制，在变压器中进一步推广了模范间学习。据我们所知，这是第一项探索VLP中可视化特征学习的变压器的利益的研究。我们在广泛的视觉语言任务中验证了我们的方法，包括图像文本检索，视觉问题应答（VQA），视觉征求和视觉推理。我们的方法不仅优于最先进的VLP性能，而且还显示了对IMF度量的好处。

Few Shot Instance Segmentation (FSIS) requires models to detect and segment novel classes with limited several support examples. In this work, we explore a simple yet unified solution for FSIS as well as its incremental variants, and introduce a new framework named Reference Twice (RefT) to fully explore the relationship between support/query features based on a Transformer-like framework. Our key insights are two folds: Firstly, with the aid of support masks, we can generate dynamic class centers more appropriately to re-weight query features. Secondly, we find that support object queries have already encoded key factors after base training. In this way, the query features can be enhanced twice from two aspects, i.e., feature-level and instance-level. In particular, we firstly design a mask-based dynamic weighting module to enhance support features and then propose to link object queries for better calibration via cross-attention. After the above steps, the novel classes can be improved significantly over our strong baseline. Additionally, our new framework can be easily extended to incremental FSIS with minor modification. When benchmarking results on the COCO dataset for FSIS, gFSIS, and iFSIS settings, our method achieves a competitive performance compared to existing approaches across different shots, e.g., we boost nAP by noticeable +8.2/+9.4 over the current state-of-the-art FSIS method for 10/30-shot. We further demonstrate the superiority of our approach on Few Shot Object Detection. Code and model will be available.

This paper focuses on designing efficient models with low parameters and FLOPs for dense predictions. Even though CNN-based lightweight methods have achieved stunning results after years of research, trading-off model accuracy and constrained resources still need further improvements. This work rethinks the essential unity of efficient Inverted Residual Block in MobileNetv2 and effective Transformer in ViT, inductively abstracting a general concept of Meta-Mobile Block, and we argue that the specific instantiation is very important to model performance though sharing the same framework. Motivated by this phenomenon, we deduce a simple yet efficient modern \textbf{I}nverted \textbf{R}esidual \textbf{M}obile \textbf{B}lock (iRMB) for mobile applications, which absorbs CNN-like efficiency to model short-distance dependency and Transformer-like dynamic modeling capability to learn long-distance interactions. Furthermore, we design a ResNet-like 4-phase \textbf{E}fficient \textbf{MO}del (EMO) based only on a series of iRMBs for dense applications. Massive experiments on ImageNet-1K, COCO2017, and ADE20K benchmarks demonstrate the superiority of our EMO over state-of-the-art methods, \eg, our EMO-1M/2M/5M achieve 71.5, 75.1, and 78.4 Top-1 that surpass \textbf{SoTA} CNN-/Transformer-based models, while trading-off the model accuracy and efficiency well.

Despite significant progress in object categorization, in recent years, a number of important challenges remain; mainly, the ability to learn from limited labeled data and to recognize object classes within large, potentially open, set of labels. Zero-shot learning is one way of addressing these challenges, but it has only been shown to work with limited sized class vocabularies and typically requires separation between supervised and unsupervised classes, allowing former to inform the latter but not vice versa. We propose the notion of vocabulary-informed learning to alleviate the above mentioned challenges and address problems of supervised, zero-shot, generalized zero-shot and open set recognition using a unified framework. Specifically, we propose a weighted maximum margin framework for semantic manifold-based recognition that incorporates distance constraints from (both supervised and unsupervised) vocabulary atoms. Distance constraints ensure that labeled samples are projected closer to their correct prototypes, in the embedding space, than to others. We illustrate that resulting model shows improvements in supervised, zero-shot, generalized zero-shot, and large open set recognition, with up to 310K class vocabulary on Animal with Attributes and ImageNet datasets.

Deploying reliable deep learning techniques in interdisciplinary applications needs learned models to output accurate and ({even more importantly}) explainable predictions. Existing approaches typically explicate network outputs in a post-hoc fashion, under an implicit assumption that faithful explanations come from accurate predictions/classifications. We have an opposite claim that explanations boost (or even determine) classification. That is, end-to-end learning of explanation factors to augment discriminative representation extraction could be a more intuitive strategy to inversely assure fine-grained explainability, e.g., in those neuroimaging and neuroscience studies with high-dimensional data containing noisy, redundant, and task-irrelevant information. In this paper, we propose such an explainable geometric deep network dubbed as NeuroExplainer, with applications to uncover altered infant cortical development patterns associated with preterm birth. Given fundamental cortical attributes as network input, our NeuroExplainer adopts a hierarchical attention-decoding framework to learn fine-grained attentions and respective discriminative representations to accurately recognize preterm infants from term-born infants at term-equivalent age. NeuroExplainer learns the hierarchical attention-decoding modules under subject-level weak supervision coupled with targeted regularizers deduced from domain knowledge regarding brain development. These prior-guided constraints implicitly maximizes the explainability metrics (i.e., fidelity, sparsity, and stability) in network training, driving the learned network to output detailed explanations and accurate classifications. Experimental results on the public dHCP benchmark suggest that NeuroExplainer led to quantitatively reliable explanation results that are qualitatively consistent with representative neuroimaging studies.

Improving the visual quality of the given degraded observation by correcting exposure level is a fundamental task in the computer vision community. Existing works commonly lack adaptability towards unknown scenes because of the data-driven patterns (deep networks) and limited regularization (traditional optimization), and they usually need time-consuming inference. These two points heavily limit their practicability. In this paper, we establish a Practical Exposure Corrector (PEC) that assembles the characteristics of efficiency and performance. To be concrete, we rethink the exposure correction to provide a linear solution with exposure-sensitive compensation. Around generating the compensation, we introduce an exposure adversarial function as the key engine to fully extract valuable information from the observation. By applying the defined function, we construct a segmented shrinkage iterative scheme to generate the desired compensation. Its shrinkage nature supplies powerful support for algorithmic stability and robustness. Extensive experimental evaluations fully reveal the superiority of our proposed PEC. The code is available at https://rsliu.tech/PEC.