最近，通过引入大规模的数据集和强大的变压器网络，视频预培训表明尤其是检索的巨大成功。然而，现有的视频语言变压器模型没有明确细粒度的语义对齐。在这项工作中，我们呈现了对象感知的变换器，以对象为中心的方法，该对象方法扩展了视频语言变压器来合并对象表示。关键的想法是利用边界框和对象标签来指导培训过程。我们在四个广泛使用的基准测试中评估了我们的三个标准子任务的模型。我们还提供了深入的分析和详细消融关于所提出的方法。我们在考虑的所有任务和数据集中表现出清晰的性能，展示将对象表示的模型中的型号集成到视频架构中。代码将以\ URL {https://github.com/fingerrec/oa -transformer}释放。

Vision-Language Pre-Training (VLP) has shown promising capabilities to align image and text pairs, facilitating a broad variety of cross-modal learning tasks. However, we observe that VLP models often lack the visual grounding/localization capability which is critical for many downstream tasks such as visual reasoning. In this work, we propose a novel Position-guided Text Prompt (PTP) paradigm to enhance the visual grounding ability of cross-modal models trained with VLP. Specifically, in the VLP phase, PTP divides the image into $N\times N$ blocks, and identifies the objects in each block through the widely used object detector in VLP. It then reformulates the visual grounding task into a fill-in-the-blank problem given a PTP by encouraging the model to predict the objects in the given blocks or regress the blocks of a given object, e.g. filling `P" or ``O" in aPTP ``The block P has a O". This mechanism improves the visual grounding capability of VLP models and thus helps them better handle various downstream tasks. By introducing PTP into several state-of-the-art VLP frameworks, we observe consistently significant improvements across representative cross-modal learning model architectures and several benchmarks, e.g. zero-shot Flickr30K Retrieval (+4.8 in average recall@1) for ViLT \cite{vilt} baseline, and COCO Captioning (+5.3 in CIDEr) for SOTA BLIP \cite{blip} baseline. Moreover, PTP achieves comparable results with object-detector based methods, and much faster inference speed since PTP discards its object detector for inference while the later cannot. Our code and pre-trained weight will be released at \url{https://github.com/sail-sg/ptp}.

视频文本预培训旨在通过对齐视觉和文本信息之间的语义来对齐大规模视频文本对学习可转换表示。最先进的方法以端到端的方式从原始像素提取视觉特征。然而，这些方法直接在帧级运行，从而忽略了视频中对象的时空结构，这在文本描述中具有名词的强烈协同作用。在这项工作中，我们提出了一个简单而有效的模块，即用于视频文本表示学习，即RegionLearner，它可以考虑在大规模视频文本对预训练中的对象结构。给定视频，我们的模块（1）首先将可视特征量化为语义集群，然后（2）生成被动掩码并使用它们聚合属于同一语义区域的功能，最后（3）模拟不同聚合区域之间的交互。与使用现成的对象探测器相比，我们所提出的模块不需要明确的监督，并且更加计算效率。我们在公共WebVID2M和CC3M数据集上预先列车。对四个下游视频文本检索基准测试的广泛评估清楚地展示了我们的地区learner的有效性。代码将在https://github.com/ruiyan1995/region_learner上获得。

Most existing text-video retrieval methods focus on cross-modal matching between the visual content of offline videos and textual query sentences. However, in real scenarios, online videos are frequently accompanied by relevant text information such as titles, tags, and even subtitles, which can be utilized to match textual queries. This inspires us to generate associated captions from offline videos to help with existing text-video retrieval methods. To do so, we propose to use the zero-shot video captioner with knowledge of pre-trained web-scale models (e.g., CLIP and GPT-2) to generate captions for offline videos without any training. Given the captions, one question naturally arises: what can auxiliary captions do for text-video retrieval? In this paper, we present a novel framework Cap4Video, which makes use of captions from three aspects: i) Input data: The video and captions can form new video-caption pairs as data augmentation for training. ii) Feature interaction: We perform feature interaction between video and caption to yield enhanced video representations. iii) Output score: The Query-Caption matching branch can be complementary to the original Query-Video matching branch for text-video retrieval. We conduct thorough ablation studies to demonstrate the effectiveness of our method. Without any post-processing, our Cap4Video achieves state-of-the-art performance on MSR-VTT (51.4%), VATEX (66.6%), MSVD (51.8%), and DiDeMo (52.0%).

视频语言（VIDL）建模的巨大挑战在于从图像/视频理解模型和下游Vidl数据中提取的固定视频表示之间的断开。最近的研究试图通过端到端培训来减轻这种断开连接。为了使其进行计算可行，先前的作品倾向于“想象”视频输入，即，将一些稀疏的采样帧馈送到2D CNN中，然后是简单的均值汇集或连接以获得整体视频表示。虽然实现了有希望的结果，但这种简单的方法可能会失去对于执行下游VIDL任务至关重要的时间信息。在这项工作中，我们呈现紫罗兰色，全新的视频语言变压器，采用视频变压器，明确地模拟视频输入的时间动态。此外，与以前的研究不同，发现视频输入上的预训练任务（例如，屏蔽帧建模）不是非常有效的，我们设计了一个新的预训练任务，屏蔽了视觉令牌建模（MVM），以获得更好的视频建模。具体地，原始视频帧修补程序将“令牌化”转换为离散的视觉令牌，目标是基于蒙面的贴片恢复原始的视觉令牌。综合分析展示了通过视频变压器和MVM显式时间建模的有效性。因此，紫罗兰在5个视频问题的回答任务和4个文本到视频检索任务中实现了新的最先进的性能。

We present a simple yet effective end-to-end Video-language Pre-training (VidLP) framework, Masked Contrastive Video-language Pretraining (MAC), for video-text retrieval tasks. Our MAC aims to reduce video representation's spatial and temporal redundancy in the VidLP model by a mask sampling mechanism to improve pre-training efficiency. Comparing conventional temporal sparse sampling, we propose to randomly mask a high ratio of spatial regions and only feed visible regions into the encoder as sparse spatial sampling. Similarly, we adopt the mask sampling technique for text inputs for consistency. Instead of blindly applying the mask-then-prediction paradigm from MAE, we propose a masked-then-alignment paradigm for efficient video-text alignment. The motivation is that video-text retrieval tasks rely on high-level alignment rather than low-level reconstruction, and multimodal alignment with masked modeling encourages the model to learn a robust and general multimodal representation from incomplete and unstable inputs. Coupling these designs enables efficient end-to-end pre-training: reduce FLOPs (60% off), accelerate pre-training (by 3x), and improve performance. Our MAC achieves state-of-the-art results on various video-text retrieval datasets, including MSR-VTT, DiDeMo, and ActivityNet. Our approach is omnivorous to input modalities. With minimal modifications, we achieve competitive results on image-text retrieval tasks.

本文介绍了Omnivl，这是一种新的基础模型，旨在使用一种通用体系结构来支持图像语言和视频语言任务。它为图像和视频输入采用了统一的基于变压器的视觉编码器，因此可以执行联合图像语言和视频语言预处理。我们首次证明了这样的范式受益于图像和视频任务，而不是传统的单向传输（例如，使用图像语言来帮助视频语言）。为此，我们提出了对图像语言和视频语言的脱钩关节预处理，以有效地将视觉模型分解为空间和时间维度，并在图像和视频任务上获得性能提升。此外，我们引入了一种新颖的统一视觉对比度（UNIVLC）损失，以利用图像文本，视频文本，图像标签（例如，图像分类），视频标签（例如，视频动作识别）在一起受到监督和吵闹的监督预处理数据都尽可能多地利用。无需额外的任务适配器，Omnivl可以同时支持仅视觉任务（例如，图像分类，视频操作识别），跨模式对齐任务（例如，图像/视频 - 文本检索）和多模式理解和生成任务（例如，图像/视频问答，字幕）。我们在各种下游任务上评估Omnivl，并以相似的模型大小和数据量表获得最新的或竞争结果。

Video-language pre-training has advanced the performance of various downstream video-language tasks. However, most previous methods directly inherit or adapt typical image-language pre-training paradigms to video-language pre-training, thus not fully exploiting the unique characteristic of video, i.e., temporal. In this paper, we propose a Hierarchical Temporal-Aware video-language pre-training framework, HiTeA, with two novel pre-training tasks for modeling cross-modal alignment between moments and texts as well as the temporal relations of video-text pairs. Specifically, we propose a cross-modal moment exploration task to explore moments in videos, which results in detailed video moment representation. Besides, the inherent temporal relations are captured by aligning video-text pairs as a whole in different time resolutions with multi-modal temporal relation exploration task. Furthermore, we introduce the shuffling test to evaluate the temporal reliance of datasets and video-language pre-training models. We achieve state-of-the-art results on 15 well-established video-language understanding and generation tasks, especially on temporal-oriented datasets (e.g., SSv2-Template and SSv2-Label) with 8.6% and 11.1% improvement respectively. HiTeA also demonstrates strong generalization ability when directly transferred to downstream tasks in a zero-shot manner. Models and demo will be available on ModelScope.

现有视觉语言预训练（VLP）方法主要依赖于配对的图像文本数据集，这些数据集由大量人类劳动注释，或者从互联网上爬行，然后是精心制作的数据清洁技术。为了减少对良好的图像文本对的依赖，有望直接利用仅大规模的仅文本和仅图像的语料库。本文提出了一种数据增强方法，即跨模式cutmix（CMC），用于在未配对的VLP中进行隐式跨模式对齐学习。具体而言，CMC将自然句子从文本视图转换为多模式视图，在该视图中，句子中的视觉词语单词被带有相似语义的各种图像贴片随机替换。拟议中的CMC有几个吸引人的礼节。首先，它增强了数据多样性，同时保持语义含义完好无损地解决了对齐数据稀缺的问题；其次，通过将跨模式噪声连接到单模式数据上，它指导模型以学习跨模态的令牌级相互作用，以更好地降级。此外，我们提出了一种名为VLMIXER的新的未配对VLP方法，该方法将CMC与对比度学习集成在一起，以将Uni-Mododal和多模式视图汇总在一起，以在不同模式之间进行更好的实例级别对齐。在五个下游任务上进行的广泛实验表明，VLMIXER可以超过以前最先进的未配对VLP方法。

蒙版的视觉建模（MVM）最近已被证明对视觉预训练有效。虽然在视频输入（例如，蒙版框架建模）上进行了类似的重建目标，在视频语言（VIDL）预训练中探索了类似的重建目标，但先前研究中的预提取的视频功能在预训练期间无法通过MVM进行完善，因此无法通过MVM进行完善为下游性能不满意。在这项工作中，我们系统地检查了MVM在VIDL学习的背景下的潜力。具体而言，我们的研究基于完全端到端的视频变压器（Violet），该视频变压器（Violet）减轻了固定视频表示与MVM培训之间的断开连接。总共探索了MVM的八个不同的重建目标，从低级像素值和定向梯度到高级深度图，光流，离散的视觉令牌和潜在的视觉特征。我们进行全面的实验，并就导致有效MVM培训的因素提供见解。从经验上讲，我们展示了通过MVM目标预先训练的紫罗兰色，可以在13个VIDL基准测试中取得显着改进，从视频问题回答，视频字幕到文本到视频检索等等。

The foundation models have recently shown excellent performance on a variety of downstream tasks in computer vision. However, most existing vision foundation models simply focus on image-level pretraining and adpation, which are limited for dynamic and complex video-level understanding tasks. To fill the gap, we present general video foundation models, InternVideo, by taking advantage of both generative and discriminative self-supervised video learning. Specifically, InternVideo efficiently explores masked video modeling and video-language contrastive learning as the pretraining objectives, and selectively coordinates video representations of these two complementary frameworks in a learnable manner to boost various video applications. Without bells and whistles, InternVideo achieves state-of-the-art performance on 39 video datasets from extensive tasks including video action recognition/detection, video-language alignment, and open-world video applications. Especially, our methods can obtain 91.1% and 77.2% top-1 accuracy on the challenging Kinetics-400 and Something-Something V2 benchmarks, respectively. All of these results effectively show the generality of our InternVideo for video understanding. The code will be released at https://github.com/OpenGVLab/InternVideo .

探索大规模预处理的基础模型对计算机视觉具有重大兴趣，因为这些模型可以快速转移到许多下游任务中。本文介绍了对比字幕（COCA），这是一种极简主义的设计，旨在为图像文本编码器编码器基础模型预算与对比度损失和字幕损失，从而从剪辑和诸如simvlm之类的生成方法之类的对比方法中包含模型能力。与所有解码器层都参与编码器输出的标准编码器 - 模块变压器相反，可口可乐省略了解码器层的上半部分的交叉注意，以编码单峰文本表示，并串联到剩余的解码器层，这些解码器与图像编码器相交的解码器层多模式图像文本表示。除了对多模态解码器输出的字幕损失外，我们还应用了单峰图像和文本嵌入之间的对比损失，该输出可以预测文本令牌自动加压。通过共享相同的计算图，可以用最小的开销有效地计算两个培训目标。可口可乐是端到端和从头开始的网络尺度alt-text数据和带注释的图像，通过将所有标签视为文本，无缝地统一自然语言监督以进行表示。从经验上讲，可口可乐通过零拍传输或在广泛的下游任务上进行零摄像转移或最少的特定任务适应，跨越视觉识别（Imagenet，Kinetics-400/600/700，瞬间， ），交叉模式检索（MSCOCO，FLICKR30K，MSR-VTT），多模式理解（VQA，SNLI-VE，NLVR2）和图像字幕（MSCOCO，NOCAPS）。值得注意的是，在Imagenet分类方面，COCA获得了86.3％的TOP-1准确性，带有冷冻编码器和学习的分类头90.6％，以及带有填充编码器的Imagenet上的新最先进的91.0％Top-1 Top-1精度。

以前的视觉语言预训练模型主要构建具有令牌和对象（像素）的多模式输入，然后在它们之间执行交叉模式相互作用。我们认为，只有令牌和对象的输入限制了诸如短语到区域接地之类的高级语义对齐。同时，多层次对齐本质上是一致的，并且能够协同促进表示形式学习。因此，在本文中，我们建议学习视觉预训练（MVPTR）的多级语义一致性。在MVPTR中，我们遵循两种方式的嵌套结构，以引入概念为高级语义。为了简化从多模式多级输入的学习，我们的框架分为两个阶段，第一阶段着重于模式内多级表示学习，第二阶段通过粗粒和细粒度跨模态强化了跨模式的交互语义对齐任务。除了常用的图像文本匹配和掩盖语言模型任务外，我们还引入了第一阶段蒙版概念恢复任务以增强概念表示学习，第二阶段的另外两个任务在第二阶段中，以明确鼓励跨跨层次的多层次对准方式。我们的代码可在https://github.com/junction4nako/mvp_pytorch上找到。

预先培训用于学习可转让的视频文本表示的模型，以近年来引起了很多关注。以前的主导作品主要采用两个独立的编码器来有效检索，但忽略视频和文本之间的本地关联。另一种研究使用联合编码器与文本交互视频，但是由于每个文本视频对需要馈送到模型中的低效率。在这项工作中，我们能够通过新颖的借口任务进行微粒视频文本交互，以便通过新颖的借口任务进行检索，称为多项选择题（MCQ），其中参数模块BridgeFormer培训以接受由此构建的“问题”。文本功能通过诉诸视频功能。具体来说，我们利用了文本的丰富语义（即，名词和动词）来构建问题，可以培训视频编码器以捕获更多区域内容和时间动态。以问题和答案的形式，可以正确建立本地视频文本功能之间的语义关联。 BridgeFormer能够删除下游检索，只有两个编码器渲染高效且灵活的模型。我们的方法在具有不同实验设置（即零拍摄和微调）的五个数据集中，在五个数据集中优于最先进的方法，包括不同的实验设置（即零拍摄和微调），包括HOWTO100M（一百万个视频）。我们进一步开展零射击动作识别，可以作为视频到文本检索，我们的方法也显着超越了其对应物。作为额外的好处，我们的方法在单模下游任务中实现了竞争力，在单模下游任务上具有更短的预训练视频，例如，使用线性评估的动作识别。

Vision-Language Transformers can be learned without human labels (e.g. class labels, bounding boxes, etc). Existing work, whether explicitly utilizing bounding boxes or patches, assumes that the visual backbone must first be trained on ImageNet class prediction before being integrated into a multimodal linguistic pipeline. We show that this is not necessary and introduce a new model Vision-Language from Captions (VLC) built on top of Masked Auto-Encoders that does not require this supervision. In fact, in a head-to-head comparison between ViLT, the current state-of-the-art patch-based vision-language transformer which is pretrained with supervised object classification, and our model, VLC, we find that our approach 1. outperforms ViLT on standard benchmarks, 2. provides more interpretable and intuitive patch visualizations, and 3. is competitive with many larger models that utilize ROIs trained on annotated bounding-boxes.

预先训练的图像文本模型（如剪辑）已经证明了从大规模的Web收集的图像文本数据中学到的视觉表示的强大力量。鉴于学习良好的视觉特征，一些现有的作品将图像表示转移到视频域并取得良好的结果。但是，如何利用图像语言预训练的模型（例如，剪辑）进行视频培训（后培训）仍在探索。在本文中，我们研究了两个问题：1）阻碍后期剪辑的因素是什么因素，以进一步提高视频语言任务的性能？ 2）如何减轻这些因素的影响？通过一系列比较实验和分析，我们发现语言源之间的数据量表和域间隙具有很大的影响。由这些动机，我们提出了一种配备了视频代理机制的Omnisource跨模式学习方法，即剪辑，即剪辑VIP。广泛的结果表明，我们的方法可以提高视频检索的剪辑的性能。我们的模型还可以在包括MSR-VTT，DIDEMO，LSMDC和ActivityNet在内的各种数据集上实现SOTA结果。我们在https://github.com/microsoft/xpretrain/tree/main/main/main/clip-vip上发布了代码和预训练的剪辑模型。

Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatenate image region features and text features as input to the model to be pre-trained and use selfattention to learn image-text semantic alignments in a brute force manner, in this paper, we propose a new learning method Oscar 1 , which uses object tags detected in images as anchor points to significantly ease the learning of alignments. Our method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. We pre-train an Oscar model on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks. 2

视觉检索中的大多数现有方法是通过比较其全局特征向量的两种方式，该矢量错过了足够的信息并缺乏可解释性，检测图像或视频中的对象，并将文本与依赖复杂的模型设计或建模的精细元素对齐通过较低效率遭受视觉和文本令牌的交叉注意相互作用。为了解决这些局限性，最近的一些作品简单地汇总了代币的相似性以实现细粒度的对齐方式，但它们缺乏直观的解释，并且忽略了令牌级特征和具有高级语义的全球表示之间的关系。在这项工作中，我们重新考虑细粒度的跨模式对准，并为其设计一种新的模型不合命固式配方。我们还揭开了最近的流行作品的神秘面纱，并将其纳入我们的计划。此外，受最佳运输理论的启发，我们引入了\ emph {tokenflow}，这是对拟议方案的实例化。通过仅修改相似性函数，我们方法的性能与主要视频文本检索基准上具有重型模型设计的SOTA算法相当。可视化进一步表明\ emph {tokenflow}成功利用细粒度的信息并获得了更好的解释性。

This work explores an efficient approach to establish a foundational video-text model for tasks including open-vocabulary video classification, text-to-video retrieval, video captioning and video question-answering. We present VideoCoCa that reuses a pretrained image-text contrastive captioner (CoCa) model and adapt it to video-text tasks with minimal extra training. While previous works adapt image-text models with various cross-frame fusion modules (for example, cross-frame attention layer or perceiver resampler) and finetune the modified architecture on video-text data, we surprisingly find that the generative attentional pooling and contrastive attentional pooling layers in the image-text CoCa design are instantly adaptable to ``flattened frame embeddings'', yielding a strong zero-shot transfer baseline for many video-text tasks. Specifically, the frozen image encoder of a pretrained image-text CoCa takes each video frame as inputs and generates \(N\) token embeddings per frame for totally \(T\) video frames. We flatten \(N \times T\) token embeddings as a long sequence of frozen video representation and apply CoCa's generative attentional pooling and contrastive attentional pooling on top. All model weights including pooling layers are directly loaded from an image-text CoCa pretrained model. Without any video or video-text data, VideoCoCa's zero-shot transfer baseline already achieves state-of-the-art results on zero-shot video classification on Kinetics 400/600/700, UCF101, HMDB51, and Charades, as well as zero-shot text-to-video retrieval on MSR-VTT and ActivityNet Captions. We also explore lightweight finetuning on top of VideoCoCa, and achieve strong results on video question-answering (iVQA, MSRVTT-QA, MSVD-QA) and video captioning (MSR-VTT, ActivityNet, Youcook2). Our approach establishes a simple and effective video-text baseline for future research.

远见和语言预测已成为解决多模式下游任务的普遍方法。当前的趋势是朝着更大的模型和预处理数据集迈进。从长远来看，这一计算头急促似乎是不合理的，而是朝着可持续的解决方案迈进，事实上，排除了资源有限的学术实验室。在这项工作中，我们提出了一个称为VICHA的新框架，该框架有效利用输入数据以通过以下方式提高学习，以： ，（c）利用图像级注释，称为视觉概念，使用现有基础模型（例如剪辑）获得，以提高图像编码器的性能。尽管对数据的预估计少了四倍，但我们的VICHA策略在下游任务（例如图像文本检索，VQA，视觉推理，视觉上和视觉接地）上的其他方法优于其他方法。该代码将在此处公开提供：https：//github.com/mshukor/vicha