Video representation learning has been successful in video-text pre-training for zero-shot transfer, where each sentence is trained to be close to the paired video clips in a common feature space. For long videos, given a paragraph of description where the sentences describe different segments of the video, by matching all sentence-clip pairs, the paragraph and the full video are aligned implicitly. However, such unit-level similarity measure may ignore the global temporal context over a long time span, which inevitably limits the generalization ability. In this paper, we propose a contrastive learning framework TempCLR to compare the full video and the paragraph explicitly. As the video/paragraph is formulated as a sequence of clips/sentences, under the constraint of their temporal order, we use dynamic time warping to compute the minimum cumulative cost over sentence-clip pairs as the sequence-level distance. To explore the temporal dynamics, we break the consistency of temporal order by shuffling the video clips or sentences according to the temporal granularity. In this way, we obtain the representations for clips/sentences, which perceive the temporal information and thus facilitate the sequence alignment. In addition to pre-training on the video and paragraph, our approach can also generalize on the matching between different video instances. We evaluate our approach on video retrieval, action step localization, and few-shot action recognition, and achieve consistent performance gain over all three tasks. Detailed ablation studies are provided to justify the approach design.

From a visual scene containing multiple people, human is able to distinguish each individual given the context descriptions about what happened before, their mental/physical states or intentions, etc. Above ability heavily relies on human-centric commonsense knowledge and reasoning. For example, if asked to identify the "person who needs healing" in an image, we need to first know that they usually have injuries or suffering expressions, then find the corresponding visual clues before finally grounding the person. We present a new commonsense task, Human-centric Commonsense Grounding, that tests the models' ability to ground individuals given the context descriptions about what happened before, and their mental/physical states or intentions. We further create a benchmark, HumanCog, a dataset with 130k grounded commonsensical descriptions annotated on 67k images, covering diverse types of commonsense and visual scenes. We set up a context-object-aware method as a strong baseline that outperforms previous pre-trained and non-pretrained models. Further analysis demonstrates that rich visual commonsense and powerful integration of multi-modal commonsense are essential, which sheds light on future works. Data and code will be available https://github.com/Hxyou/HumanCog.

Visual commonsense understanding requires Vision Language (VL) models to not only understand image and text but also cross-reference in-between to fully integrate and achieve comprehension of the visual scene described. Recently, various approaches have been developed and have achieved high performance on visual commonsense benchmarks. However, it is unclear whether the models really understand the visual scene and underlying commonsense knowledge due to limited evaluation data resources. To provide an in-depth analysis, we present a Multimodal Evaluation (ME) pipeline to automatically generate question-answer pairs to test models' understanding of the visual scene, text, and related knowledge. We then take a step further to show that training with the ME data boosts the model's performance in standard VCR evaluation. Lastly, our in-depth analysis and comparison reveal interesting findings: (1) semantically low-level information can assist the learning of high-level information but not the opposite; (2) visual information is generally under utilization compared with text.

Video event extraction aims to detect salient events from a video and identify the arguments for each event as well as their semantic roles. Existing methods focus on capturing the overall visual scene of each frame, ignoring fine-grained argument-level information. Inspired by the definition of events as changes of states, we propose a novel framework to detect video events by tracking the changes in the visual states of all involved arguments, which are expected to provide the most informative evidence for the extraction of video events. In order to capture the visual state changes of arguments, we decompose them into changes in pixels within objects, displacements of objects, and interactions among multiple arguments. We further propose Object State Embedding, Object Motion-aware Embedding and Argument Interaction Embedding to encode and track these changes respectively. Experiments on various video event extraction tasks demonstrate significant improvements compared to state-of-the-art models. In particular, on verb classification, we achieve 3.49% absolute gains (19.53% relative gains) in F1@5 on Video Situation Recognition.

大规模的多模式对比预训练已经证明了通过将多种模式映射到共享嵌入空间中的一系列下游任务的可转移功能。通常，这对每种模式都采用了单独的编码器。但是，最近的工作表明，变形金刚可以支持跨多种方式学习并允许知识共享。受此启发，我们研究了各种模式共享的对比语言图像预训练（MS-CLIP）框架。更具体地说，我们质疑在对比预训练期间可以在跨模态共享变压器模型的多少个参数，并严格检查建筑设计选择，以将沿频谱共享的参数比例定位。在研究的条件下，我们观察到，视觉和语言信号的主要统一编码器优于所有其他分离更多参数的变体。此外，我们发现特定于特定于模态的平行模块进一步提高了性能。实验结果表明，所提出的MS-CLIP方法在零摄像机分类中（在YFCC-100M上进行了预训练）中，最多可超过13 \％相对的香草夹，同时支持降低参数。此外，在24个下游视觉任务的集合中，我们的方法在线性探测中优于Vanilla剪辑。此外，我们发现共享参数导致语义概念来自不同方式在嵌入空间中更接近地编码，从而促进了共同的语义结构（例如注意力模式）从语言到视觉的传递。代码可在\ href {https://github.com/hxyou/msclip} {url}中获得。

了解多媒体内容中描述或显示的事件彼此相关是开发可用于真实世界媒体的强大人工智能系统的关键组成部分。尽管许多研究专门用于文本，图像和视频域中的事件理解，但没有一个研究探索事件跨域中经历的复杂关系。例如，新闻文章可能会描述“抗议”事件，而视频显示“逮捕”事件。认识到视觉“逮捕”事件是更广泛的“抗议”事件的一个子事件，这是一个具有挑战性但重要的问题，但前面的工作尚未探讨。在本文中，我们提出了多模式事件关系关系的新任务，以识别这种跨模式事件关系。我们贡献了一个大规模数据集，该数据集由100K视频新文章对组成，以及密集注释的数据的基准。我们还提出了一种弱监督的多模式方法，该方法将来自外部知识库（KB）的常识性知识整合在一起，以预测丰富的多模式事件层次结构。实验表明，我们的模型在我们提出的基准上优于许多竞争基线。我们还对模型的性能进行了详细的分析，并建议未来研究的方向。

很少有射击对象检测（FSOD），目的是使用很少的培训示例来检测新颖的对象，最近对社区引起了极大的研究兴趣。基于度量学习的方法已证明使用基于两分支的暹罗网络对此任务有效，并计算图像区域之间的相似性和几乎没有射击示例以进行检测。但是，在以前的工作中，两个分支之间的相互作用仅在检测头中受到限制，而将其余数百个层留在单独的特征提取中。受到有关视觉变压器和视觉变压器的最新工作的启发，我们通过将交叉转换器纳入功能骨干和检测头中，提出了一种新颖的FSOD基于跨变速器的模型（FCT）。提出了不对称批次的交叉注意，以从不同批次大小的两个分支中汇总关键信息。我们的模型可以通过引入多级交互来改善两个分支之间的几个相似性学习。对Pascal VOC和MSCOCO FSOD基准测试的全面实验证明了我们模型的有效性。

在本文中，我们考虑了从长时间的视频到几分钟的长视频进行分类的问题（例如，烹饪不同的食谱，烹饪不同的食谱，进行不同的家庭装修，创建各种形式的艺术和手工艺品）。准确地对这些活动进行分类，不仅需要识别构成任务的单个步骤，还需要捕获其时间依赖性。这个问题与传统的动作分类大不相同，在传统的动作分类中，模型通常在跨越几秒钟的视频上进行了优化，并且手动修剪以包含简单的原子动作。虽然步骤注释可以使模型的培训能够识别程序活动的各个步骤，但由于长时间视频中手动注释时间界的超级注释，因此该领域的现有大规模数据集不包括此类段标签。为了解决这个问题，我们建议通过利用文本知识库（Wikihow）的遥远监督来自动确定教学视频中的步骤，其中包括对执行各种复杂活动所需的步骤的详细描述。我们的方法使用语言模型来匹配视频中自动转录的语音，以在知识库中逐步描述。我们证明，经过训练的视频模型可以识别这些自动标记的步骤（无手动监督）产生了在四个下游任务上实现卓越的概括性能的表示：识别程序活动，步骤分类，步骤预测和以自我为中心的视频分类。

Contrastive language-image pretraining (CLIP) links vision and language modalities into a unified embedding space, yielding the tremendous potential for vision-language (VL) tasks. While early concurrent works have begun to study this potential on a subset of tasks, important questions remain: 1) What is the benefit of CLIP on unstudied VL tasks? 2) Does CLIP provide benefit in low-shot or domain-shifted scenarios? 3) Can CLIP improve existing approaches without impacting inference or pretraining complexity? In this work, we seek to answer these questions through two key contributions. First, we introduce an evaluation protocol that includes Visual Commonsense Reasoning (VCR), Visual Entailment (SNLI-VE), and Visual Question Answering (VQA), across a variety of data availability constraints and conditions of domain shift. Second, we propose an approach, named CLIP Targeted Distillation (CLIP-TD), to intelligently distill knowledge from CLIP into existing architectures using a dynamically weighted objective applied to adaptively selected tokens per instance. Experiments demonstrate that our proposed CLIP-TD leads to exceptional gains in the low-shot (up to 51.9%) and domain-shifted (up to 71.3%) conditions of VCR, while simultaneously improving performance under standard fully-supervised conditions (up to 2%), achieving state-of-art performance on VCR compared to other single models that are pretrained with image-text data only. On SNLI-VE, CLIP-TD produces significant gains in low-shot conditions (up to 6.6%) as well as fully supervised (up to 3%). On VQA, CLIP-TD provides improvement in low-shot (up to 9%), and in fully-supervised (up to 1.3%). Finally, CLIP-TD outperforms concurrent works utilizing CLIP for finetuning, as well as baseline naive distillation approaches. Code will be made available.

Vision-Language（V + L）预先润廓模型通过了解图像和文本之间的对齐来支持多媒体应用程序取得了巨大成功。虽然现有的视觉预押模型主要专注于了解文本中的图像或实体中的对象，但它们通常会忽略事件级别的对齐及其参数结构。 ％在这项工作中，我们提出了一种对比的学习框架来强制执行愿景 - 语言预押模型来理解事件和相关参数（参与者）角色。为此，我们利用文本信息提取技术来获得事件结构知识，并利用多个提示函数来通过操纵事件结构来对比难度的负面描述。我们还基于最佳传输来设计事件图对齐损耗以捕获事件参数结构。此外，我们收集了一个大型活动的数据集（106,875张图片），用于预磨平，这提供了更具挑战性的图像检索基准，以评估对复杂冗长的句子的理解。实验表明，我们的零射剪辑事件优于在多媒体事件提取中的参数提取中的最先进的监督模型，从而实现了事件提取中的5±绝对f得分增益，以及显着改进零拍摄设置下的各种下游任务。