Pre-trained Vision-Language Models (VLMs) such as CLIP have shown impressive generalization capability in downstream vision tasks with appropriate text prompts. Instead of designing prompts manually, Context Optimization (CoOp) has been recently proposed to learn continuous prompts using task-specific training data. Despite the performance improvements on downstream tasks, several studies have reported that CoOp suffers from the overfitting issue in two aspects: (i) the test accuracy on base classes first gets better and then gets worse during training; (ii) the test accuracy on novel classes keeps decreasing. However, none of the existing studies can understand and mitigate such overfitting problem effectively. In this paper, we first explore the cause of overfitting by analyzing the gradient flow. Comparative experiments reveal that CoOp favors generalizable and spurious features in the early and later training stages respectively, leading to the non-overfitting and overfitting phenomenon. Given those observations, we propose Subspace Prompt Tuning (SubPT) to project the gradients in back-propagation onto the low-rank subspace spanned by the early-stage gradient flow eigenvectors during the entire training process, and successfully eliminate the overfitting problem. Besides, we equip CoOp with Novel Feature Learner (NFL) to enhance the generalization ability of the learned prompts onto novel categories beyond the training set, needless of image training data. Extensive experiments on 11 classification datasets demonstrate that SubPT+NFL consistently boost the performance of CoOp and outperform the state-of-the-art approach CoCoOp. Experiments on more challenging vision downstream tasks including open-vocabulary object detection and zero-shot semantic segmentation also verify the effectiveness of the proposed method. Codes can be found at https://tinyurl.com/mpe64f89.

Abstractive dialogue summarization has long been viewed as an important standalone task in natural language processing, but no previous work has explored the possibility of whether abstractive dialogue summarization can also be used as a means to boost an NLP system's performance on other important dialogue comprehension tasks. In this paper, we propose a novel type of dialogue summarization task - STRUctured DiaLoguE Summarization - that can help pre-trained language models to better understand dialogues and improve their performance on important dialogue comprehension tasks. We further collect human annotations of STRUDEL summaries over 400 dialogues and introduce a new STRUDEL dialogue comprehension modeling framework that integrates STRUDEL into a graph-neural-network-based dialogue reasoning module over transformer encoder language models to improve their dialogue comprehension abilities. In our empirical experiments on two important downstream dialogue comprehension tasks - dialogue question answering and dialogue response prediction - we show that our STRUDEL dialogue comprehension model can significantly improve the dialogue comprehension performance of transformer encoder language models.

尖峰神经网络（SNN）引起了脑启发的人工智能和计算神经科学的广泛关注。它们可用于在多个尺度上模拟大脑中的生物信息处理。更重要的是，SNN是适当的抽象水平，可以将大脑和认知的灵感带入人工智能。在本文中，我们介绍了脑启发的认知智力引擎（Braincog），用于创建脑启发的AI和脑模拟模型。 Braincog将不同类型的尖峰神经元模型，学习规则，大脑区域等作为平台提供的重要模块。基于这些易于使用的模块，BrainCog支持各种受脑启发的认知功能，包括感知和学习，决策，知识表示和推理，运动控制和社会认知。这些受脑启发的AI模型已在各种受监督，无监督和强化学习任务上有效验证，并且可以用来使AI模型具有多种受脑启发的认知功能。为了进行大脑模拟，Braincog实现了决策，工作记忆，神经回路的结构模拟以及小鼠大脑，猕猴大脑和人脑的整个大脑结构模拟的功能模拟。一个名为BORN的AI引擎是基于Braincog开发的，它演示了如何将Braincog的组件集成并用于构建AI模型和应用。为了使科学追求解码生物智能的性质并创建AI，Braincog旨在提供必要且易于使用的构件，并提供基础设施支持，以开发基于脑部的尖峰神经网络AI，并模拟认知大脑在多个尺度上。可以在https://github.com/braincog-x上找到Braincog的在线存储库。

旨在选择最有用的培训样本子集的CoreSet选择是一个长期存在的学习问题，可以使许多下游任务受益，例如数据效率学习，持续学习，神经体系结构搜索，主动学习等。但是，许多现有的核心选择方法不是为深度学习而设计的，这些方法可能具有很高的复杂性和不良的概括性能。此外，最近提出的方法在模型，数据集和不同复杂性的设置上进行评估。为了促进深度学习中核心选择的研究，我们贡献了一个全面的代码库，即深核，并就CIFAR10和Imagenet数据集的流行核心选择方法提供了经验研究。关于CIFAR10和Imagenet数据集的广泛实验验证，尽管在某些实验设置中具有优势，但随机选择仍然是一个强大的基线。

软机器人抓手有助于富含接触的操作，包括对各种物体的强大抓握。然而，软抓手的有益依从性也会导致重大变形，从而使精确的操纵具有挑战性。我们提出视觉压力估计与控制（VPEC），这种方法可以使用外部摄像头的RGB图像施加的软握力施加的压力。当气动抓地力和肌腱握力与平坦的表面接触时，我们为视觉压力推断提供了结果。我们还表明，VPEC可以通过对推断压力图像的闭环控制进行精确操作。在我们的评估中，移动操纵器（来自Hello Robot的拉伸RE1）使用Visual Servoing在所需的压力下进行接触；遵循空间压力轨迹；并掌握小型低调的物体，包括microSD卡，一分钱和药丸。总体而言，我们的结果表明，对施加压力的视觉估计可以使软抓手能够执行精确操作。

人们经常通过双手施加压力来与周围环境互动。虽然可以通过在手和环境之间放置压力传感器来测量手动压力，但这样做可以改变接触力学，干扰人类触觉感知，需要昂贵的传感器，并且对大型环境的扩展很差。我们探索使用常规的RGB摄像头推断手动压力的可能性，从而使机器对无爆炸的手和表面的手动压力感知。中心洞察力是，通过手的施加压力会导致内容丰富的外观变化。手共有生物力学特性，从而产生相似的可观察现象，例如软组织变形，血液分布，手姿势和铸造阴影。我们收集了36位参与者的视频，这些参与者具有不同的肤色，向仪器的平面表面施加压力。然后，我们训练了一个深层模型（压力visionnet），以从单个RGB图像中推断出压力图像。我们的模型会在培训数据外降低给参与者的压力，并且表现优于基准。我们还表明，我们的模型的输出取决于手的外观，并在接触区域附近投射阴影。总体而言，我们的结果表明，可以使用以前未观察到的人手的出现来准确推断施加压力。数据，代码和模型可在线提供。

Knowledge graphs (KG) have served as the key component of various natural language processing applications. Commonsense knowledge graphs (CKG) are a special type of KG, where entities and relations are composed of free-form text. However, previous works in KG completion and CKG completion suffer from long-tail relations and newly-added relations which do not have many know triples for training. In light of this, few-shot KG completion (FKGC), which requires the strengths of graph representation learning and few-shot learning, has been proposed to challenge the problem of limited annotated data. In this paper, we comprehensively survey previous attempts on such tasks in the form of a series of methods and applications. Specifically, we first introduce FKGC challenges, commonly used KGs, and CKGs. Then we systematically categorize and summarize existing works in terms of the type of KGs and the methods. Finally, we present applications of FKGC models on prediction tasks in different areas and share our thoughts on future research directions of FKGC.

Unsupervised domain adaptation (UDA) for semantic segmentation is a promising task freeing people from heavy annotation work. However, domain discrepancies in low-level image statistics and high-level contexts compromise the segmentation performance over the target domain. A key idea to tackle this problem is to perform both image-level and feature-level adaptation jointly. Unfortunately, there is a lack of such unified approaches for UDA tasks in the existing literature. This paper proposes a novel UDA pipeline for semantic segmentation that unifies image-level and feature-level adaptation. Concretely, for image-level domain shifts, we propose a global photometric alignment module and a global texture alignment module that align images in the source and target domains in terms of image-level properties. For feature-level domain shifts, we perform global manifold alignment by projecting pixel features from both domains onto the feature manifold of the source domain; and we further regularize category centers in the source domain through a category-oriented triplet loss and perform target domain consistency regularization over augmented target domain images. Experimental results demonstrate that our pipeline significantly outperforms previous methods. In the commonly tested GTA5$\rightarrow$Cityscapes task, our proposed method using Deeplab V3+ as the backbone surpasses previous SOTA by 8%, achieving 58.2% in mIoU.

Given the increasingly intricate forms of partial differential equations (PDEs) in physics and related fields, computationally solving PDEs without analytic solutions inevitably suffers from the trade-off between accuracy and efficiency. Recent advances in neural operators, a kind of mesh-independent neural-network-based PDE solvers, have suggested the dawn of overcoming this challenge. In this emerging direction, Koopman neural operator (KNO) is a representative demonstration and outperforms other state-of-the-art alternatives in terms of accuracy and efficiency. Here we present KoopmanLab, a self-contained and user-friendly PyTorch module of the Koopman neural operator family for solving partial differential equations. Beyond the original version of KNO, we develop multiple new variants of KNO based on different neural network architectures to improve the general applicability of our module. These variants are validated by mesh-independent and long-term prediction experiments implemented on representative PDEs (e.g., the Navier-Stokes equation and the Bateman-Burgers equation) and ERA5 (i.e., one of the largest high-resolution data sets of global-scale climate fields). These demonstrations suggest the potential of KoopmanLab to be considered in diverse applications of partial differential equations.

Different people speak with diverse personalized speaking styles. Although existing one-shot talking head methods have made significant progress in lip sync, natural facial expressions, and stable head motions, they still cannot generate diverse speaking styles in the final talking head videos. To tackle this problem, we propose a one-shot style-controllable talking face generation framework. In a nutshell, we aim to attain a speaking style from an arbitrary reference speaking video and then drive the one-shot portrait to speak with the reference speaking style and another piece of audio. Specifically, we first develop a style encoder to extract dynamic facial motion patterns of a style reference video and then encode them into a style code. Afterward, we introduce a style-controllable decoder to synthesize stylized facial animations from the speech content and style code. In order to integrate the reference speaking style into generated videos, we design a style-aware adaptive transformer, which enables the encoded style code to adjust the weights of the feed-forward layers accordingly. Thanks to the style-aware adaptation mechanism, the reference speaking style can be better embedded into synthesized videos during decoding. Extensive experiments demonstrate that our method is capable of generating talking head videos with diverse speaking styles from only one portrait image and an audio clip while achieving authentic visual effects. Project Page: https://github.com/FuxiVirtualHuman/styletalk.