在过去的几十年中，面部识别（FR）在计算机视觉和模式识别社会中进行了积极研究。最近，由于深度学习的进步，FR技术在大多数基准数据集中都显示出高性能。但是，当将FR算法应用于现实世界的情况时，该性能仍然不令人满意。这主要归因于训练和测试集之间的不匹配。在此类不匹配中，训练和测试面之间的面部不对对准是阻碍成功的FR的因素之一。为了解决这一限制，我们提出了一个脸型引导的深度特征对齐框架，以使fr稳健地对脸错位。基于面部形状的先验（例如，面部关键点），我们通过引入对齐方式和未对准的面部图像之间的对齐过程，即像素和特征对齐方式来训练所提出的深网。通过像从面部图像和面部形状提取的聚合特征解码的像素对齐过程，我们添加了辅助任务以重建良好的面部图像。由于汇总功能通过特征对齐过程链接到面部功能提取网络作为指南，因此我们将强大的面部功能训练到面部未对准。即使在训练阶段需要面部形状估计，通常在传统的FR管道中纳入的额外面部对齐过程在测试阶段不一定需要。通过比较实验，我们验证了提出的方法与FR数据集的面部未对准的有效性。

唇读旨在仅基于唇部运动来预测语音。当它专注于视觉信息以建模语音时，其性能本质上对个人唇部外观和动作敏感。这使得唇读模型由于训练和测试条件之间的不匹配而将其应用于看不见的说话者时显示出降级的性能。演讲者的适应技术旨在减少火车和测试扬声器之间的不匹配，从而指导训练有素的模型，以专注于对语音内容进行建模而不由说话者变化介入。与数十年来基于音频的语音识别所做的努力相反，扬声器适应方法在唇部阅读中尚未得到很好的研究。在本文中，为了纠正看不见的扬声器的唇读模型的性能降解，我们提出了一种扬声器自适应的唇部阅读方法，即用户依赖用户。依赖用户的填充是一种特定于扬声器的输入，可以参与预训练的唇读模型的视觉特征提取阶段。因此，可以在编码视觉功能期间考虑不同扬声器的唇外观和动作信息，适合单个扬声器。此外，所提出的方法不需要1）任何其他层，2）修改预训练模型的学习权重，以及3）预训练期间使用的火车数据的扬声器标签。它只能以受监督或无监督的方式仅学习用户依赖的填充，直接适应了看不见的说话者。最后，为了减轻公共唇阅读数据库中的扬声器信息不足，我们将众所周知的视听数据库的扬声器标记为LRW，并设计出一种名为LRW-ID的不可见语的唇lip阅读方案。

本文着重于设计一种噪声端到端音频语音识别（AVSR）系统。为此，我们提出了视觉上下文驱动的音频功能增强模块（V-Cafe），以在视听通讯的帮助下增强输入噪声音频语音。所提出的V-Cafe旨在捕获唇部运动的过渡，即视觉上下文，并通过考虑获得的视觉上下文来产生降噪面膜。通过与上下文相关的建模，可以完善掩模生成Viseme-to-phoneme映射中的歧义。嘈杂的表示用降噪面膜掩盖，从而增强了音频功能。增强的音频功能与视觉特征融合在一起，并将其带入由构象异构体和变压器组成的编码器模型，以进行语音识别。我们显示了带有V-fafe的端到端AVSR，可以进一步改善AVSR的噪声。使用两个最大的视听数据集LRS2和LRS3评估了所提出方法的有效性。

这项工作的目的是从无声说话的脸部视频中重建演讲。最近的研究表明，来自无声说话面部视频的综合语音表现令人印象深刻。但是，他们尚未明确考虑不同扬声器的不同身份特征，这些特征在视频到语音综合中构成了挑战，这对于不可见的扬声器设置变得更加至关重要。与以前的方法不同，我们的方法是将语音内容和外观风格与给定的无声说话的面部视频分开。通过指导模型独立专注于建模这两个表示形式，即使给出了看不见主题的输入视频，我们也可以从模型中获得高清晰度的语音。为此，我们介绍了语音视觉选择模块，该模块将语音内容和扬声器身份与输入视频的视觉特征分开。分散的表示形式通过基于VISAGE风格的合成器共同纳入综合语音，该合成器通过在维护语音内容的同时涂上VISAGE风格来产生语音。因此，提议的框架带来了合成语音包含正确内容的优势，即使给出了看不见的主题的无声说话的脸部视频。我们验证了在网格，TCD-TIMIT志愿者和LRW数据集上提出的框架的有效性。可以在补充材料中听到综合语音。

Few Shot Instance Segmentation (FSIS) requires models to detect and segment novel classes with limited several support examples. In this work, we explore a simple yet unified solution for FSIS as well as its incremental variants, and introduce a new framework named Reference Twice (RefT) to fully explore the relationship between support/query features based on a Transformer-like framework. Our key insights are two folds: Firstly, with the aid of support masks, we can generate dynamic class centers more appropriately to re-weight query features. Secondly, we find that support object queries have already encoded key factors after base training. In this way, the query features can be enhanced twice from two aspects, i.e., feature-level and instance-level. In particular, we firstly design a mask-based dynamic weighting module to enhance support features and then propose to link object queries for better calibration via cross-attention. After the above steps, the novel classes can be improved significantly over our strong baseline. Additionally, our new framework can be easily extended to incremental FSIS with minor modification. When benchmarking results on the COCO dataset for FSIS, gFSIS, and iFSIS settings, our method achieves a competitive performance compared to existing approaches across different shots, e.g., we boost nAP by noticeable +8.2/+9.4 over the current state-of-the-art FSIS method for 10/30-shot. We further demonstrate the superiority of our approach on Few Shot Object Detection. Code and model will be available.

For Prognostics and Health Management (PHM) of Lithium-ion (Li-ion) batteries, many models have been established to characterize their degradation process. The existing empirical or physical models can reveal important information regarding the degradation dynamics. However, there is no general and flexible methods to fuse the information represented by those models. Physics-Informed Neural Network (PINN) is an efficient tool to fuse empirical or physical dynamic models with data-driven models. To take full advantage of various information sources, we propose a model fusion scheme based on PINN. It is implemented by developing a semi-empirical semi-physical Partial Differential Equation (PDE) to model the degradation dynamics of Li-ion-batteries. When there is little prior knowledge about the dynamics, we leverage the data-driven Deep Hidden Physics Model (DeepHPM) to discover the underlying governing dynamic models. The uncovered dynamics information is then fused with that mined by the surrogate neural network in the PINN framework. Moreover, an uncertainty-based adaptive weighting method is employed to balance the multiple learning tasks when training the PINN. The proposed methods are verified on a public dataset of Li-ion Phosphate (LFP)/graphite batteries.

New architecture GPUs like A100 are now equipped with multi-instance GPU (MIG) technology, which allows the GPU to be partitioned into multiple small, isolated instances. This technology provides more flexibility for users to support both deep learning training and inference workloads, but efficiently utilizing it can still be challenging. The vision of this paper is to provide a more comprehensive and practical benchmark study for MIG in order to eliminate the need for tedious manual benchmarking and tuning efforts. To achieve this vision, the paper presents MIGPerf, an open-source tool that streamlines the benchmark study for MIG. Using MIGPerf, the authors conduct a series of experiments, including deep learning training and inference characterization on MIG, GPU sharing characterization, and framework compatibility with MIG. The results of these experiments provide new insights and guidance for users to effectively employ MIG, and lay the foundation for further research on the orchestration of hybrid training and inference workloads on MIGs. The code and results are released on https://github.com/MLSysOps/MIGProfiler. This work is still in progress and more results will be published soon.

In the scenario of black-box adversarial attack, the target model's parameters are unknown, and the attacker aims to find a successful adversarial perturbation based on query feedback under a query budget. Due to the limited feedback information, existing query-based black-box attack methods often require many queries for attacking each benign example. To reduce query cost, we propose to utilize the feedback information across historical attacks, dubbed example-level adversarial transferability. Specifically, by treating the attack on each benign example as one task, we develop a meta-learning framework by training a meta-generator to produce perturbations conditioned on benign examples. When attacking a new benign example, the meta generator can be quickly fine-tuned based on the feedback information of the new task as well as a few historical attacks to produce effective perturbations. Moreover, since the meta-train procedure consumes many queries to learn a generalizable generator, we utilize model-level adversarial transferability to train the meta-generator on a white-box surrogate model, then transfer it to help the attack against the target model. The proposed framework with the two types of adversarial transferability can be naturally combined with any off-the-shelf query-based attack methods to boost their performance, which is verified by extensive experiments.

Medical image segmentation (MIS) is essential for supporting disease diagnosis and treatment effect assessment. Despite considerable advances in artificial intelligence (AI) for MIS, clinicians remain skeptical of its utility, maintaining low confidence in such black box systems, with this problem being exacerbated by low generalization for out-of-distribution (OOD) data. To move towards effective clinical utilization, we propose a foundation model named EvidenceCap, which makes the box transparent in a quantifiable way by uncertainty estimation. EvidenceCap not only makes AI visible in regions of uncertainty and OOD data, but also enhances the reliability, robustness, and computational efficiency of MIS. Uncertainty is modeled explicitly through subjective logic theory to gather strong evidence from features. We show the effectiveness of EvidenceCap in three segmentation datasets and apply it to the clinic. Our work sheds light on clinical safe applications and explainable AI, and can contribute towards trustworthiness in the medical domain.

Depression is a leading cause of death worldwide, and the diagnosis of depression is nontrivial. Multimodal learning is a popular solution for automatic diagnosis of depression, and the existing works suffer two main drawbacks: 1) the high-order interactions between different modalities can not be well exploited; and 2) interpretability of the models are weak. To remedy these drawbacks, we propose a multimodal multi-order factor fusion (MMFF) method. Our method can well exploit the high-order interactions between different modalities by extracting and assembling modality factors under the guide of a shared latent proxy. We conduct extensive experiments on two recent and popular datasets, E-DAIC-WOZ and CMDC, and the results show that our method achieve significantly better performance compared with other existing approaches. Besides, by analyzing the process of factor assembly, our model can intuitively show the contribution of each factor. This helps us understand the fusion mechanism.