Temporal modeling is crucial for various video learning tasks. Most recent approaches employ either factorized (2D+1D) or joint (3D) spatial-temporal operations to extract temporal contexts from the input frames. While the former is more efficient in computation, the latter often obtains better performance. In this paper, we attribute this to a dilemma between the sufficiency and the efficiency of interactions among various positions in different frames. These interactions affect the extraction of task-relevant information shared among frames. To resolve this issue, we prove that frame-by-frame alignments have the potential to increase the mutual information between frame representations, thereby including more task-relevant information to boost effectiveness. Then we propose Alignment-guided Temporal Attention (ATA) to extend 1-dimensional temporal attention with parameter-free patch-level alignments between neighboring frames. It can act as a general plug-in for image backbones to conduct the action recognition task without any model-specific design. Extensive experiments on multiple benchmarks demonstrate the superiority and generality of our module.

将现有的旅游照片从部分捕获的场景扩展到完整的场景是摄影应用的理想体验之一。尽管对照片的外推进行了充分的研究，但是将照片（即自拍照）从狭窄的视野推断到更广阔的视野，同时保持相似的视觉样式是更具挑战性的。在本文中，我们提出了一个分解的神经重新渲染模型，以从混乱的户外互联网照片集中产生逼真的新颖观点，该视图可以使应用程序包括可控场景重新渲染，照片外推甚至外推3D照片生成。具体而言，我们首先开发出一种新颖的分解重新渲染管道，以处理几何，外观和照明分解中的歧义。我们还提出了一种合成的培训策略，以应对互联网图像中意外的阻塞。此外，为了推断旅游照片时增强照片现实主义，我们提出了一个新颖的现实主义增强过程来补充外观细节，该过程会自动传播质地细节，从狭窄的捕获照片到外推神经渲染图像。室外场景上的实验和照片编辑示例证明了我们在照片现实主义和下游应用中提出的方法的出色性能。

基于学习的多视图立体声（MVS）方法取得了令人印象深刻的进步，并且近年来超越了传统方法。但是，它们的准确性和完整性仍在挣扎。在本文中，我们提出了一种新方法，以增强受对比度学习和功能匹配启发的现有网络的性能。首先，我们提出了一个对比匹配损失（CML），该损失将正确的匹配点视为正样品，将正确的匹配点视为正样本，并将其他点视为阴性样本，并根据特征的相似性计算对比度损失。我们进一步提出了一个加权局灶性损失（WFL），以提高分类能力，从而削弱了根据预测的置信度，在不重要的区域中低信任像素对损失的贡献。在DTU，坦克和寺庙和混合MVS数据集上进行的广泛实验表明，我们的方法可实现最先进的性能，并在基线网络上取得了重大改进。

以前通过一个位置的历史轨迹可能有助于推断该位置当前代理的未来轨迹。尽管在高清图的指导下进行了轨迹预测的大大改善，但只有少数作品探讨了这种当地历史信息。在这项工作中，我们将这些信息重新引入了轨迹预测系统的新类型的输入数据：本地行为数据，我们将其概念化为特定于位置的历史轨迹的集合。局部行为数据有助于系统强调预测区域，并更好地了解静态地图对象对移动代理的影响。我们提出了一个新型的本地行为感知（LBA）预测框架，该框架通过从观察到的轨迹，高清图和局部行为数据中融合信息来提高预测准确性。同样，如果这种历史数据不足或不可用，我们采用了本地行为（LBF）预测框架，该框架采用了基于知识依据的架构来推断缺失数据的影响。广泛的实验表明，通过这两个框架升级现有方法可显着提高其性能。特别是，LBA框架将SOTA方法在Nuscenes数据集上的性能提高了至少14％的K = 1度量。

软演员 - 评论家（SAC）是最先进的偏离策略强化学习（RL）算法之一，其在基于最大熵的RL框架内。 SAC被证明在具有良好稳定性和稳健性的持续控制任务的列表中表现得非常好。 SAC了解一个随机高斯政策，可以最大限度地提高预期奖励和政策熵之间的权衡。要更新策略，SAC可最大限度地减少当前策略密度与软值函数密度之间的kl分歧。然后用于获得这种分歧的近似梯度的回报。在本文中，我们提出了跨熵策略优化（SAC-CEPO）的软演员 - 评论家，它使用跨熵方法（CEM）来优化SAC的政策网络。初始思想是使用CEM来迭代地对软价函数密度的最接近的分布进行采样，并使用结果分布作为更新策略网络的目标。为了降低计算复杂性，我们还介绍了一个解耦的策略结构，该策略结构将高斯策略解耦为一个策略，了解了学习均值的均值和另一个策略，以便只有CEM训练平均政策。我们表明，这种解耦的政策结构确实会聚到最佳，我们还通过实验证明SAC-CEPO实现对原始囊的竞争性能。

In this paper, we propose a robust 3D detector, named Cross Modal Transformer (CMT), for end-to-end 3D multi-modal detection. Without explicit view transformation, CMT takes the image and point clouds tokens as inputs and directly outputs accurate 3D bounding boxes. The spatial alignment of multi-modal tokens is performed implicitly, by encoding the 3D points into multi-modal features. The core design of CMT is quite simple while its performance is impressive. CMT obtains 73.0% NDS on nuScenes benchmark. Moreover, CMT has a strong robustness even if the LiDAR is missing. Code will be released at https://github.com/junjie18/CMT.

Dataset distillation has emerged as a prominent technique to improve data efficiency when training machine learning models. It encapsulates the knowledge from a large dataset into a smaller synthetic dataset. A model trained on this smaller distilled dataset can attain comparable performance to a model trained on the original training dataset. However, the existing dataset distillation techniques mainly aim at achieving the best trade-off between resource usage efficiency and model utility. The security risks stemming from them have not been explored. This study performs the first backdoor attack against the models trained on the data distilled by dataset distillation models in the image domain. Concretely, we inject triggers into the synthetic data during the distillation procedure rather than during the model training stage, where all previous attacks are performed. We propose two types of backdoor attacks, namely NAIVEATTACK and DOORPING. NAIVEATTACK simply adds triggers to the raw data at the initial distillation phase, while DOORPING iteratively updates the triggers during the entire distillation procedure. We conduct extensive evaluations on multiple datasets, architectures, and dataset distillation techniques. Empirical evaluation shows that NAIVEATTACK achieves decent attack success rate (ASR) scores in some cases, while DOORPING reaches higher ASR scores (close to 1.0) in all cases. Furthermore, we conduct a comprehensive ablation study to analyze the factors that may affect the attack performance. Finally, we evaluate multiple defense mechanisms against our backdoor attacks and show that our attacks can practically circumvent these defense mechanisms.

Automatic music generation with artificial intelligence typically requires a large amount of data which is hard to obtain for many less common genres and musical instruments. To tackle this issue, we present ongoing work and preliminary findings on the possibility for deep models to transfer knowledge from language to music, by finetuning large language models pre-trained on a massive text corpus on only hundreds of MIDI files of drum performances. We show that by doing so, one of the largest, state-of-the-art models (GPT3) is capable of generating reasonable drum grooves, while models that are not pre-trained (Transformer) shows no such ability beyond naive repetition. Evaluating generated music is a challenging task, more so is evaluating drum grooves with little precedence in literature. Hence, we propose a tailored structural evaluation method and analyze drum grooves produced by GPT3 compared to those played by human professionals, exposing the strengths and weaknesses of such generation by language-to-music transfer. Our findings suggest that language-to-music transfer learning with large language models is viable and promising.

Few Shot Instance Segmentation (FSIS) requires models to detect and segment novel classes with limited several support examples. In this work, we explore a simple yet unified solution for FSIS as well as its incremental variants, and introduce a new framework named Reference Twice (RefT) to fully explore the relationship between support/query features based on a Transformer-like framework. Our key insights are two folds: Firstly, with the aid of support masks, we can generate dynamic class centers more appropriately to re-weight query features. Secondly, we find that support object queries have already encoded key factors after base training. In this way, the query features can be enhanced twice from two aspects, i.e., feature-level and instance-level. In particular, we firstly design a mask-based dynamic weighting module to enhance support features and then propose to link object queries for better calibration via cross-attention. After the above steps, the novel classes can be improved significantly over our strong baseline. Additionally, our new framework can be easily extended to incremental FSIS with minor modification. When benchmarking results on the COCO dataset for FSIS, gFSIS, and iFSIS settings, our method achieves a competitive performance compared to existing approaches across different shots, e.g., we boost nAP by noticeable +8.2/+9.4 over the current state-of-the-art FSIS method for 10/30-shot. We further demonstrate the superiority of our approach on Few Shot Object Detection. Code and model will be available.

Graph Neural Networks (GNNs) have shown satisfying performance on various graph learning tasks. To achieve better fitting capability, most GNNs are with a large number of parameters, which makes these GNNs computationally expensive. Therefore, it is difficult to deploy them onto edge devices with scarce computational resources, e.g., mobile phones and wearable smart devices. Knowledge Distillation (KD) is a common solution to compress GNNs, where a light-weighted model (i.e., the student model) is encouraged to mimic the behavior of a computationally expensive GNN (i.e., the teacher GNN model). Nevertheless, most existing GNN-based KD methods lack fairness consideration. As a consequence, the student model usually inherits and even exaggerates the bias from the teacher GNN. To handle such a problem, we take initial steps towards fair knowledge distillation for GNNs. Specifically, we first formulate a novel problem of fair knowledge distillation for GNN-based teacher-student frameworks. Then we propose a principled framework named RELIANT to mitigate the bias exhibited by the student model. Notably, the design of RELIANT is decoupled from any specific teacher and student model structures, and thus can be easily adapted to various GNN-based KD frameworks. We perform extensive experiments on multiple real-world datasets, which corroborates that RELIANT achieves less biased GNN knowledge distillation while maintaining high prediction utility.