来自多个来源的一些数据可以建模为具有不同采样频率，数据组成，时间关系和特征不同的多模式时间序列事件。不同类型的事件具有复杂的非线性关系，每个事件的时间都是不规则的。经典的复发性神经网络（RNN）模型和当前最新变压器模型都无法很好地处理这些功能。在本文中，基于长期短期存储网络（LSTM）提出了用于多模式不规则时间序列事件的特征融合框架。首先，根据不同事件的不规则模式提取复杂特征。其次，将复杂特征之间的非线性相关性和复杂的时间依赖关系捕获并融合到张量中。最后，使用特征门来控制不同张量的访问频率。对MIMIC-III数据集进行的广泛实验表明，所提出的框架在AUC（接收器操作特征曲线下的区域）和AP（平均精度）方面显着优于现有方法。

近年来，通过应用数百种gan变体，艺术绘画取得了重大进展。但是，据报道，对抗性训练是不稳定的，可能导致模式崩溃。最近，扩散模型已在没有对抗训练的情况下达到了GAN级样品质量。使用自动编码器将原始图像投射到压缩的潜在空间中，并随着扩散的骨干而增强了U-NET，潜在扩散模型已达到稳定且高生育力图像的产生。在本文中，我们专注于在两个方向上增强当前潜在扩散模型的创造性绘画能力，即使用Wikiart DataSet的文本条件扩展和模型再培训。通过文本条件扩展，用户的输入提示在时间和空间方向上扩展，以深入了解和解释提示。 Wikiart DataSet包含了800年来由1,000多名著名艺术家的丰富风格和流派绘制的80K著名艺术品。通过再培训，我们能够要求这些艺术家在现代主题上绘制新颖和创意的绘画。

Citrinet是基于端到端卷积连接派时间分类（CTC）自动语音识别（ASR）模型。为了捕获本地和全球上下文信息，Citrinet中使用了1D时间通道可分开的卷积与子词编码和挤压和兴奋（SE）的结合（SE），使整个体系结构与23个块和235个卷积层一样深和46个线性层。这种纯净的卷积和深度建筑使得critrinet在收敛时相对较慢。在本文中，我们建议在Citrinet块中的卷积模块中引入多头关注，同时保持SE模块和残留模块不变。为了加速加速，我们在每个注意力增强的Citrinet块中删除了8个卷积层，并将23个块减少到13个。日本CSJ-500H和Magic-1600h的实验表明，注意力增强的Citrinet具有较少的层和块，并更快地将其构图和嵌段。比（1）Citrinet具有80 \％训练时间的CITRINET，并且具有40 \％训练时间和29.8％型号的构象异构体。

通过利用变形金刚捕获基于内容的全球互动和卷积神经网络对本地特征的利用，Condormer在自动语音识别（ASR）方面取得了令人印象深刻的结果。在构象异构体中，两个具有一半剩余连接的马卡龙状进料层将多头的自我注意和卷积模块夹在一起，然后是后层的归一化。我们在两个方向上提高了构象异构器的长序列能力，\ emph {sparser}和\ emph {更深层次}。我们使用$ \ Mathcal {o}（l \ text {log} l）$在时间复杂性和内存使用情况下调整稀疏的自我发挥机制。在执行剩余连接时，将使用深层的归一化策略，以确保我们对一百级构象体块的培训。在日本CSJ-500H数据集上，这种深稀疏的构象异构体分别达到5.52 \％，4.03 \％和4.50 \％在三个评估集上和4.16 \％，2.84 \％\％和3.20 \％时，当结合五个深度稀疏的稀疏配置符号时从12到16、17、50，最后100个编码器层的变体。

Masked image modeling (MIM) performs strongly in pre-training large vision Transformers (ViTs). However, small models that are critical for real-world applications cannot or only marginally benefit from this pre-training approach. In this paper, we explore distillation techniques to transfer the success of large MIM-based pre-trained models to smaller ones. We systematically study different options in the distillation framework, including distilling targets, losses, input, network regularization, sequential distillation, etc, revealing that: 1) Distilling token relations is more effective than CLS token- and feature-based distillation; 2) An intermediate layer of the teacher network as target perform better than that using the last layer when the depth of the student mismatches that of the teacher; 3) Weak regularization is preferred; etc. With these findings, we achieve significant fine-tuning accuracy improvements over the scratch MIM pre-training on ImageNet-1K classification, using all the ViT-Tiny, ViT-Small, and ViT-base models, with +4.2%/+2.4%/+1.4% gains, respectively. Our TinyMIM model of base size achieves 52.2 mIoU in AE20K semantic segmentation, which is +4.1 higher than the MAE baseline. Our TinyMIM model of tiny size achieves 79.6% top-1 accuracy on ImageNet-1K image classification, which sets a new record for small vision models of the same size and computation budget. This strong performance suggests an alternative way for developing small vision Transformer models, that is, by exploring better training methods rather than introducing inductive biases into architectures as in most previous works. Code is available at https://github.com/OliverRensu/TinyMIM.

In this paper, we propose a robust 3D detector, named Cross Modal Transformer (CMT), for end-to-end 3D multi-modal detection. Without explicit view transformation, CMT takes the image and point clouds tokens as inputs and directly outputs accurate 3D bounding boxes. The spatial alignment of multi-modal tokens is performed implicitly, by encoding the 3D points into multi-modal features. The core design of CMT is quite simple while its performance is impressive. CMT obtains 73.0% NDS on nuScenes benchmark. Moreover, CMT has a strong robustness even if the LiDAR is missing. Code will be released at https://github.com/junjie18/CMT.

Dataset distillation has emerged as a prominent technique to improve data efficiency when training machine learning models. It encapsulates the knowledge from a large dataset into a smaller synthetic dataset. A model trained on this smaller distilled dataset can attain comparable performance to a model trained on the original training dataset. However, the existing dataset distillation techniques mainly aim at achieving the best trade-off between resource usage efficiency and model utility. The security risks stemming from them have not been explored. This study performs the first backdoor attack against the models trained on the data distilled by dataset distillation models in the image domain. Concretely, we inject triggers into the synthetic data during the distillation procedure rather than during the model training stage, where all previous attacks are performed. We propose two types of backdoor attacks, namely NAIVEATTACK and DOORPING. NAIVEATTACK simply adds triggers to the raw data at the initial distillation phase, while DOORPING iteratively updates the triggers during the entire distillation procedure. We conduct extensive evaluations on multiple datasets, architectures, and dataset distillation techniques. Empirical evaluation shows that NAIVEATTACK achieves decent attack success rate (ASR) scores in some cases, while DOORPING reaches higher ASR scores (close to 1.0) in all cases. Furthermore, we conduct a comprehensive ablation study to analyze the factors that may affect the attack performance. Finally, we evaluate multiple defense mechanisms against our backdoor attacks and show that our attacks can practically circumvent these defense mechanisms.

Blind image quality assessment (BIQA) remains challenging due to the diversity of distortion and image content variation, which complicate the distortion patterns crossing different scales and aggravate the difficulty of the regression problem for BIQA. However, existing BIQA methods often fail to consider multi-scale distortion patterns and image content, and little research has been done on learning strategies to make the regression model produce better performance. In this paper, we propose a simple yet effective Progressive Multi-Task Image Quality Assessment (PMT-IQA) model, which contains a multi-scale feature extraction module (MS) and a progressive multi-task learning module (PMT), to help the model learn complex distortion patterns and better optimize the regression issue to align with the law of human learning process from easy to hard. To verify the effectiveness of the proposed PMT-IQA model, we conduct experiments on four widely used public datasets, and the experimental results indicate that the performance of PMT-IQA is superior to the comparison approaches, and both MS and PMT modules improve the model's performance.

Automatic music generation with artificial intelligence typically requires a large amount of data which is hard to obtain for many less common genres and musical instruments. To tackle this issue, we present ongoing work and preliminary findings on the possibility for deep models to transfer knowledge from language to music, by finetuning large language models pre-trained on a massive text corpus on only hundreds of MIDI files of drum performances. We show that by doing so, one of the largest, state-of-the-art models (GPT3) is capable of generating reasonable drum grooves, while models that are not pre-trained (Transformer) shows no such ability beyond naive repetition. Evaluating generated music is a challenging task, more so is evaluating drum grooves with little precedence in literature. Hence, we propose a tailored structural evaluation method and analyze drum grooves produced by GPT3 compared to those played by human professionals, exposing the strengths and weaknesses of such generation by language-to-music transfer. Our findings suggest that language-to-music transfer learning with large language models is viable and promising.

Few Shot Instance Segmentation (FSIS) requires models to detect and segment novel classes with limited several support examples. In this work, we explore a simple yet unified solution for FSIS as well as its incremental variants, and introduce a new framework named Reference Twice (RefT) to fully explore the relationship between support/query features based on a Transformer-like framework. Our key insights are two folds: Firstly, with the aid of support masks, we can generate dynamic class centers more appropriately to re-weight query features. Secondly, we find that support object queries have already encoded key factors after base training. In this way, the query features can be enhanced twice from two aspects, i.e., feature-level and instance-level. In particular, we firstly design a mask-based dynamic weighting module to enhance support features and then propose to link object queries for better calibration via cross-attention. After the above steps, the novel classes can be improved significantly over our strong baseline. Additionally, our new framework can be easily extended to incremental FSIS with minor modification. When benchmarking results on the COCO dataset for FSIS, gFSIS, and iFSIS settings, our method achieves a competitive performance compared to existing approaches across different shots, e.g., we boost nAP by noticeable +8.2/+9.4 over the current state-of-the-art FSIS method for 10/30-shot. We further demonstrate the superiority of our approach on Few Shot Object Detection. Code and model will be available.