Video Super-Resolution (VSR) aims to restore high-resolution (HR) videos from low-resolution (LR) videos. Existing VSR techniques usually recover HR frames by extracting pertinent textures from nearby frames with known degradation processes. Despite significant progress, grand challenges are remained to effectively extract and transmit high-quality textures from high-degraded low-quality sequences, such as blur, additive noises, and compression artifacts. In this work, a novel Frequency-Transformer (FTVSR) is proposed for handling low-quality videos that carry out self-attention in a combined space-time-frequency domain. First, video frames are split into patches and each patch is transformed into spectral maps in which each channel represents a frequency band. It permits a fine-grained self-attention on each frequency band, so that real visual texture can be distinguished from artifacts. Second, a novel dual frequency attention (DFA) mechanism is proposed to capture the global frequency relations and local frequency relations, which can handle different complicated degradation processes in real-world scenarios. Third, we explore different self-attention schemes for video processing in the frequency domain and discover that a ``divided attention'' which conducts a joint space-frequency attention before applying temporal-frequency attention, leads to the best video enhancement quality. Extensive experiments on three widely-used VSR datasets show that FTVSR outperforms state-of-the-art methods on different low-quality videos with clear visual margins. Code and pre-trained models are available at https://github.com/researchmm/FTVSR.
translated by 谷歌翻译
We propose the first joint audio-video generation framework that brings engaging watching and listening experiences simultaneously, towards high-quality realistic videos. To generate joint audio-video pairs, we propose a novel Multi-Modal Diffusion model (i.e., MM-Diffusion), with two-coupled denoising autoencoders. In contrast to existing single-modal diffusion models, MM-Diffusion consists of a sequential multi-modal U-Net for a joint denoising process by design. Two subnets for audio and video learn to gradually generate aligned audio-video pairs from Gaussian noises. To ensure semantic consistency across modalities, we propose a novel random-shift based attention block bridging over the two subnets, which enables efficient cross-modal alignment, and thus reinforces the audio-video fidelity for each other. Extensive experiments show superior results in unconditional audio-video generation, and zero-shot conditional tasks (e.g., video-to-audio). In particular, we achieve the best FVD and FAD on Landscape and AIST++ dancing datasets. Turing tests of 10k votes further demonstrate dominant preferences for our model. The code and pre-trained models can be downloaded at https://github.com/researchmm/MM-Diffusion.
translated by 谷歌翻译
The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.
translated by 谷歌翻译
Low-light stereo image enhancement (LLSIE) is a relatively new task to enhance the quality of visually unpleasant stereo images captured in dark conditions. So far, very few studies on deep LLSIE have been explored due to certain challenging issues, i.e., the task has not been well addressed, and current methods clearly suffer from two shortages: 1) insufficient cross-view interaction; 2) lacking long-range dependency for intra-view learning. In this paper, we therefore propose a novel LLSIE model, termed \underline{Suf}ficient C\underline{r}oss-View \underline{In}teraction Network (SufrinNet). To be specific, we present sufficient inter-view interaction module (SIIM) to enhance the information exchange across views. SIIM not only discovers the cross-view correlations at different scales, but also explores the cross-scale information interaction. Besides, we present a spatial-channel information mining block (SIMB) for intra-view feature extraction, and the benefits are twofold. One is the long-range dependency capture to build spatial long-range relationship, and the other is expanded channel information refinement that enhances information flow in channel dimension. Extensive experiments on Flickr1024, KITTI 2012, KITTI 2015 and Middlebury datasets show that our method obtains better illumination adjustment and detail recovery, and achieves SOTA performance compared to other related methods. Our codes, datasets and models will be publicly available.
translated by 谷歌翻译
AI Illustrator旨在自动设计具有视觉吸引力的图像,以激发丰富的思想和情感。为了实现这一目标,我们提出了一个框架,将具有复杂语义的原始描述转换为语义相应的图像。主要的挑战在于原始描述语义的复杂性,可能很难可视化(\ textit {e}。通常,它对现有方法构成了处理此类描述的挑战。为了解决这个问题,我们建议基于rompt \ textbf {c} ross- \ textbf {m} odal generation \ textbf {frame} work(pcm-frame)利用两个强大的预培养模型,,包括剪辑和Stylegan。我们的框架由两个组件组成:\ textIt {textIt嵌入} s到\ textit {image嵌入} s的投影模块,基于提示以及一个构建的适应图像生成模块,该模块构建了\ textit {image嵌入{image Embedding} s作为输入并受到共同语义一致性损失的训练。为了弥合现实图像和插图设计之间的差距,我们进一步采用了风格化模型作为后处理,以获得更好的视觉效果。受益于预先训练的模型,我们的方法可以处理复杂的描述,并且不需要外部配对数据进行培训。此外,我们已经建立了一个由200个原始描述组成的基准。我们进行了一项用户研究,以证明我们对复杂文本的竞争方法的优势。我们在https://github.com/researchmm/ai \ _illustrator} {https://github.com/researchmem/researchmm/ai \_illustrator上发布代码
translated by 谷歌翻译
图像增强旨在通过修饰颜色和音调来提高照片的美学视觉质量,并且是专业数字摄影的必不可少的技术。近年来,基于学习的图像增强算法已达到有希望的表现,并吸引了日益普及。但是,典型的努力试图为所有像素的颜色转换构建一个均匀的增强子。它忽略了对照片重要的不同内容(例如,天空,海洋等)之间的像素差异,从而导致结果不令人满意。在本文中,我们提出了一个新颖的可学习背景知觉的4维查找表(4D LUT),该表通过适应性地学习照片上下文来实现每个图像中不同内容的增强。特别是,我们首先引入一个轻量级上下文编码器和一个参数编码器,以分别学习像素级类别的上下文图和一组图像自适应系数。然后,通过通过系数集成多个基础4D LUT来生成上下文感知的4D LUT。最后,可以通过将源图像和上下文图馈入融合的上下文感知的4D〜LUT来获得增强的图像。与传统的3D LUT(即RGB映射到RGB)相比,通常用于摄像机成像管道系统或工具,4D LUT,即RGBC(RGB+上下文)映射到RGB,可实现具有不同像素的颜色转换的最佳控制每个图像中的内容,即使它们具有相同的RGB值。实验结果表明,我们的方法在广泛使用的基准中优于其他最先进的方法。
translated by 谷歌翻译
关于语言引导的图像操纵的最新作品在提供丰富的语义方面表现出了极大的语言力量,尤其是对于面部图像。但是,语言中的其他自然信息,动作的探索较少。在本文中,我们利用运动信息并研究一项新颖的任务,语言引导的面部动画,旨在在语言的帮助下对静态面部图像进行动画。为了更好地利用语言的语义和动作,我们提出了一个简单而有效的框架。具体而言,我们提出了一个经常性运动生成器,以从语言中提取一系列语义和运动信息,并将其与视觉信息一起提供给预训练的样式,以生成高质量的帧。为了优化所提出的框架,提出了三个精心设计的损失功能,包括保持面部身份的正规化损失,路径长度正规化损失以确保运动平滑度和对比度损失,以在一个模型中使用各种语言指导启用视频综合。对不同领域的定性和定量评估进行了广泛的实验(\ textit {ef。语。代码将在https://github.com/tiankaihang/language-guided-animation.git上找到。
translated by 谷歌翻译
压缩视频超分辨率(VSR)旨在从压缩的低分辨率对应物中恢复高分辨率帧。最近的VSR方法通常通过借用相邻视频帧的相关纹理来增强输入框架。尽管已经取得了一些进展,但是从压缩视频中有效提取和转移高质量纹理的巨大挑战,这些视频通常会高度退化。在本文中,我们提出了一种用于压缩视频超分辨率(FTVSR)的新型频率转换器,该频率在联合时空频域中进行自我注意。首先,我们将视频框架分为斑块,然后将每个贴片转换为DCT光谱图,每个通道代表频带。这样的设计使每个频带都可以进行细粒度的自我注意力,因此可以将真实的视觉纹理与伪影区分开,并进一步用于视频框架修复。其次,我们研究了不同的自我发场方案,并发现在对每个频带上应用暂时关注之前,会引起关节空间的注意力,从而带来最佳的视频增强质量。两个广泛使用的视频超分辨率基准的实验结果表明,FTVSR在未压缩和压缩视频的最先进的方法中都具有清晰的视觉边距。代码可在https://github.com/researchmm/ftvsr上找到。
translated by 谷歌翻译
近年来,基于深度学习的模型在视频超分辨率(VSR)方面取得了显着性能,但是这些模型中的大多数不适用于在线视频应用程序。这些方法仅考虑失真质量,而忽略了在线应用程序的关键要求,例如低延迟和模型较低的复杂性。在本文中,我们专注于在线视频传输,其中需要VSR算法来实时生成高分辨率的视频序列。为了应对此类挑战,我们提出了一种基于一种新的内核知识转移方法,称为卷积核旁路移植物(CKBG)。首先,我们设计了一个轻巧的网络结构,该结构不需要将来的帧作为输入,并节省了缓存这些帧的额外时间成本。然后,我们提出的CKBG方法通过用``核移植物)''绕过原始网络来增强这种轻巧的基础模型,这些网络是包含外部预验证图像SR模型的先验知识的额外卷积内核。在测试阶段,我们通过将其转换为简单的单路结构来进一步加速移植的多支球网络。实验结果表明,我们提出的方法可以处理高达110 fps的在线视频序列,并且模型复杂性非常低和竞争性SR性能。
translated by 谷歌翻译
很少有人提出了几乎没有阶级的课程学习(FSCIL),目的是使深度学习系统能够逐步学习有限的数据。最近,一位先驱声称,通常使用的基于重播的课堂学习方法(CIL)是无效的,因此对于FSCIL而言并不是首选。如果真理,这对FSCIL领域产生了重大影响。在本文中,我们通过经验结果表明,采用数据重播非常有利。但是,存储和重播旧数据可能会导致隐私问题。为了解决此问题,我们或建议使用无数据重播,该重播可以通过发电机综合数据而无需访问真实数据。在观察知识蒸馏的不确定数据的有效性时,我们在发电机培训中强加了熵正则化,以鼓励更不确定的例子。此外,我们建议使用单速样标签重新标记生成的数据。这种修改使网络可以通过完全减少交叉渗透损失来学习,从而减轻了在常规知识蒸馏方法中平衡不同目标的问题。最后,我们对CIFAR-100,Miniimagenet和Cub-200展示了广泛的实验结果和分析,以证明我们提出的效果。
translated by 谷歌翻译