Standard video frame interpolation methods first estimate optical flow between input frames and then synthesize an intermediate frame guided by motion. Recent ap-proaches merge these two steps into a single convolution process by convolving input frames with spatially adaptive kernels that account for motion and re-sampling simultaneously. These methods require large kernels to handle large motion, which limits the number of pixels whose kernels can be estimated at once due to the large memory demand. To address this problem, this paper formulates frame interpolation as local separable convolution over input frames using pairs of 1D kernels. Compared to regular 2D kernels, the 1D kernels require significantly fewer parameters to be estimated. Our method develops a deep fully convolutional neural network that takes two input frames and estimates pairs of 1D kernels for all pixels simultaneously. Since our method is able to estimate kernels and synthesizes the whole video frame at once, it allows for the incorporation of perceptual loss to train the neural network to produce visually pleasing frames. This deep neural network is trained end-to-end using widely available video data without any human annotation. Both qualitative and quantitative experiments show that our method provides a practical solution to high-quality video frame interpolation.

We address the problem of synthesizing new video frames in an existing video, either in-between existing frames (interpolation), or subsequent to them (extrapolation). This problem is challenging because video appearance and motion can be highly complex. Traditional optical-flow-based solutions often fail where flow estimation is challenging, while newer neural-network-based methods that hallucinate pixel values directly often produce blurry results. We combine the advantages of these two methods by training a deep network that learns to synthesize video frames by flowing pixel values from existing ones, which we call deep voxel flow. Our method requires no human supervision, and any video can be used as training data by dropping, and then learning to predict, existing frames. The technique is efficient, and can be applied at any video resolution. We demonstrate that our method produces results that both quantitatively and qualitatively improve upon the state-ofthe-art.

Given two consecutive frames, video interpolation aims at generating intermediate frame(s) to form both spatially and temporally coherent video sequences. While most existing methods focus on single-frame interpolation, we propose an end-to-end convolutional neural network for variable-length multi-frame video interpolation, where the motion interpretation and occlusion reasoning are jointly modeled. We start by computing bi-directional optical flow between the input images using a U-Net architecture. These flows are then linearly combined at each time step to approximate the intermediate bi-directional optical flows. These approximate flows, however, only work well in locally smooth regions and produce artifacts around motion boundaries. To address this shortcoming, we employ another U-Net to refine the approximated flow and also predict soft visibility maps. Finally, the two input images are warped and linearly fused to form each intermediate frame. By applying the visibility maps to the warped images before fusion, we exclude the contribution of occluded pixels to the interpolated intermediate frame to avoid artifacts. Since none of our learned network parameters are time-dependent, our approach is able to produce as many intermediate frames as needed. To train our network, we use 1,132 240-fps video clips, containing 300K individual video frames. Experimental results on several datasets, predicting different numbers of interpolated frames, demonstrate that our approach performs consistently better than existing methods.

A difficult example for video frame interpolation. Our approach produces a high-quality result in spite of the delicate flamingo leg that is subject to large motion. This is a video figure that is best viewed using Adobe Reader.

我们提出了一种用于视频帧插值（VFI）的实时中流估计算法。许多最近的基于流的VFI方法首先估计双向光学流，然后缩放并将它们倒转到近似中间流动，导致运动边界上的伪像。RIFE使用名为IFNET的神经网络，可以直接估计中间流量从粗细流，速度更好。我们设计了一种用于训练中间流动模型的特权蒸馏方案，这导致了大的性能改善。Rife不依赖于预先训练的光流模型，可以支持任意时间的帧插值。实验表明，普里埃雷在若干公共基准上实现了最先进的表现。\ url {https://github.com/hzwer/arxiv2020-rife}。

Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a selfsupervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution. IntroductionMotion estimation is a key component in video processing tasks such as temporal frame interpolation, video denoising,

我们提出了一种称为基于DNN的基于DNN的框架，称为基于增强的相关匹配的视频帧插值网络，以支持4K的高分辨率，其具有大规模的运动和遮挡。考虑到根据分辨率的网络模型的可扩展性，所提出的方案采用经常性金字塔架构，该架构分享每个金字塔层之间的参数进行光学流量估计。在所提出的流程估计中，通过追踪具有最大相关性的位置来递归地改进光学流。基于前扭曲的相关匹配可以通过排除遮挡区域周围的错误扭曲特征来提高流量更新的准确性。基于最终双向流动，使用翘曲和混合网络合成任意时间位置的中间帧，通过细化网络进一步改善。实验结果表明，所提出的方案在4K视频数据和低分辨率基准数据集中占据了之前的工作，以及具有最小型号参数的客观和主观质量。

Motion blur from camera shake is a major problem in videos captured by hand-held devices. Unlike single-image deblurring, video-based approaches can take advantage of the abundant information that exists across neighboring frames. As a result the best performing methods rely on the alignment of nearby frames. However, aligning images is a computationally expensive and fragile procedure, and methods that aggregate information must therefore be able to identify which regions have been accurately aligned and which have not, a task that requires high level scene understanding. In this work, we introduce a deep learning solution to video deblurring, where a CNN is trained end-toend to learn how to accumulate information across frames. To train this network, we collected a dataset of real videos recorded with a high frame rate camera, which we use to generate synthetic motion blur for supervision. We show that the features learned from this dataset extend to deblurring motion blur that arises due to camera shake in a wide range of videos, and compare the quality of results to a number of other baselines 1 .

视频帧插值（VFI）目前是一个非常活跃的研究主题，具有跨越计算机视觉，后期生产和视频编码的应用程序。 VFI可能非常具有挑战性，特别是在含有大型运动，闭塞或动态纹理的序列中，现有方法未能提供感知鲁棒的插值性能。在这种情况下，我们基于时空多流量架构介绍了一种基于深度学习的VFI方法ST-MFNET。 ST-MFNET采用新的多尺度多流量预测器来估计多对一的中间流动，它们与传统的一对一光流组合以捕获大型和复杂的运动。为了增强各种纹理的插值性能，还用于在扩展时间窗口上模拟内容动态的3D CNN。此外，ST-MFNET已经在ST-GaN框架内培训，该框架最初是为纹理合成而开发的，目的是进一步提高感知插值质量。我们的方法已被全面评估 - 与十四个最先进的VFI算法相比 - 清楚地展示了ST-MFNET在各种和代表性测试数据集上始终如一地优于这些基准，在PSNR中具有显着的收益，用于案件在PSNR中高达1.09dB包括大型运动和动态纹理。项目页面：https://danielism97.github.io/st-mfnet。

基于DNN的框架插值从两个连续的帧中生成中间帧，通常取决于具有大量功能的模型体系结构，从而阻止其在具有有限资源的系统（例如移动设备）上部署。我们提出了一种用于框架插值的压缩驱动的网络设计，该设计通过稀疏性诱导优化来利用模型，以大大降低模型大小，同时达到更高的性能。具体而言，我们首先压缩了最近提出的ADACOF模型，并证明了10次压缩ADACOF的性能类似于其原始对应物，在各种超参数设置下，对使用layerwise稀疏信息作为指导的不同策略进行了全面研究。然后，我们通过引入一个多分辨率翘曲模块来增强这种压缩模型，从而提高了视觉一致性，并通过多层次的细节来提高视觉一致性。结果，我们通过原始AdaCof的四分之一获得了可观的性能增长。此外，我们的模型在各种数据集上对其他最先进的方法都表现出色。我们注意到，建议的压缩驱动框​​架是通用的，可以轻松地传输到其他基于DNN的框架插值算法中。源代码可在https://github.com/tding1/cdfi上获得。

可以通过定期预测未来的框架以增强虚拟现实应用程序中的用户体验，从而解决了低计算设备上图形渲染高帧速率视频的挑战。这是通过时间视图合成（TVS）的问题来研究的，该问题的目标是预测给定上一个帧的视频的下一个帧以及上一个和下一个帧的头部姿势。在这项工作中，我们考虑了用户和对象正在移动的动态场景的电视。我们设计了一个将运动解散到用户和对象运动中的框架，以在预测下一帧的同时有效地使用可用的用户运动。我们通过隔离和估计过去框架的3D对象运动，然后推断它来预测对象的运动。我们使用多平面图像（MPI）作为场景的3D表示，并将对象运动作为MPI表示中相应点之间的3D位移建模。为了在估计运动时处理MPI中的稀疏性，我们将部分卷积和掩盖的相关层纳入了相应的点。然后将预测的对象运动与给定的用户或相机运动集成在一起，以生成下一帧。使用不合格的填充模块，我们合成由于相机和对象运动而发现的区域。我们为动态场景的电视开发了一个新的合成数据集，该数据集由800个以全高清分辨率组成的视频组成。我们通过数据集和MPI Sintel数据集上的实验表明我们的模型优于文献中的所有竞争方法。

现有的视频框架插值方法只能在给定的中间时间步骤中插值框架，例如1/2。在本文中，我们旨在探索一种更广泛的视频框架插值，该视频框架在任意时步。为此，我们考虑在元学习的帮助下以统一的方式处理不同的时间阶段。具体而言，我们开发了一个双元学习的帧插值框架，以通过上下文信息和光流的指导以及将时间步长为附带信息，将中间框架合成中间框架。首先，构建了一个内容感知的元学习流程模块，以提高基于输入帧的下采样版本的光流估计的准确性。其次，以精致的光流和时间步长为输入，运动吸引的元学习框架插值模块为在粗翘曲版本的特征图上使用的每个像素生成卷积内核，以生成输入的特征图上的每个像素生成预测帧的帧。广泛的定性和定量评估以及消融研究表明，通过以如此精心设计的方式在我们的框架中引入元学习，我们的方法不仅可以实现优于先进的框架插值方法，还可以实现优越的性能还拥有在任意时间步长以支持插值的扩展能力。

Video frame interpolation(VFI) is the task that synthesizes the intermediate frame given two consecutive frames. Most of the previous studies have focused on appropriate frame warping operations and refinement modules for the warped frames. These studies have been conducted on natural videos containing only continuous motions. However, many practical videos contain various unnatural objects with discontinuous motions such as logos, user interfaces and subtitles. We propose three techniques to make the existing deep learning-based VFI architectures robust to these elements. First is a novel data augmentation strategy called figure-text mixing(FTM) which can make the models learn discontinuous motions during training stage without any extra dataset. Second, we propose a simple but effective module that predicts a map called discontinuity map(D-map), which densely distinguishes between areas of continuous and discontinuous motions. Lastly, we propose loss functions to give supervisions of the discontinuous motion areas which can be applied along with FTM and D-map. We additionally collect a special test benchmark called Graphical Discontinuous Motion(GDM) dataset consisting of some mobile games and chatting videos. Applied to the various state-of-the-art VFI networks, our method significantly improves the interpolation qualities on the videos from not only GDM dataset, but also the existing benchmarks containing only continuous motions such as Vimeo90K, UCF101, and DAVIS.

视频框架合成由插值和外推组成，是一种必不可少的视频处理技术，可应用于各种情况。但是，大多数现有方法无法处理小物体或大型运动，尤其是在高分辨率视频（例如4K视频）中。为了消除此类局限性，我们引入了基于流动帧合成的邻居对应匹配（NCM）算法。由于当前的帧在视频框架合成中不可用，因此NCM以当前框架的方式进行，以在每个像素的空间型社区中建立多尺度对应关系。基于NCM的强大运动表示能力，我们进一步建议在异质的粗到细节方案中估算框架合成的中间流。具体而言，粗尺度模块旨在利用邻居的对应关系来捕获大型运动，而细尺度模块在计算上更有效地加快了估计过程。两个模块都经过逐步训练，以消除培训数据集和现实世界视频之间的分辨率差距。实验结果表明，NCM在多个基准测试中实现了最先进的性能。此外，NCM可以应用于各种实践场景，例如视频压缩，以实现更好的性能。

我们提出了Tain（视频插值的变压器和注意力），这是一个用于视频插值的残留神经网络，旨在插入中间框架，并在其周围连续两个图像框架下进行插值。我们首先提出一个新型的视觉变压器模块，称为交叉相似性（CS），以与预测插值框架相似的外观相似的外观。然后，这些CS特征用于完善插值预测。为了说明CS功能中的遮挡，我们提出了一个图像注意（IA）模块，以使网络可以从另一个框架上关注CS功能。此外，我们还使用封闭式贴片来增强培训数据集，该补丁可以跨帧移动，以改善网络对遮挡和大型运动的稳健性。由于现有方法产生平滑的预测，尤其是在MB附近，因此我们根据图像梯度使用额外的训练损失来产生更清晰的预测。胜过不需要流量估计并与基于流程的方法相当执行的现有方法，同时在VIMEO90K，UCF101和SNU-FILM基准的推理时间上具有计算有效的效率。

Non-uniform blind deblurring for general dynamic scenes is a challenging computer vision problem as blurs arise not only from multiple object motions but also from camera shake, scene depth variation. To remove these complicated motion blurs, conventional energy optimization based methods rely on simple assumptions such that blur kernel is partially uniform or locally linear. Moreover, recent machine learning based methods also depend on synthetic blur datasets generated under these assumptions. This makes conventional deblurring methods fail to remove blurs where blur kernel is difficult to approximate or parameterize (e.g. object motion boundaries). In this work, we propose a multi-scale convolutional neural network that restores sharp images in an end-to-end manner where blur is caused by various sources. Together, we present multiscale loss function that mimics conventional coarse-to-fine approaches. Furthermore, we propose a new large-scale dataset that provides pairs of realistic blurry image and the corresponding ground truth sharp image that are obtained by a high-speed camera. With the proposed model trained on this dataset, we demonstrate empirically that our method achieves the state-of-the-art performance in dynamic scene deblurring not only qualitatively, but also quantitatively.

用于深度卷积神经网络的视频插值的现有方法，因此遭受其内在限制，例如内部局限性核心权重和受限制的接收领域。为了解决这些问题，我们提出了一种基于变换器的视频插值框架，允许内容感知聚合权重，并考虑具有自我关注操作的远程依赖性。为避免全球自我关注的高计算成本，我们将当地注意的概念引入视频插值并将其扩展到空间域。此外，我们提出了一个节省时间的分离策略，以节省内存使用，这也提高了性能。此外，我们开发了一种多尺度帧合成方案，以充分实现变压器的潜力。广泛的实验证明了所提出的模型对最先进的方法来说，定量和定性地在各种基准数据集上进行定量和定性。

隐式神经表示（INR）被出现为代表信号的强大范例，例如图像，视频，3D形状等。尽管它已经示出了能够表示精细细节的能力，但其效率尚未得到广泛研究数据表示。在INR中，数据以神经网络的参数的形式存储，并且通用优化算法通常不会利用信号中的空间和时间冗余。在本文中，我们建议通过明确地删除数据冗余来表示和压缩视频的新型INR方法。我们提出了跨视频帧和残差的主体剩余流场（NRFF）而不是存储原始RGB颜色，而不是存储原始RGB颜色。维护通常更光滑和更复杂的运动信息，比原始信号更少，需要更少的参数。此外，重用冗余像素值进一步提高了网络参数效率。实验结果表明，所提出的方法优于基线方法的显着边际。代码可用于https://github.com/daniel03c1/eff_video_repruseentation。