With the advent of Neural Style Transfer (NST), stylizing an image has become quite popular. A convenient way for extending stylization techniques to videos is by applying them on a per-frame basis. However, such per-frame application usually lacks temporal-consistency expressed by undesirable flickering artifacts. Most of the existing approaches for enforcing temporal-consistency suffers from one or more of the following drawbacks. They (1) are only suitable for a limited range of stylization techniques, (2) can only be applied in an offline fashion requiring the complete video as input, (3) cannot provide consistency for the task of stylization, or (4) do not provide interactive consistency-control. Note that existing consistent video-filtering approaches aim to completely remove flickering artifacts and thus do not respect any specific consistency-control aspect. For stylization tasks, however, consistency-control is an essential requirement where a certain amount of flickering can add to the artistic look and feel. Moreover, making this control interactive is paramount from a usability perspective. To achieve the above requirements, we propose an approach that can stylize video streams while providing interactive consistency-control. Apart from stylization, our approach also supports various other image processing filters. For achieving interactive performance, we develop a lite optical-flow network that operates at 80 Frames per second (FPS) on desktop systems with sufficient accuracy. We show that the final consistent video-output using our flow network is comparable to that being obtained using state-of-the-art optical-flow network. Further, we employ an adaptive combination of local and global consistent features and enable interactive selection between the two. By objective and subjective evaluation, we show that our method is superior to state-of-the-art approaches.

由于视频处理方法的稀缺性，图像处理操作通过独立处理每个框架来天真地扩展到视频域。这种无视视频处理中的时间连接通常会导致严重的时间不一致。解决这些不一致之处的最先进的技术取决于未经加工的视频的可用性来虹吸一致的视频动态，以恢复框架处理的视频的时间一致性。我们为这项任务提出了一个新颖的通用框架，该框架学会从不一致的视频中推断出一致的运动动力学，以减轻时间闪烁，同时保留时间相邻和相对较远的框架的感知质量。提出的框架在两个大规模数据集（戴维斯和videvo.net）上产生最新的结果，这些数据集以馈送方式处理众多图像处理任务进行处理。接受代码和训练有素的模型将在接受后发布。

Given two consecutive frames, video interpolation aims at generating intermediate frame(s) to form both spatially and temporally coherent video sequences. While most existing methods focus on single-frame interpolation, we propose an end-to-end convolutional neural network for variable-length multi-frame video interpolation, where the motion interpretation and occlusion reasoning are jointly modeled. We start by computing bi-directional optical flow between the input images using a U-Net architecture. These flows are then linearly combined at each time step to approximate the intermediate bi-directional optical flows. These approximate flows, however, only work well in locally smooth regions and produce artifacts around motion boundaries. To address this shortcoming, we employ another U-Net to refine the approximated flow and also predict soft visibility maps. Finally, the two input images are warped and linearly fused to form each intermediate frame. By applying the visibility maps to the warped images before fusion, we exclude the contribution of occluded pixels to the interpolated intermediate frame to avoid artifacts. Since none of our learned network parameters are time-dependent, our approach is able to produce as many intermediate frames as needed. To train our network, we use 1,132 240-fps video clips, containing 300K individual video frames. Experimental results on several datasets, predicting different numbers of interpolated frames, demonstrate that our approach performs consistently better than existing methods.

基于图像的艺术渲染可以使用算法图像过滤合成各种表达式。与基于深度学习的方法相反，这些基于启发式的过滤技术可以在高分辨率图像上运行，可以解释，并且可以根据各个设计方面进行参数化。但是，适应或扩展这些技术生产新样式通常是一项乏味且容易出错的任务，需要专家知识。我们提出了一个新的范式来减轻此问题：实现算法图像过滤技术作为可区分的操作，可以学习与某些参考样式一致的参数化。为此，我们提出了明智的，这是一个基于示例的图像处理系统，可以在公共框架内处理多种风格化技术，例如水彩，油或卡通风格。通过训练全局和本地滤波器参数化的参数预测网络，我们可以同时适应参考样式和图像内容，例如增强面部特征。我们的方法可以在样式转移框架中进行优化，也可以在用于图像到图像翻译的生成对流设置中学习。我们证明，共同训练XDOG滤波器和用于后处理的CNN可以与基于GAN的最新方法获得可比的结果。

Videos shot by laymen using hand-held cameras contain undesirable shaky motion. Estimating the global motion between successive frames, in a manner not influenced by moving objects, is central to many video stabilization techniques, but poses significant challenges. A large body of work uses 2D affine transformations or homography for the global motion. However, in this work, we introduce a more general representation scheme, which adapts any existing optical flow network to ignore the moving objects and obtain a spatially smooth approximation of the global motion between video frames. We achieve this by a knowledge distillation approach, where we first introduce a low pass filter module into the optical flow network to constrain the predicted optical flow to be spatially smooth. This becomes our student network, named as \textsc{GlobalFlowNet}. Then, using the original optical flow network as the teacher network, we train the student network using a robust loss function. Given a trained \textsc{GlobalFlowNet}, we stabilize videos using a two stage process. In the first stage, we correct the instability in affine parameters using a quadratic programming approach constrained by a user-specified cropping limit to control loss of field of view. In the second stage, we stabilize the video further by smoothing global motion parameters, expressed using a small number of discrete cosine transform coefficients. In extensive experiments on a variety of different videos, our technique outperforms state of the art techniques in terms of subjective quality and different quantitative measures of video stability. The source code is publicly available at \href{https://github.com/GlobalFlowNet/GlobalFlowNet}{https://github.com/GlobalFlowNet/GlobalFlowNet}

可以通过定期预测未来的框架以增强虚拟现实应用程序中的用户体验，从而解决了低计算设备上图形渲染高帧速率视频的挑战。这是通过时间视图合成（TVS）的问题来研究的，该问题的目标是预测给定上一个帧的视频的下一个帧以及上一个和下一个帧的头部姿势。在这项工作中，我们考虑了用户和对象正在移动的动态场景的电视。我们设计了一个将运动解散到用户和对象运动中的框架，以在预测下一帧的同时有效地使用可用的用户运动。我们通过隔离和估计过去框架的3D对象运动，然后推断它来预测对象的运动。我们使用多平面图像（MPI）作为场景的3D表示，并将对象运动作为MPI表示中相应点之间的3D位移建模。为了在估计运动时处理MPI中的稀疏性，我们将部分卷积和掩盖的相关层纳入了相应的点。然后将预测的对象运动与给定的用户或相机运动集成在一起，以生成下一帧。使用不合格的填充模块，我们合成由于相机和对象运动而发现的区域。我们为动态场景的电视开发了一个新的合成数据集，该数据集由800个以全高清分辨率组成的视频组成。我们通过数据集和MPI Sintel数据集上的实验表明我们的模型优于文献中的所有竞争方法。

快速移动受试者的运动模糊是摄影中的一个长期问题，由于收集效率有限，尤其是在弱光条件下，在手机上非常常见。尽管近年来我们目睹了图像脱毛的巨大进展，但大多数方法都需要显着的计算能力，并且在处理高分辨率照片的情况下具有严重的局部动作。为此，我们根据手机的双摄像头融合技术开发了一种新颖的面部脱毛系统。该系统检测到主题运动以动态启用参考摄像头，例如，最近在高级手机上通常可用的Ultrawide Angle摄像机，并捕获带有更快快门设置的辅助照片。虽然主镜头是低噪音但模糊的，但参考镜头却很锋利，但嘈杂。我们学习ML模型，以对齐和融合这两张镜头，并在没有运动模糊的情况下输出清晰的照片。我们的算法在Google Pixel 6上有效运行，每次拍摄需要463毫秒的开销。我们的实验证明了系统对替代单片，多帧，面部特异性和视频脱张算法以及商业产品的优势和鲁棒性。据我们所知，我们的工作是第一个用于面部运动脱毛的移动解决方案，在各种运动和照明条件下，在数千个图像中可靠地工作。

视频帧插值（VFI）目前是一个非常活跃的研究主题，具有跨越计算机视觉，后期生产和视频编码的应用程序。 VFI可能非常具有挑战性，特别是在含有大型运动，闭塞或动态纹理的序列中，现有方法未能提供感知鲁棒的插值性能。在这种情况下，我们基于时空多流量架构介绍了一种基于深度学习的VFI方法ST-MFNET。 ST-MFNET采用新的多尺度多流量预测器来估计多对一的中间流动，它们与传统的一对一光流组合以捕获大型和复杂的运动。为了增强各种纹理的插值性能，还用于在扩展时间窗口上模拟内容动态的3D CNN。此外，ST-MFNET已经在ST-GaN框架内培训，该框架最初是为纹理合成而开发的，目的是进一步提高感知插值质量。我们的方法已被全面评估 - 与十四个最先进的VFI算法相比 - 清楚地展示了ST-MFNET在各种和代表性测试数据集上始终如一地优于这些基准，在PSNR中具有显着的收益，用于案件在PSNR中高达1.09dB包括大型运动和动态纹理。项目页面：https://danielism97.github.io/st-mfnet。

We pose video object segmentation as spectral graph clustering in space and time, with one graph node for each pixel and edges forming local space-time neighborhoods. We claim that the strongest cluster in this video graph represents the salient object. We start by introducing a novel and efficient method based on 3D filtering for approximating the spectral solution, as the principal eigenvector of the graph's adjacency matrix, without explicitly building the matrix. This key property allows us to have a fast parallel implementation on GPU, orders of magnitude faster than classical approaches for computing the eigenvector. Our motivation for a spectral space-time clustering approach, unique in video semantic segmentation literature, is that such clustering is dedicated to preserving object consistency over time, which we evaluate using our novel segmentation consistency measure. Further on, we show how to efficiently learn the solution over multiple input feature channels. Finally, we extend the formulation of our approach beyond the segmentation task, into the realm of object tracking. In extensive experiments we show significant improvements over top methods, as well as over powerful ensembles that combine them, achieving state-of-the-art on multiple benchmarks, both for tracking and segmentation.

在许多重要的科学和工程应用中发现了卷数据。渲染此数据以高质量和交互速率为苛刻的应用程序（例如虚拟现实）的可视化化，即使使用专业级硬件也无法实现。我们介绍了Fovolnet - 一种可显着提高数量数据可视化的性能的方法。我们开发了一种具有成本效益的渲染管道，该管道稀疏地对焦点进行了量度，并使用深层神经网络重建了全帧。 FOVEATED渲染是一种优先考虑用户焦点渲染计算的技术。这种方法利用人类视觉系统的属性，从而在用户视野的外围呈现数据时节省了计算资源。我们的重建网络结合了直接和内核预测方法，以产生快速，稳定和感知令人信服的输出。凭借纤细的设计和量化的使用，我们的方法在端到端框架时间和视觉质量中都优于最先进的神经重建技术。我们对系统的渲染性能，推理速度和感知属性进行了广泛的评估，并提供了与竞争神经图像重建技术的比较。我们的测试结果表明，Fovolnet始终在保持感知质量的同时，在传统渲染上节省了大量时间。

动态对象对机器人对环境的看法产生了重大影响，这降低了本地化和映射等基本任务的性能。在这项工作中，我们通过在由动态对象封闭的区域中合成合理的颜色，纹理和几何形状来解决这个问题。我们提出了一种新的几何感知Dynafill架构，其遵循粗略拓扑，并将我们所通用的经常性反馈机制结合到自适应地融合来自之前的时间步来的信息。我们使用对抗性培训来优化架构，以综合精细的现实纹理，使其能够以空间和时间相干的方式在线在线遮挡地区的幻觉和深度结构，而不依赖于未来的帧信息。将我们的待遇问题作为图像到图像到图像的翻译任务，我们的模型还纠正了与场景中动态对象的存在相关的区域，例如阴影或反射。我们引入了具有RGB-D图像，语义分段标签，摄像机的大型高估数据集，以及遮挡区域的地面RGB-D信息。广泛的定量和定性评估表明，即使在挑战天气条件下，我们的方法也能实现最先进的性能。此外，我们使用综合图像显示基于检索的视觉本地化的结果，该图像证明了我们方法的效用。

人类活动的上采样视频是一个有趣但具有挑战性的任务，具有许多潜在的应用，从游戏到娱乐和运动广播。在该设置中合成视频帧的主要困难源于人类运动的高度复杂和非线性性质和身体的复杂外观和质地。我们建议在运动引导框架上采样框架中解决这些问题，该框架上采样框架能够产生现实的人类运动和外观。通过利用大规模运动捕获数据集（Amass）培训新颖运动模型，推动帧之间的非线性骨架运动。然后，神经渲染管线使用高帧速率姿态预测以产生全帧输出，考虑姿势和背景一致性。我们的管道只需要低帧速率视频和未配对的人类运动数据，但不需要高帧率视频进行培训。此外，我们贡献了第一个评估数据集，该数据集包括用于此任务的人类活动的高质量和高帧速率视频。与最先进的视频插值技术相比，我们的方法在具有更高质量和精度的帧之间产生的帧，这是通过最先进的导致像素级，分布度量和比较用户评估的结果。我们的代码和收集的数据集可以在https://git.io/render-in-botween中找到。

视频框架插值〜（VFI）算法近年来由于数据驱动算法及其实现的前所未有的进展，近年来有了显着改善。最近的研究引入了高级运动估计或新颖的扭曲方法，以解决具有挑战性的VFI方案。但是，没有发表的VFI作品认为插值误差（IE）的空间不均匀特征。这项工作引入了这样的解决方案。通过密切检查光流与IE之间的相关性，本文提出了新的错误预测指标，该指标将中间框架分为与不同IE水平相对应的不同区域。它基于IE驱动的分割，并通过使用新颖的错误控制损耗函数，引入了一组空间自适应插值单元的合奏，该单元逐步处理并集成了分段区域。这种空间合奏会产生有效且具有诱人的VFI解决方案。对流行视频插值基准测试的广泛实验表明，所提出的解决方案在当前兴趣的应用中优于当前最新（SOTA）。

The ubiquity of camera-embedded devices and the advances in deep learning have stimulated various intelligent mobile video applications. These applications often demand on-device processing of video streams to deliver real-time, high-quality services for privacy and robustness concerns. However, the performance of these applications is constrained by the raw video streams, which tend to be taken with small-aperture cameras of ubiquitous mobile platforms in dim light. Despite extensive low-light video enhancement solutions, they are unfit for deployment to mobile devices due to their complex models and and ignorance of system dynamics like energy budgets. In this paper, we propose AdaEnlight, an energy-aware low-light video stream enhancement system on mobile devices. It achieves real-time video enhancement with competitive visual quality while allowing runtime behavior adaptation to the platform-imposed dynamic energy budgets. We report extensive experiments on diverse datasets, scenarios, and platforms and demonstrate the superiority of AdaEnlight compared with state-of-the-art low-light image and video enhancement solutions.

Motion blur from camera shake is a major problem in videos captured by hand-held devices. Unlike single-image deblurring, video-based approaches can take advantage of the abundant information that exists across neighboring frames. As a result the best performing methods rely on the alignment of nearby frames. However, aligning images is a computationally expensive and fragile procedure, and methods that aggregate information must therefore be able to identify which regions have been accurately aligned and which have not, a task that requires high level scene understanding. In this work, we introduce a deep learning solution to video deblurring, where a CNN is trained end-toend to learn how to accumulate information across frames. To train this network, we collected a dataset of real videos recorded with a high frame rate camera, which we use to generate synthetic motion blur for supervision. We show that the features learned from this dataset extend to deblurring motion blur that arises due to camera shake in a wide range of videos, and compare the quality of results to a number of other baselines 1 .

半监控视频对象分段（VOS）旨在在视频序列中分段一些移动对象，其中通过注释第一帧来指定这些对象。已经考虑了许多现有的半监督VOS方法以提高分割精度的光学流程。然而，由于光学流量估计的高复杂性，光流基的半监控VOS方法不能实时运行。在该研究中提出了由特征提取网络（F），外观网络（A），运动网络（A）和集成网络（I）组成的FAMINET，以解决上述问题。外观网络基于对象的静态外观输出初始分割结果。运动网络通过很少的参数估计光学流量，这些参数通过在线记忆算法快速优化，该算法被称为松弛最陡血迹。集成网络使用光流来改进初始分割结果。广泛的实验表明，FAMINET在DAVIS和YOUTUBE-VOS基准上表现出其他最先进的半监督VOS方法，并且它在准确性和效率之间实现了良好的权衡。我们的代码可在https://github.com/liuziyang123/faminet获得。

着色是一个计算机辅助过程，旨在为灰色图像或视频赋予色彩。它可用于增强黑白图像，包括黑白照片，老式电影和科学成像结果。相反，不着色是将颜色图像或视频转换为灰度。灰度图像或视频是指没有颜色信息的亮度信息的图像或视频。它是一些下游图像处理应用程序的基础，例如模式识别，图像分割和图像增强。与图像脱色不同，视频脱色不仅应考虑每个视频框架中的图像对比度保存，而且还应尊重视频框架之间的时间和空间一致性。研究人员致力于通过平衡时空的一致性和算法效率来开发脱色方法。随着数码相机和手机的流行，研究人员越来越关注图像和视频着色和脱色。本文概述了过去二十年来图像和视频着色和脱色方法的进度。

The FlowNet demonstrated that optical flow estimation can be cast as a learning problem. However, the state of the art with regard to the quality of the flow has still been defined by traditional methods. Particularly on small displacements and real-world data, FlowNet cannot compete with variational methods. In this paper, we advance the concept of end-to-end learning of optical flow and make it work really well. The large improvements in quality and speed are caused by three major contributions: first, we focus on the training data and show that the schedule of presenting data during training is very important. Second, we develop a stacked architecture that includes warping of the second image with intermediate optical flow. Third, we elaborate on small displacements by introducing a subnetwork specializing on small motions. FlowNet 2.0 is only marginally slower than the original FlowNet but decreases the estimation error by more than 50%. It performs on par with state-of-the-art methods, while running at interactive frame rates. Moreover, we present faster variants that allow optical flow computation at up to 140fps with accuracy matching the original FlowNet.

视频是一种流行的媒体形式，其中在线视频流最近聚集了很多人气。在这项工作中，我们提出了一种新颖的实时视频稳定方法 - 将摇晃视频转换为稳定的视频，仿佛它实时通过万向节稳定。我们的框架是以自我监督的方式进行培训，不需要使用特殊硬件设置（即，在立体声钻机或附加运动传感器上的两个摄像机）捕获的数据。我们的框架包括在给定帧之间的转换估计器，用于全局稳定性调整，然后通过空间平滑的光学流动的场景视差减少模块，以进一步稳定。然后，保证金修整模块填充稳定期间创建的缺失的边缘区域，以减少裁剪后的数量。这些顺序步骤将失真和边距减少到最小，同时增强稳定性。因此，我们的方法优于最先进的实时视频稳定方法以及需要相机轨迹优化的离线方法。无论分辨率（例如，480p或1080p），我们的方法程序大约需要41 fps的24.3 ms。

我们提出了一种用于视频帧插值（VFI）的实时中流估计算法。许多最近的基于流的VFI方法首先估计双向光学流，然后缩放并将它们倒转到近似中间流动，导致运动边界上的伪像。RIFE使用名为IFNET的神经网络，可以直接估计中间流量从粗细流，速度更好。我们设计了一种用于训练中间流动模型的特权蒸馏方案，这导致了大的性能改善。Rife不依赖于预先训练的光流模型，可以支持任意时间的帧插值。实验表明，普里埃雷在若干公共基准上实现了最先进的表现。\ url {https://github.com/hzwer/arxiv2020-rife}。