基于学习的光流量估计已经与成本量的管道管道，具有用于流回归的卷曲，其固有地限于本地相关性，因此很难解决大型位移的长期挑战。为了缓解这一点，通过大量迭代细化产生一系列流动更新，实现最先进的方法，即筏，逐渐提高其预测的质量，实现了显着的性能，但减慢推理速度。为了实现高精度和效率的光学流量估计，我们通过将光学流作为全球匹配问题重新重新重新重新匹配，完全改造主导流回归管道。具体而言，我们提出了一个GMFlow框架，它由三个主要组件组成：用于功能增强的自定义变压器，全局特征匹配的相关和软邮件，以及用于流传播的自我注意层。此外，我们进一步介绍了一种改进步骤，该步骤在较高分辨率下重复使用GMFlow以进行残余流量预测。我们的新框架优于32次迭代RAFT在挑战的Sintel基准测试中的性能，同时仅使用一个细化并更快地运行，为高效和准确的光学流量估算提供了新的可能性。代码将在https://github.com/haofeixu/gmflow上使用。

We present a unified formulation and model for three motion and 3D perception tasks: optical flow, rectified stereo matching and unrectified stereo depth estimation from posed images. Unlike previous specialized architectures for each specific task, we formulate all three tasks as a unified dense correspondence matching problem, which can be solved with a single model by directly comparing feature similarities. Such a formulation calls for discriminative feature representations, which we achieve using a Transformer, in particular the cross-attention mechanism. We demonstrate that cross-attention enables integration of knowledge from another image via cross-view interactions, which greatly improves the quality of the extracted features. Our unified model naturally enables cross-task transfer since the model architecture and parameters are shared across tasks. We outperform RAFT with our unified model on the challenging Sintel dataset, and our final model that uses a few additional task-specific refinement steps outperforms or compares favorably to recent state-of-the-art methods on 10 popular flow, stereo and depth datasets, while being simpler and more efficient in terms of model design and inference speed.

我们介绍了光流变压器，被称为流动型，这是一种基于变压器的神经网络体系结构，用于学习光流。流动形式将图像对构建的4D成本量构成，将成本令牌编码为成本记忆，并在新颖的潜在空间中使用备用组变压器（AGT）层编码成本记忆，并通过反复的变压器解码器与动态位置成本查询来解码成本记忆。在SINTEL基准测试中，流动型在干净和最终通行证上达到1.144和2.183平均末端PONIT-ERROR（AEPE），从最佳发布的结果（1.388和2.47）降低了17.6％和11.6％的误差。此外，流程度还达到了强大的概括性能。在不接受Sintel的培训的情况下，FlowFormer在Sintel训练套装清洁通行证上达到了0.95 AEPE，优于最佳发布结果（1.29），提高了26.9％。

We introduce Recurrent All-Pairs Field Transforms (RAFT), a new deep network architecture for optical flow. RAFT extracts perpixel features, builds multi-scale 4D correlation volumes for all pairs of pixels, and iteratively updates a flow field through a recurrent unit that performs lookups on the correlation volumes. RAFT achieves stateof-the-art performance. On KITTI, RAFT achieves an F1-all error of 5.10%, a 16% error reduction from the best published result (6.10%). On Sintel (final pass), RAFT obtains an end-point-error of 2.855 pixels, a 30% error reduction from the best published result (4.098 pixels). In addition, RAFT has strong cross-dataset generalization as well as high efficiency in inference time, training speed, and parameter count. Code is available at https://github.com/princeton-vl/RAFT.

Optical flow, which computes the apparent motion from a pair of video frames, is a critical tool for scene motion estimation. Correlation volume is the central component of optical flow computational neural models. It estimates the pairwise matching costs between cross-frame features, and is then used to decode optical flow. However, traditional correlation volume is frequently noisy, outlier-prone, and sensitive to motion blur. We observe that, although the recent RAFT algorithm also adopts the traditional correlation volume, its additional context encoder provides semantically representative features to the flow decoder, implicitly compensating for the deficiency of the correlation volume. However, the benefits of this context encoder has been barely discussed or exploited. In this paper, we first investigate the functionality of RAFT's context encoder, then propose a new Context Guided Correlation Volume (CGCV) via gating and lifting schemes. CGCV can be universally integrated with RAFT-based flow computation methods for enhanced performance, especially effective in the presence of motion blur, de-focus blur and atmospheric effects. By incorporating the proposed CGCV with previous Global Motion Aggregation (GMA) method, at a minor cost of 0.5% extra parameters, the rank of GMA is lifted by 23 places on KITTI 2015 Leader Board, and 3 places on Sintel Leader Board. Moreover, at a similar model size, our correlation volume achieves competitive or superior performance to state of the art peer supervised models that employ Transformers or Graph Reasoning, as verified by extensive experiments.

We present a compact but effective CNN model for optical flow, called PWC-Net. PWC-Net has been designed according to simple and well-established principles: pyramidal processing, warping, and the use of a cost volume. Cast in a learnable feature pyramid, PWC-Net uses the current optical flow estimate to warp the CNN features of the second image. It then uses the warped features and features of the first image to construct a cost volume, which is processed by a CNN to estimate the optical flow. PWC-Net is 17 times smaller in size and easier to train than the recent FlowNet2 model. Moreover, it outperforms all published optical flow methods on the MPI Sintel final pass and KITTI 2015 benchmarks, running at about 35 fps on Sintel resolution (1024×436) images. Our models are available on https://github.com/NVlabs/PWC-Net.

在统一功能对应模型中建模稀疏和致密的图像匹配最近引起了研究的兴趣。但是，现有的努力主要集中于提高匹配的准确性，同时忽略其效率，这对于现实世界的应用至关重要。在本文中，我们提出了一种有效的结构，该结构以粗到精细的方式找到对应关系，从而显着提高了功能对应模型的效率。为了实现这一目标，多个变压器块是阶段范围连接的，以逐步完善共享的多尺度特征提取网络上的预测坐标。给定一对图像和任意查询坐标，所有对应关系均在单个进纸传球内预测。我们进一步提出了一种自适应查询聚类策略和基于不确定性的离群检测模块，以与提出的框架合作，以进行更快，更好的预测。对各种稀疏和密集的匹配任务进行的实验证明了我们方法在效率和有效性上对现有的最新作品的优势。

Optical flow estimation is a classical yet challenging task in computer vision. One of the essential factors in accurately predicting optical flow is to alleviate occlusions between frames. However, it is still a thorny problem for current top-performing optical flow estimation methods due to insufficient local evidence to model occluded areas. In this paper, we propose the Super Kernel Flow Network (SKFlow), a CNN architecture to ameliorate the impacts of occlusions on optical flow estimation. SKFlow benefits from the super kernels which bring enlarged receptive fields to complement the absent matching information and recover the occluded motions. We present efficient super kernel designs by utilizing conical connections and hybrid depth-wise convolutions. Extensive experiments demonstrate the effectiveness of SKFlow on multiple benchmarks, especially in the occluded areas. Without pre-trained backbones on ImageNet and with a modest increase in computation, SKFlow achieves compelling performance and ranks $\textbf{1st}$ among currently published methods on the Sintel benchmark. On the challenging Sintel clean and final passes (test), SKFlow surpasses the best-published result in the unmatched areas ($7.96$ and $12.50$) by $9.09\%$ and $7.92\%$. The code is available at \href{https://github.com/littlespray/SKFlow}{https://github.com/littlespray/SKFlow}.

Convolutional neural networks (CNNs) have recently been very successful in a variety of computer vision tasks, especially on those linked to recognition. Optical flow estimation has not been among the tasks where CNNs were successful. In this paper we construct appropriate CNNs which are capable of solving the optical flow estimation problem as a supervised learning task. We propose and compare two architectures: a generic architecture and another one including a layer that correlates feature vectors at different image locations.Since existing ground truth datasets are not sufficiently large to train a CNN, we generate a synthetic Flying Chairs dataset. We show that networks trained on this unrealistic data still generalize very well to existing datasets such as Sintel and KITTI, achieving competitive accuracy at frame rates of 5 to 10 fps.

我们提出了一个新颖的建筑，以实现密集的对应关系。当前的最新方法是基于变压器的方法，它们专注于功能描述符或成本量集合。但是，尽管关节聚集会通过提供一个人（即图像的结构或语义信息）或像素匹配的相似性来提高一个或另一个，但并非两者都聚集，但并非两者都汇总，尽管关节聚集会相互促进。在这项工作中，我们提出了一个基于变压器的新型网络，该网络以利用其互补信息的方式交织了两种形式的聚合。具体而言，我们设计了一个自我发项层，该层利用描述符来消除嘈杂的成本量，并且还利用成本量以促进准确匹配的方式汇总特征。随后的跨意思层执行进一步的聚合，该聚集在图像的描述上，并由早期层的聚合输出有助于。我们通过层次处理进一步提高了性能，在该处理中，更粗糙的聚合指导那些处于优质水平的过程。我们评估了所提出的方法对密集匹配任务的有效性，并在所有主要基准上实现最先进的性能。还提供了广泛的消融研究来验证我们的设计选择。

现有的光流估计器通常采用通常用于图像分类的网络体系结构作为提取人均功能的编码器。但是，由于任务之间的自然差异，用于图像分类的架构可能是最佳的流量估计。为了解决此问题，我们建议一种名为Falownas的神经体系结构搜索方法，以自动找到用于流估计任务的更好的编码器体系结构。我们首先设计一个合适的搜索空间，包括各种卷积运算符，并构建一个体重共享的超级网络，以有效评估候选体系结构。然后，为了更好地训练超级网络，我们提出了特征对齐蒸馏，该蒸馏利用训练有素的流量估计器来指导超级网络的训练。最后，利用资源约束的进化算法找到最佳体系结构（即子网络）。实验结果表明，从超级网络继承的权重的发现的结构达到了4.67 \％f1-able kitti上的误差，这是RAFT基线的8.4 \％降低，超过了先进的手工制作的型号GMA和AGFlow，同时降低模型的复杂性和延迟。源代码和训练有素的模型将在https://github.com/vdigpku/flownas中发布。

We present a novel method for local image feature matching. Instead of performing image feature detection, description, and matching sequentially, we propose to first establish pixel-wise dense matches at a coarse level and later refine the good matches at a fine level. In contrast to dense methods that use a cost volume to search correspondences, we use self and cross attention layers in Transformer to obtain feature descriptors that are conditioned on both images. The global receptive field provided by Transformer enables our method to produce dense matches in low-texture areas, where feature detectors usually struggle to produce repeatable interest points. The experiments on indoor and outdoor datasets show that LoFTR outperforms state-of-the-art methods by a large margin. LoFTR also ranks first on two public benchmarks of visual localization among the published methods. Code is available at our project page: https://zju3dv.github.io/loftr/.

光学流量估计是视频分析领域的一个重要而有挑战性问题。卷积神经网络的不同语义级别/层的特征可以提供不同粒度的信息。为了利用如此灵活和全面的信息，我们提出了一个半监督的特征金字塔形相关和残余重建网络（FPCR-Net），用于框架对的光学流量估计。它由两个主要模块组成：金字塔相关映射和剩余重建。金字塔相关映射模块利用全局/本地补丁的多尺度相关性来通过聚合不同尺度的特征来形成多级成本卷。剩余重建模块旨在重建每个阶段中更精细的光学流的子带高频残差。基于金字塔相关映射，我们进一步提出了相关 - 扭曲 - 归一化（CWN）模块，以有效地利用相关性依赖性。实验结果表明，该方案在针对竞争基线方法的平均终点误差（AEE）方面，实现了最先进的性能，改善了0.80,1.15和0.10 - Flownet2，LiteFlowNet和PWC-Net Sintel DataSet的最终通过。

Recent works have shown that optical flow can be learned by deep networks from unlabelled image pairs based on brightness constancy assumption and smoothness prior. Current approaches additionally impose an augmentation regularization term for continual self-supervision, which has been proved to be effective on difficult matching regions. However, this method also amplify the inevitable mismatch in unsupervised setting, blocking the learning process towards optimal solution. To break the dilemma, we propose a novel mutual distillation framework to transfer reliable knowledge back and forth between the teacher and student networks for alternate improvement. Concretely, taking estimation of off-the-shelf unsupervised approach as pseudo labels, our insight locates at defining a confidence selection mechanism to extract relative good matches, and then add diverse data augmentation for distilling adequate and reliable knowledge from teacher to student. Thanks to the decouple nature of our method, we can choose a stronger student architecture for sufficient learning. Finally, better student prediction is adopted to transfer knowledge back to the efficient teacher without additional costs in real deployment. Rather than formulating it as a supervised task, we find that introducing an extra unsupervised term for multi-target learning achieves best final results. Extensive experiments show that our approach, termed MDFlow, achieves state-of-the-art real-time accuracy and generalization ability on challenging benchmarks. Code is available at https://github.com/ltkong218/MDFlow.

场景流估计在场景之间提取点运动的场景估计正在成为许多计算机视觉任务的关键任务。但是，所有现有的估计方法仅利用单向特征，从而限制了准确性和通用性。本文使用双向流嵌入层提出了一种新颖的场景估计架构。所提出的双向层学习沿向前和向后方向的功能，从而增强了估计性能。此外，层次功能提取和翘曲可改善性能并减少计算开销。实验结果表明，拟议的架构通过在FlyingThings3D和Kitti基准测试中优于其他方法，从而实现了新的最新记录。代码可在https://github.com/cwc1260/biflow上找到。

在图像之间生成健壮和可靠的对应关系是多种应用程序的基本任务。为了在全球和局部粒度上捕获上下文，我们提出了Aspanformer，这是一种基于变压器的无探测器匹配器，建立在层次的注意力结构上，采用了一种新颖的注意操作，能够以自适应方式调整注意力跨度。为了实现这一目标，首先，在每个跨注意阶段都会回归流图，以定位搜索区域的中心。接下来，在中心周围生成一个采样网格，其大小不是根据固定的经验配置为固定的，而是根据与流图一起估计的像素不确定性的自适应计算。最后，在派生区域内的两个图像上计算注意力，称为注意跨度。通过这些方式，我们不仅能够维持长期依赖性，而且能够在高相关性的像素之间获得细粒度的注意，从而补偿基本位置和匹配任务中的零件平滑度。在广泛的评估基准上的最新准确性验证了我们方法的强匹配能力。

相邻帧的比对被认为是视频超分辨率（VSR）中的重要操作。高级VSR模型，包括最新的VSR变形金刚，通常配备精心设计的对齐模块。但是，自我注意机制的进步可能违反了这种常识。在本文中，我们重新考虑了对齐在VSR变压器中的作用，并进行了几种违反直觉的观察。我们的实验表明：（i）VSR变形金刚可以直接利用来自非对齐视频的多帧信息，并且（ii）现有的对齐方法有时对VSR变形金刚有害。这些观察结果表明，我们可以仅通过删除对齐模块并采用更大的注意力窗口来进一步提高VSR变压器的性能。然而，这样的设计将大大增加计算负担，无法处理大型动议。因此，我们提出了一种称为斑块对齐的新的，有效的对准方法，该方法将图像贴片而不是像素对齐。配备贴片对齐的VSR变形金刚可以在多个基准测试上证明最先进的性能。我们的工作提供了有关如何在VSR中使用多帧信息以及如何为不同网络/数据集选择对齐方法的宝贵见解。代码和模型将在https://github.com/xpixelgroup/rethinkvsralignment上发布。

通常将视频中的跟踪像素作为光流估计问题进行研究，其中每个像素都用位移向量描述，该位移向量将其定位在下一帧中。即使可以免费获得更广泛的时间上下文，但要考虑到这一点的事先努力仅在2框方法上产生了少量收益。在本文中，我们重新访问Sand and Teller的“粒子视频”方法，并将像素跟踪作为远程运动估计问题，其中每个像素都用轨迹描述，该轨迹将其定位在以后的多个帧中。我们使用该组件重新构建了这种经典方法，这些组件可以驱动流量和对象跟踪中最新的最新方法，例如密集的成本图，迭代优化和学习的外观更新。我们使用从现有的光流数据中挖掘出的远程Amodal点轨迹来训练我们的模型，并通过多帧的遮挡合成增强，这些轨迹会增强。我们在轨迹估计基准和关键点标签传播任务中测试我们的方法，并与最新的光流和功能跟踪方法进行比较。

尽管运动补偿大大提高了视频质量，但单独执行运动补偿和视频脱张需要大量的计算开销。本文提出了一个实时视频Deblurring框架，该框架由轻巧的多任务单元组成，该单元以有效的方式支持视频脱张和运动补偿。多任务单元是专门设计的，用于使用单个共享网络处理两个任务的大部分，并由多任务详细网络和简单的网络组成，用于消除和运动补偿。多任务单元最大程度地减少了将运动补偿纳入视频Deblurring的成本，并实现了实时脱毛。此外，通过堆叠多个多任务单元，我们的框架在成本和过度质量之间提供了灵活的控制。我们通过实验性地验证了方法的最先进的质量，与以前的方法相比，该方法的运行速度要快得多，并显示了实时的实时性能（在DVD数据集中测量了30.99db@30fps）。

无监督的对光流计算的深度学习取得了令人鼓舞的结果。大多数现有的基于深网的方法都依赖图像亮度一致性和局部平滑度约束来训练网络。他们的性能在发生重复纹理或遮挡的区域降低。在本文中，我们提出了深层的外两极流，这是一种无监督的光流方法，将全局几何约束结合到网络学习中。特别是，我们研究了多种方式在流量估计中强制执行外两极约束。为了减轻在可能存在多个动作的动态场景中遇到的“鸡肉和蛋”类型的问题，我们提出了一个低级别的约束以及对培训的订婚结合的约束。各种基准测试数据集的实验结果表明，与监督方法相比，我们的方法实现了竞争性能，并且优于最先进的无监督深度学习方法。