与无监督培训相比，对光流预测因子的监督培训通常会产生更好的准确性。但是，改进的性能通常以较高的注释成本。半监督的培训与注释成本相比，准确性的准确性。我们使用一种简单而有效的半监督训练方法来表明，即使一小部分标签也可以通过无监督的训练来提高流量准确性。此外，我们提出了基于简单启发式方法的主动学习方法，以进一步减少实现相同目标准确性所需的标签数量。我们对合成和真实光流数据集的实验表明，我们的半监督网络通常需要大约50％的标签才能达到接近全标签的精度，而在Sintel上有效学习只有20％左右。我们还分析并展示了有关可能影响主动学习绩效的因素的见解。代码可在https://github.com/duke-vision/optical-flow-active-learning-release上找到。

光流CNNS的训练管道由合成数据集的预处理阶段组成，然后在目标数据集上进行微调阶段。但是，从目标视频中获得地面真理需要巨大的努力。本文提出了一种实用的微调方法，将预处理的模型调整到没有地面真相流的目标数据集中，但尚未进行广泛探讨。具体而言，我们为自我划分的流程主管提出了一个流程主管，其中包括参数分离和学生量连接。该设计的目的是稳定的收敛性和更好的准确性，而在微调任务上是不稳定的传统自我实施方法。实验结果表明，与半监督学习的不同自学方法相比，我们方法的有效性。此外，我们通过利用其他未标记的数据集来实现对Sintel和Kitti基准测试的最先进的光流模型的有意义的改进。代码可在https://github.com/iwbn/flow-supervisor上找到。

Recently, AutoFlow has shown promising results on learning a training set for optical flow, but requires ground truth labels in the target domain to compute its search metric. Observing a strong correlation between the ground truth search metric and self-supervised losses, we introduce self-supervised AutoFlow to handle real-world videos without ground truth labels. Using self-supervised loss as the search metric, our self-supervised AutoFlow performs on par with AutoFlow on Sintel and KITTI where ground truth is available, and performs better on the real-world DAVIS dataset. We further explore using self-supervised AutoFlow in the (semi-)supervised setting and obtain competitive results against the state of the art.

Recent works have shown that optical flow can be learned by deep networks from unlabelled image pairs based on brightness constancy assumption and smoothness prior. Current approaches additionally impose an augmentation regularization term for continual self-supervision, which has been proved to be effective on difficult matching regions. However, this method also amplify the inevitable mismatch in unsupervised setting, blocking the learning process towards optimal solution. To break the dilemma, we propose a novel mutual distillation framework to transfer reliable knowledge back and forth between the teacher and student networks for alternate improvement. Concretely, taking estimation of off-the-shelf unsupervised approach as pseudo labels, our insight locates at defining a confidence selection mechanism to extract relative good matches, and then add diverse data augmentation for distilling adequate and reliable knowledge from teacher to student. Thanks to the decouple nature of our method, we can choose a stronger student architecture for sufficient learning. Finally, better student prediction is adopted to transfer knowledge back to the efficient teacher without additional costs in real deployment. Rather than formulating it as a supervised task, we find that introducing an extra unsupervised term for multi-target learning achieves best final results. Extensive experiments show that our approach, termed MDFlow, achieves state-of-the-art real-time accuracy and generalization ability on challenging benchmarks. Code is available at https://github.com/ltkong218/MDFlow.

Synthetic datasets are often used to pretrain end-to-end optical flow networks, due to the lack of a large amount of labeled, real-scene data. But major drops in accuracy occur when moving from synthetic to real scenes. How do we better transfer the knowledge learned from synthetic to real domains? To this end, we propose CLIP-FLow, a semi-supervised iterative pseudo-labeling framework to transfer the pretraining knowledge to the target real domain. We leverage large-scale, unlabeled real data to facilitate transfer learning with the supervision of iteratively updated pseudo-ground truth labels, bridging the domain gap between the synthetic and the real. In addition, we propose a contrastive flow loss on reference features and the warped features by pseudo ground truth flows, to further boost the accurate matching and dampen the mismatching due to motion, occlusion, or noisy pseudo labels. We adopt RAFT as the backbone and obtain an F1-all error of 4.11%, i.e. a 19% error reduction from RAFT (5.10%) and ranking 2$^{nd}$ place at submission on the KITTI 2015 benchmark. Our framework can also be extended to other models, e.g. CRAFT, reducing the F1-all error from 4.79% to 4.66% on KITTI 2015 benchmark.

We present a compact but effective CNN model for optical flow, called PWC-Net. PWC-Net has been designed according to simple and well-established principles: pyramidal processing, warping, and the use of a cost volume. Cast in a learnable feature pyramid, PWC-Net uses the current optical flow estimate to warp the CNN features of the second image. It then uses the warped features and features of the first image to construct a cost volume, which is processed by a CNN to estimate the optical flow. PWC-Net is 17 times smaller in size and easier to train than the recent FlowNet2 model. Moreover, it outperforms all published optical flow methods on the MPI Sintel final pass and KITTI 2015 benchmarks, running at about 35 fps on Sintel resolution (1024×436) images. Our models are available on https://github.com/NVlabs/PWC-Net.

无监督的对光流计算的深度学习取得了令人鼓舞的结果。大多数现有的基于深网的方法都依赖图像亮度一致性和局部平滑度约束来训练网络。他们的性能在发生重复纹理或遮挡的区域降低。在本文中，我们提出了深层的外两极流，这是一种无监督的光流方法，将全局几何约束结合到网络学习中。特别是，我们研究了多种方式在流量估计中强制执行外两极约束。为了减轻在可能存在多个动作的动态场景中遇到的“鸡肉和蛋”类型的问题，我们提出了一个低级别的约束以及对培训的订婚结合的约束。各种基准测试数据集的实验结果表明，与监督方法相比，我们的方法实现了竞争性能，并且优于最先进的无监督深度学习方法。

Recent work has shown that optical flow estimation can be formulated as a supervised learning task and can be successfully solved with convolutional networks. Training of the so-called FlowNet was enabled by a large synthetically generated dataset. The present paper extends the concept of optical flow estimation via convolutional networks to disparity and scene flow estimation. To this end, we propose three synthetic stereo video datasets with sufficient realism, variation, and size to successfully train large networks. Our datasets are the first large-scale datasets to enable training and evaluating scene flow methods. Besides the datasets, we present a convolutional network for real-time disparity estimation that provides state-of-the-art results. By combining a flow and disparity estimation network and training it jointly, we demonstrate the first scene flow estimation with a convolutional network.

从视频中获得地面真相标签很具有挑战性，因为在像素流标签的手动注释非常昂贵且费力。此外，现有的方法试图将合成数据集的训练模型调整到真实的视频中，该视频不可避免地遭受了域差异并阻碍了现实世界应用程序的性能。为了解决这些问题，我们提出了RealFlow，这是一个基于期望最大化的框架，可以直接从任何未标记的现实视频中创建大规模的光流数据集。具体而言，我们首先估计一对视频帧之间的光流，然后根据预测流从该对中合成新图像。因此，新图像对及其相应的流可以被视为新的训练集。此外，我们设计了一种逼真的图像对渲染（RIPR）模块，该模块采用软磁性裂口和双向孔填充技术来减轻图像合成的伪像。在E-Step中，RIPR呈现新图像以创建大量培训数据。在M-Step中，我们利用生成的训练数据来训练光流网络，该数据可用于估计下一个E步骤中的光流。在迭代学习步骤中，流网络的能力逐渐提高，流量的准确性以及合成数据集的质量也是如此。实验结果表明，REALFLOW的表现优于先前的数据集生成方法。此外，基于生成的数据集，我们的方法与受监督和无监督的光流方法相比，在两个标准基准测试方面达到了最先进的性能。我们的代码和数据集可从https://github.com/megvii-research/realflow获得

Convolutional neural networks (CNNs) have recently been very successful in a variety of computer vision tasks, especially on those linked to recognition. Optical flow estimation has not been among the tasks where CNNs were successful. In this paper we construct appropriate CNNs which are capable of solving the optical flow estimation problem as a supervised learning task. We propose and compare two architectures: a generic architecture and another one including a layer that correlates feature vectors at different image locations.Since existing ground truth datasets are not sufficiently large to train a CNN, we generate a synthetic Flying Chairs dataset. We show that networks trained on this unrealistic data still generalize very well to existing datasets such as Sintel and KITTI, achieving competitive accuracy at frame rates of 5 to 10 fps.

As an important data selection schema, active learning emerges as the essential component when iterating an Artificial Intelligence (AI) model. It becomes even more critical given the dominance of deep neural network based models, which are composed of a large number of parameters and data hungry, in application. Despite its indispensable role for developing AI models, research on active learning is not as intensive as other research directions. In this paper, we present a review of active learning through deep active learning approaches from the following perspectives: 1) technical advancements in active learning, 2) applications of active learning in computer vision, 3) industrial systems leveraging or with potential to leverage active learning for data iteration, 4) current limitations and future research directions. We expect this paper to clarify the significance of active learning in a modern AI model manufacturing process and to bring additional research attention to active learning. By addressing data automation challenges and coping with automated machine learning systems, active learning will facilitate democratization of AI technologies by boosting model production at scale.

光流估计的最新方法取决于深度学习，这需要复杂的顺序训练方案才能在现实世界中达到最佳性能。在这项工作中，我们介绍了组合深网，该网络明确利用了传统方法中使用的亮度恒定（BC）模型。由于卑诗省是在几种情况下违反的一个近似物理模型，因此我们建议训练一个与数据驱动网络相辅相成的物理约束网络。我们在物理先验和数据驱动的补体之间引入了独特而有意义的流动分解，包括对BC模型的不确定性量化。我们得出了一个联合培训计划，用于学习分解的不同组成部分，以确保在受监督的情况下，但在半监督的环境中进行最佳合作。实验表明，组合可以改善对最先进的监督网络的性能，例如木筏在几个基准测试中达到最先进的结果。我们强调组合如何利用BC模型并适应其局限性。最后，我们表明我们的半监督方法可以显着简化训练程序。

在本文中，通过以自我监督的方式将基于几何的方法纳入深度学习架构来实现强大的视觉测量（VO）的基本问题。通常，基于纯几何的算法与特征点提取和匹配中的深度学习不那么稳健，但由于其成熟的几何理论，在自我运动估计中表现良好。在这项工作中，首先提出了一种新颖的光学流量网络（PANET）内置于位置感知机构。然后，提出了一种在没有典型网络的情况下共同估计深度，光学流动和自我运动来学习自我运动的新系统。所提出的系统的关键组件是一种改进的束调节模块，其包含多个采样，初始化的自我运动，动态阻尼因子调整和Jacobi矩阵加权。另外，新颖的相对光度损耗函数先进以提高深度估计精度。该实验表明，所提出的系统在基于基于基于基于基于基于基于基于学习的基于学习的方法之间的深度，流量和VO估计方面不仅优于其他最先进的方法，而且与几何形状相比，也显着提高了鲁棒性 - 基于，基于学习和混合VO系统。进一步的实验表明，我们的模型在挑战室内（TMU-RGBD）和室外（KAIST）场景中实现了出色的泛化能力和性能。

时间动作定位（TAL）旨在预测未修剪视频（即开始和结束时间）中动作实例的动作类别和时间边界。通常在大多数现有作品中都采用了完全监督的解决方案，并被证明是有效的。这些解决方案中的实际瓶颈之一是所需的大量标记培训数据。为了降低昂贵的人类标签成本，本文着重于很少调查但实用的任务，称为半监督TAL，并提出了一种有效的主动学习方法，名为Al-Stal。我们利用四个步骤来积极选择具有很高信息性的视频样本，并培训本地化模型，名为\ emph {火车，查询，注释，附加}。考虑定位模型的不确定性的两个评分函数配备了ALSTAL，从而促进了视频样本等级和选择。一个人将预测标签分布的熵作为不确定性的度量，称为时间提案熵（TPE）。另一个引入了基于相邻行动建议之间的共同信息的新指标，并评估视频样本的信息性，称为时间上下文不一致（TCI）。为了验证拟议方法的有效性，我们在两个基准数据集Thumos'14和ActivityNet 1.3上进行了广泛的实验。实验结果表明，与完全监督的学习相比，AL-Stal的表现优于现有竞争对手，并实现令人满意的表现。

We propose LiDAL, a novel active learning method for 3D LiDAR semantic segmentation by exploiting inter-frame uncertainty among LiDAR frames. Our core idea is that a well-trained model should generate robust results irrespective of viewpoints for scene scanning and thus the inconsistencies in model predictions across frames provide a very reliable measure of uncertainty for active sample selection. To implement this uncertainty measure, we introduce new inter-frame divergence and entropy formulations, which serve as the metrics for active selection. Moreover, we demonstrate additional performance gains by predicting and incorporating pseudo-labels, which are also selected using the proposed inter-frame uncertainty measure. Experimental results validate the effectiveness of LiDAL: we achieve 95% of the performance of fully supervised learning with less than 5% of annotations on the SemanticKITTI and nuScenes datasets, outperforming state-of-the-art active learning methods. Code release: https://github.com/hzykent/LiDAL.

The quantitative evaluation of optical flow algorithms by Barron et al. (1994) led to significant advances in performance. The challenges for optical flow algorithms today go beyond the datasets and evaluation methods proposed in that paper. Instead, they center on problems associated with complex natural scenes, including nonrigid motion, real sensor noise, and motion discontinuities. We propose a new set of benchmarks and evaluation methods for the next generation of optical flow algorithms. To that end, we contribute four types of data to test different aspects of optical flow algorithms: (1) sequences with nonrigid motion where the ground-truth flow is determined by A preliminary version of this paper appeared in the IEEE International Conference on Computer Vision (Baker et al. 2007).

自我监督的单眼深度估计使机器人能够从原始视频流中学习3D感知。假设世界主要是静态的，这种可扩展的方法利用了投射的几何形状和自我运动来通过视图综合学习。在自主驾驶和人类机器人相互作用中常见的动态场景违反了这一假设。因此，它们需要明确建模动态对象，例如通过估计像素3D运动，即场景流。但是，同时对深度和场景流的自我监督学习是不适合的，因为有许多无限的组合导致相同的3D点。在本文中，我们提出了一种草稿，这是一种通过将合成数据与几何自学意识相结合的新方法，能够共同学习深度，光流和场景流。在木筏架构的基础上，我们将光流作为中间任务，以通过三角剖分来引导深度和场景流量学习。我们的算法还利用任务之间的时间和几何一致性损失来改善多任务学习。我们的草案在标准Kitti基准的自我监督的单眼环境中，同时在所有三个任务中建立了新的最新技术状态。项目页面：https：//sites.google.com/tri.global/draft。

在本文中，我们提出了USEGSCENE，该框架用于使用卷积神经网络对立体声相机图像的深度，光流和自我感动的无监督学习。我们的框架利用语义信息来改善深度和光流图的正则化，多模式融合和遮挡填充考虑动态刚性对象运动作为独立的SE（3）转换。此外，我们与纯照相匹配匹配互补，我们提出了连续图像之间语义特征，像素类别和对象实例边界的匹配。与以前的方法相反，我们提出了一个网络体系结构，该网络体系结构可以使用共享编码器共同预测所有输出，并允许在任务域上传递信息，例如，光流的预测可以从深度的预测中受益。此外，我们明确地了解网络内部的深度和光流遮挡图，这些图被利用，以改善这些区域的预测。我们在流行的Kitti数据集上介绍了结果，并表明我们的方法以大幅度的优于其他方法。

我们解决了视频动作识别的数据增强问题。视频中的标准增强策略是手工设计的，并随机对可能的增强数据点的空间进行采样，而不知道哪个增强点会更好，或者是通过启发式方法会更好。我们建议学习是什么使良好的视频供行动识别，并仅选择高质量的样本进行增强。特别是，我们选择前景和背景视频的视频合成作为数据增强过程，从而导致各种新样本。我们了解了哪对视频要增加，而无需实际综合它们。这降低了可能的增强空间，这具有两个优势：它节省了计算成本并提高了最终训练的分类器的准确性，因为增强对的质量高于平均水平。我们在整个训练环境中介绍了实验结果：几乎没有射击，半监督和完全监督。我们观察到所有这些都对动力学，UCF101，HMDB51的基准进行了一致的改进，并在设置上实现了有限数据的新最新设置。在半监督环境中，我们看到高达8.6％的改善。

Jitendra Malik once said, "Supervision is the opium of the AI researcher". Most deep learning techniques heavily rely on extreme amounts of human labels to work effectively. In today's world, the rate of data creation greatly surpasses the rate of data annotation. Full reliance on human annotations is just a temporary means to solve current closed problems in AI. In reality, only a tiny fraction of data is annotated. Annotation Efficient Learning (AEL) is a study of algorithms to train models effectively with fewer annotations. To thrive in AEL environments, we need deep learning techniques that rely less on manual annotations (e.g., image, bounding-box, and per-pixel labels), but learn useful information from unlabeled data. In this thesis, we explore five different techniques for handling AEL.