A recent strand of work in view synthesis uses deep learning to generate multiplane images-a camera-centric, layered 3D representation-given two or more input images at known viewpoints. We apply this representation to singleview view synthesis, a problem which is more challenging but has potentially much wider application. Our method learns to predict a multiplane image directly from a single image input, and we introduce scale-invariant view synthesis for supervision, enabling us to train on online video. We show this approach is applicable to several different datasets, that it additionally generates reasonable depth maps, and that it learns to fill in content behind the edges of foreground objects in background layers.Project page at https://single-view-mpi.github.io/.

新颖的视图合成（NVS）和视频预测（VP）通常被视为计算机视觉中的不相交任务。但是，它们都可以看作是观察空间时代世界的方法：NVS的目的是从新的角度综合一个场景，而副总裁则旨在从新的时间点观看场景。这两个任务提供了互补的信号以获得场景表示形式，因为观点从空间观察中变化为深度的变化，并且时间观察为相机和单个对象的运动提供了信息。受这些观察的启发，我们建议研究时空（背心）中视频外推的问题。我们提出了一个模型，该模型利用了两项任务的自学和互补线索，而现有方法只能解决其中之一。实验表明，我们的方法比室内和室外现实世界数据集上的几种最先进的NVS和VP方法更好地实现了性能。

我们介绍了Fadiv-Syn，一种快速深入的新型观点合成方法。相关方法通常受到它们的深度估计阶段的限制，其中不正确的深度预测可能导致大的投影误差。为避免此问题，我们将输入图像有效地将输入图像呈现为目标帧，以为一系列假定的深度平面。得到的平面扫描量（PSV）直接进入我们的网络，首先以自我监督的方式估计软PSV掩模，然后直接产生新颖的输出视图。因此，我们侧行显式深度估计。这提高了透明，反光，薄，特色场景部件上的效率和性能。 Fadiv-syn可以在大规模Realestate10K数据集上执行插值和外推任务，优于最先进的外推方法。与可比方法相比，它由于其轻量级架构而实现了实时性能。我们彻底评估消融，例如去除软掩蔽网络，从更少的示例中培训以及更高的分辨率和更强深度离散化的概括。

我们介绍了与给定单个图像的任意长相机轨迹相对应的长期视图的新面积视图的问题。这是一个具有挑战性的问题，远远超出了当前视图合成方法的能力，这在提出大型摄像机运动时快速退化。用于视频生成的方法也具有有限的生产长序列的能力，并且通常不适用于场景几何形状。我们采用混合方法，它以迭代`\ emph {render}，\ emph {refine}，\ emph {重复}'框架集成了几何和图像合成，允许在数百帧之后覆盖大距离的远程生成。我们的方法可以从一组单目的视频序列训练。我们提出了一个沿海场景的空中镜头数据集，并比较了我们最近的观看综合和有条件的视频生成基线的方法，表明它可以在与现有方法相比，在大型相机轨迹上产生更长的时间范围。项目页面https://infinite-nature.github.io/。

Fast and easy handheld capture with guideline: closest object moves at most D pixels between views Promote sampled views to local light field via layered scene representation Blend neighboring local light fields to render novel views

We explore the problem of view synthesis from a narrow baseline pair of images, and focus on generating highquality view extrapolations with plausible disocclusions. Our method builds upon prior work in predicting a multiplane image (MPI), which represents scene content as a set of RGBα planes within a reference view frustum and renders novel views by projecting this content into the target viewpoints. We present a theoretical analysis showing how the range of views that can be rendered from an MPI increases linearly with the MPI disparity sampling frequency, as well as a novel MPI prediction procedure that theoretically enables view extrapolations of up to 4× the lateral viewpoint movement allowed by prior work. Our method ameliorates two specific issues that limit the range of views renderable by prior methods: 1) We expand the range of novel views that can be rendered without depth discretization artifacts by using a 3D convolutional network architecture along with a randomized-resolution training procedure to allow our model to predict MPIs with increased disparity sampling frequency. 2) We reduce the repeated texture artifacts seen in disocclusions by enforcing a constraint that the appearance of hidden content at any depth must be drawn from visible content at or behind that depth.

可以通过定期预测未来的框架以增强虚拟现实应用程序中的用户体验，从而解决了低计算设备上图形渲染高帧速率视频的挑战。这是通过时间视图合成（TVS）的问题来研究的，该问题的目标是预测给定上一个帧的视频的下一个帧以及上一个和下一个帧的头部姿势。在这项工作中，我们考虑了用户和对象正在移动的动态场景的电视。我们设计了一个将运动解散到用户和对象运动中的框架，以在预测下一帧的同时有效地使用可用的用户运动。我们通过隔离和估计过去框架的3D对象运动，然后推断它来预测对象的运动。我们使用多平面图像（MPI）作为场景的3D表示，并将对象运动作为MPI表示中相应点之间的3D位移建模。为了在估计运动时处理MPI中的稀疏性，我们将部分卷积和掩盖的相关层纳入了相应的点。然后将预测的对象运动与给定的用户或相机运动集成在一起，以生成下一帧。使用不合格的填充模块，我们合成由于相机和对象运动而发现的区域。我们为动态场景的电视开发了一个新的合成数据集，该数据集由800个以全高清分辨率组成的视频组成。我们通过数据集和MPI Sintel数据集上的实验表明我们的模型优于文献中的所有竞争方法。

虚拟现实（VR）耳机提供了一种身临其境的立体视觉体验，但以阻止用户直接观察其物理环境的代价。传递技术旨在通过利用向外的摄像头来重建否则没有耳机的用户可以看到的图像来解决此限制。这本质上是一个实时视图综合挑战，因为传递摄像机不能与眼睛进行物理共同。现有的通行技术会遭受分散重建工件的注意力，这主要是由于缺乏准确的深度信息（尤其是对于近场和分离的物体），并且表现出有限的图像质量（例如，低分辨率和单色）。在本文中，我们提出了第一种学习的传递方法，并使用包含立体声对RGB摄像机的自定义VR耳机评估其性能。通过模拟和实验，我们证明了我们所学的传递方法与最先进的方法相比提供了卓越的图像质量，同时满足了实时的，透视透视的立体视图综合的严格VR要求，从而在广泛的视野上综合用于桌面连接的耳机。

Figure 1: Our method can synthesize novel views in both space and time from a single monocular video of a dynamic scene. Here we show video results with various configurations of fixing and interpolating view and time (left), as well as a visualization of the recovered scene geometry (right). Please view with Adobe Acrobat or KDE Okular to see animations.

We present a method for novel view synthesis from input images that are freely distributed around a scene. Our method does not rely on a regular arrangement of input views, can synthesize images for free camera movement through the scene, and works for general scenes with unconstrained geometric layouts. We calibrate the input images via SfM and erect a coarse geometric scaffold via MVS. This scaffold is used to create a proxy depth map for a novel view of the scene. Based on this depth map, a recurrent encoder-decoder network processes reprojected features from nearby views and synthesizes the new view. Our network does not need to be optimized for a given scene. After training on a dataset, it works in previously unseen environments with no finetuning or per-scene optimization. We evaluate the presented approach on challenging real-world datasets, including Tanks and Temples, where we demonstrate successful view synthesis for the first time and substantially outperform prior and concurrent work.

Per-pixel ground-truth depth data is challenging to acquire at scale. To overcome this limitation, self-supervised learning has emerged as a promising alternative for training models to perform monocular depth estimation. In this paper, we propose a set of improvements, which together result in both quantitatively and qualitatively improved depth maps compared to competing self-supervised methods.Research on self-supervised monocular training usually explores increasingly complex architectures, loss functions, and image formation models, all of which have recently helped to close the gap with fully-supervised methods. We show that a surprisingly simple model, and associated design choices, lead to superior predictions. In particular, we propose (i) a minimum reprojection loss, designed to robustly handle occlusions, (ii) a full-resolution multi-scale sampling method that reduces visual artifacts, and (iii) an auto-masking loss to ignore training pixels that violate camera motion assumptions. We demonstrate the effectiveness of each component in isolation, and show high quality, state-of-the-art results on the KITTI benchmark.

Learning based methods have shown very promising results for the task of depth estimation in single images. However, most existing approaches treat depth prediction as a supervised regression problem and as a result, require vast quantities of corresponding ground truth depth data for training. Just recording quality depth data in a range of environments is a challenging problem. In this paper, we innovate beyond existing approaches, replacing the use of explicit depth data during training with easier-to-obtain binocular stereo footage.We propose a novel training objective that enables our convolutional neural network to learn to perform single image depth estimation, despite the absence of ground truth depth data. Exploiting epipolar geometry constraints, we generate disparity images by training our network with an image reconstruction loss. We show that solving for image reconstruction alone results in poor quality depth images. To overcome this problem, we propose a novel training loss that enforces consistency between the disparities produced relative to both the left and right images, leading to improved performance and robustness compared to existing approaches. Our method produces state of the art results for monocular depth estimation on the KITTI driving dataset, even outperforming supervised methods that have been trained with ground truth depth.

用于运动中的人类的新型视图综合是一个具有挑战性的计算机视觉问题，使得诸如自由视视频之类的应用。现有方法通常使用具有多个输入视图，3D监控或预训练模型的复杂设置，这些模型不会概括为新标识。旨在解决这些限制，我们提出了一种新颖的视图综合框架，以从单视图传感器捕获的任何人的看法生成现实渲染，其具有稀疏的RGB-D，类似于低成本深度摄像头，而没有参与者特定的楷模。我们提出了一种架构来学习由基于球体的神经渲染获得的小说视图中的密集功能，并使用全局上下文修复模型创建完整的渲染。此外，增强剂网络利用了整体保真度，即使在原始视图中的遮挡区域中也能够产生细节的清晰渲染。我们展示了我们的方法为单个稀疏RGB-D输入产生高质量的合成和真实人体演员的新颖视图。它概括了看不见的身份，新的姿势，忠实地重建面部表情。我们的方法优于现有人体观测合成方法，并且对不同水平的输入稀疏性具有稳健性。

我们提出了一种学习方法，可以从单个视图开始生成自然场景的无界飞行视频，在该视图中，从单个照片集中学习了这种功能，而无需每个场景的相机姿势甚至多个视图。为了实现这一目标，我们提出了一种新颖的自我监督视图生成训练范式，在这里我们采样和渲染虚拟摄像头轨迹，包括循环轨迹，使我们的模型可以从单个视图集合中学习稳定的视图生成。在测试时，尽管在训练过程中从未见过视频，但我们的方法可以拍摄单个图像，并产生长的相机轨迹，包括数百个新视图，具有现实和多样化的内容。我们将我们的方法与最新的监督视图生成方法进行了比较，该方法需要摆姿势的多视频视频，并展示了卓越的性能和综合质量。

We present an unsupervised learning framework for the task of monocular depth and camera motion estimation from unstructured video sequences. In common with recent work [10,14,16], we use an end-to-end learning approach with view synthesis as the supervisory signal. In contrast to the previous work, our method is completely unsupervised, requiring only monocular video sequences for training. Our method uses single-view depth and multiview pose networks, with a loss based on warping nearby views to the target using the computed depth and pose. The networks are thus coupled by the loss during training, but can be applied independently at test time. Empirical evaluation on the KITTI dataset demonstrates the effectiveness of our approach: 1) monocular depth performs comparably with supervised methods that use either ground-truth pose or depth for training, and 2) pose estimation performs favorably compared to established SLAM systems under comparable input settings.

我们介绍从单个视频帧预测的问题，从单个视频帧，包括实际瞬时光流的光流量的低维子空间。我们展示了几种自然场景假设如何通过差异和对象实例的表示，通过一组基流字段来识别适当的流子空间。流量子空间与新颖的丢失函数一起可用于预测单眼深度或预测深度加上对象实例嵌入的任务。这提供了一种新方法，可以使用单眼输入视频以无监督的方式学习这些任务，而无需相机内在或姿势。

代表具有多个半透明彩色图层的场景是实时新型视图合成的流行和成功的选择。现有方法在平面或球形的规则间隔层上推断颜色和透明度值。在这项工作中，我们介绍了一种基于多个半透明层的新视图综合方法，具有场景适应的几何形状。我们的方法在两个阶段中介绍了立体对的这些表示。第一阶段从给定的一对视图中缩小了少数数据自适应层的几何形状。第二阶段为这些层的颜色和透明度值产生了新颖的视图合成的最终表示。重要的是，两个阶段都通过可差异化的渲染器连接，并以端到端的方式训练。在实验中，我们展示了所提出的方法在使用定期间隔的层上的优势，没有适应场景几何形状。尽管在渲染过程中较快的数量次数，但我们的方法也优于基于隐式几何表示的最近提出的IBRNET系统。查看https://samsunglabs.github.io/stereolayers的结果。

The DeepView architecture. (a) The network takes a sparse set of input images shot from different viewpoints. (b, c) The scene is reconstructed using learned gradient descent, producing a multi-plane image (a series of fronto-parallel, RGBA textured planes). (d)The multi-plane image is suitable for real-time, high-quality rendering of novel viewpoints. The result above uses four input views in a 30cm × 20cm rectangular layout. The novel view was rendered with a virtual camera positioned at the centroid of the four input views. More results, including video and an interactive viewer, at: https://augmentedperception.github.io/deepview/

Image view synthesis has seen great success in reconstructing photorealistic visuals, thanks to deep learning and various novel representations. The next key step in immersive virtual experiences is view synthesis of dynamic scenes. However, several challenges exist due to the lack of high-quality training datasets, and the additional time dimension for videos of dynamic scenes. To address this issue, we introduce a multi-view video dataset, captured with a custom 10-camera rig in 120FPS. The dataset contains 96 high-quality scenes showing various visual effects and human interactions in outdoor scenes. We develop a new algorithm, Deep 3D Mask Volume, which enables temporally-stable view extrapolation from binocular videos of dynamic scenes, captured by static cameras. Our algorithm addresses the temporal inconsistency of disocclusions by identifying the error-prone areas with a 3D mask volume, and replaces them with static background observed throughout the video. Our method enables manipulation in 3D space as opposed to simple 2D masks, We demonstrate better temporal stability than frame-by-frame static view synthesis methods, or those that use 2D masks. The resulting view synthesis videos show minimal flickering artifacts and allow for larger translational movements.