Figure. 1. The SYNTHIA Dataset. A sample frame (Left) with its semantic labels (center) and a general view of the city (right).

具有丰富注释的高质量结构化数据是处理道路场景的智能车辆系统中的关键组件。但是，数据策展和注释需要大量投资并产生低多样性的情况。最近对合成数据的兴趣日益增长，提出了有关此类系统改进范围的问题，以及产生大量和变化的模拟数据所需的手动工作量。这项工作提出了一条合成数据生成管道，该管道利用现有数据集（如Nuscenes）来解决模拟数据集中存在的困难和域间隙。我们表明，使用现有数据集的注释和视觉提示，我们可以促进自动化的多模式数据生成，模仿具有高保真性的真实场景属性，以及以物理意义的方式使样本多样化的机制。我们通过提供定性和定量实验，并通过使用真实和合成数据来证明MIOU指标的改进，以实现CityScapes和Kitti-Step数据集的语义分割。所有相关代码和数据均在GitHub（https://github.com/shubham1810/trove_toolkit）上发布。

During the last half decade, convolutional neural networks (CNNs) have triumphed over semantic segmentation, which is a core task of various emerging industrial applications such as autonomous driving and medical imaging. However, to train CNNs requires a huge amount of data, which is difficult to collect and laborious to annotate. Recent advances in computer graphics make it possible to train CNN models on photo-realistic synthetic data with computer-generated annotations. Despite this, the domain mismatch between the real images and the synthetic data significantly decreases the models' performance. Hence we propose a curriculum-style learning approach to minimize the domain gap in semantic segmentation. The curriculum domain adaptation solves easy tasks first in order to infer some necessary properties about the target domain; in particular, the first task is to learn global label distributions over images and local distributions over landmark superpixels. These are easy to estimate because images of urban traffic scenes have strong idiosyncrasies (e.g., the size and spatial relations of buildings, streets, cars, etc.). We then train the segmentation network in such a way that the network predictions in the target domain follow those inferred properties. In experiments, our method significantly outperforms the baselines as well as the only known existing approach to the same problem.

计算机图形技术的最新进展可以使汽车驾驶环境更现实。它们使自动驾驶汽车模拟器（例如DeepGTA-V和Carla（学习采取行动））能够生成大量的合成数据，这些数据可以补充现有的现实世界数据集中，以培训自动驾驶汽车感知。此外，由于自动驾驶汽车模拟器可以完全控制环境，因此它们可以产生危险的驾驶场景，而现实世界中数据集缺乏恶劣天气和事故情况。在本文中，我们将证明将从现实世界收集的数据与模拟世界中生成的数据相结合的有效性，以训练对象检测和本地化任务的感知系统。我们还将提出一个多层次的深度学习感知框架，旨在效仿人类的学习经验，其中在某个领域中学习了一系列从简单到更困难的任务。自动驾驶汽车感知器可以从易于驱动的方案中学习，以通过模拟软件定制的更具挑战性的方案。

TU Dresden www.cityscapes-dataset.net train/val -fine annotation -3475 images train -coarse annotation -20 000 images test -fine annotation -1525 images

Modern computer vision algorithms typically require expensive data acquisition and accurate manual labeling. In this work, we instead leverage the recent progress in computer graphics to generate fully labeled, dynamic, and photo-realistic proxy virtual worlds. We propose an efficient real-to-virtual world cloning method, and validate our approach by building and publicly releasing a new video dataset, called "Virtual KITTI" 1 , automatically labeled with accurate ground truth for object detection, tracking, scene and instance segmentation, depth, and optical flow. We provide quantitative experimental evidence suggesting that (i) modern deep learning algorithms pre-trained on real data behave similarly in real and virtual worlds, and (ii) pre-training on virtual data improves performance. As the gap between real and virtual worlds is small, virtual worlds enable measuring the impact of various weather and imaging conditions on recognition performance, all other things being equal. We show these factors may affect drastically otherwise high-performing deep models for tracking.

Computer vision applications in intelligent transportation systems (ITS) and autonomous driving (AD) have gravitated towards deep neural network architectures in recent years. While performance seems to be improving on benchmark datasets, many real-world challenges are yet to be adequately considered in research. This paper conducted an extensive literature review on the applications of computer vision in ITS and AD, and discusses challenges related to data, models, and complex urban environments. The data challenges are associated with the collection and labeling of training data and its relevance to real world conditions, bias inherent in datasets, the high volume of data needed to be processed, and privacy concerns. Deep learning (DL) models are commonly too complex for real-time processing on embedded hardware, lack explainability and generalizability, and are hard to test in real-world settings. Complex urban traffic environments have irregular lighting and occlusions, and surveillance cameras can be mounted at a variety of angles, gather dirt, shake in the wind, while the traffic conditions are highly heterogeneous, with violation of rules and complex interactions in crowded scenarios. Some representative applications that suffer from these problems are traffic flow estimation, congestion detection, autonomous driving perception, vehicle interaction, and edge computing for practical deployment. The possible ways of dealing with the challenges are also explored while prioritizing practical deployment.

The last decade witnessed increasingly rapid progress in self-driving vehicle technology, mainly backed up by advances in the area of deep learning and artificial intelligence. The objective of this paper is to survey the current state-of-the-art on deep learning technologies used in autonomous driving. We start by presenting AI-based self-driving architectures, convolutional and recurrent neural networks, as well as the deep reinforcement learning paradigm. These methodologies form a base for the surveyed driving scene perception, path planning, behavior arbitration and motion control algorithms. We investigate both the modular perception-planning-action pipeline, where each module is built using deep learning methods, as well as End2End systems, which directly map sensory information to steering commands. Additionally, we tackle current challenges encountered in designing AI architectures for autonomous driving, such as their safety, training data sources and computational hardware. The comparison presented in this survey helps to gain insight into the strengths and limitations of deep learning and AI approaches for autonomous driving and assist with design choices. 1

交通场景边缘壳体的语义分割的鲁棒性是智能运输安全的重要因素。然而，交通事故的大多数关键场景都是非常动态和以前看不见的，这严重损害了语义分割方法的性能。另外，在高速驾驶期间传统相机的延迟将进一步降低时间尺寸中的上下文信息。因此，我们建议从基于事件的数据提取动态上下文，以更高的时间分辨率来增强静态RGB图像，即使对于来自运动模糊，碰撞，变形，翻转等的流量事故而言，此外，为评估分割交通事故中的性能，我们提供了一个像素 - 明智的注释事故数据集，即Dada-Seg，其中包含来自交通事故的各种临界情景。我们的实验表明，基于事件的数据可以通过在事故中保留快速移动的前景（碰撞物体）的微粒运动来提供互补信息以在不利条件下稳定语义分割。我们的方法在拟议的事故数据集中实现了+ 8.2％的性能增益，超过了20多种最先进的语义细分方法。已经证明该提案对于在多个源数据库中学到的模型，包括CityScapes，Kitti-360，BDD和Apolloscape的模型始终如一。

Semantic segmentation works on the computer vision algorithm for assigning each pixel of an image into a class. The task of semantic segmentation should be performed with both accuracy and efficiency. Most of the existing deep FCNs yield to heavy computations and these networks are very power hungry, unsuitable for real-time applications on portable devices. This project analyzes current semantic segmentation models to explore the feasibility of applying these models for emergency response during catastrophic events. We compare the performance of real-time semantic segmentation models with non-real-time counterparts constrained by aerial images under oppositional settings. Furthermore, we train several models on the Flood-Net dataset, containing UAV images captured after Hurricane Harvey, and benchmark their execution on special classes such as flooded buildings vs. non-flooded buildings or flooded roads vs. non-flooded roads. In this project, we developed a real-time UNet based model and deployed that network on Jetson AGX Xavier module.

语义分割在广泛的计算机视觉应用中起着基本作用，提供了全球对图像​​的理解的关键信息。然而，最先进的模型依赖于大量的注释样本，其比在诸如图像分类的任务中获得更昂贵的昂贵的样本。由于未标记的数据替代地获得更便宜，因此无监督的域适应达到了语义分割社区的广泛成功并不令人惊讶。本调查致力于总结这一令人难以置信的快速增长的领域的五年，这包含了语义细分本身的重要性，以及将分段模型适应新环境的关键需求。我们提出了最重要的语义分割方法;我们对语义分割的域适应技术提供了全面的调查;我们揭示了多域学习，域泛化，测试时间适应或无源域适应等较新的趋势;我们通过描述在语义细分研究中最广泛使用的数据集和基准测试来结束本调查。我们希望本调查将在学术界和工业中提供具有全面参考指导的研究人员，并有助于他们培养现场的新研究方向。

我们建议利用模拟的潜力，以域的概括方式对现实世界自动驾驶场景的语义分割。对分割网络进行了训练，没有任何目标域数据，并在看不见的目标域进行了测试。为此，我们提出了一种新的域随机化和金字塔一致性的方法，以学习具有高推广性的模型。首先，我们建议使用辅助数据集以视觉外观的方式随机将合成图像随机化，以有效地学习域不变表示。其次，我们进一步在不同的“风格化”图像和图像中实施了金字塔一致性，以分别学习域不变和规模不变的特征。关于从GTA和合成对城市景观，BDD和Mapillary的概括进行了广泛的实验；而我们的方法比最新技术取得了卓越的成果。值得注意的是，我们的概括结果与最先进的模拟域适应方法相比甚至更好，甚至比在训练时访问目标域数据的结果。

Datasets drive vision progress, yet existing driving datasets are impoverished in terms of visual content and supported tasks to study multitask learning for autonomous driving. Researchers are usually constrained to study a small set of problems on one dataset, while real-world computer vision applications require performing tasks of various complexities. We construct BDD100K 1 , the largest driving video dataset with 100K videos and 10 tasks to evaluate the exciting progress of image recognition algorithms on autonomous driving. The dataset possesses geographic, environmental, and weather diversity, which is useful for training models that are less likely to be surprised by new conditions. Based on this diverse dataset, we build a benchmark for heterogeneous multitask learning and study how to solve the tasks together. Our experiments show that special training strategies are needed for existing models to perform such heterogeneous tasks. BDD100K opens the door for future studies in this important venue.

The Mapillary Vistas Dataset is a novel, largescale street-level image dataset containing 25 000 highresolution images annotated into 66 object categories with additional, instance-specific labels for 37 classes. Annotation is performed in a dense and fine-grained style by using polygons for delineating individual objects. Our dataset is 5× larger than the total amount of fine annotations for Cityscapes and contains images from all around the world, captured at various conditions regarding weather, season and daytime. Images come from different imaging devices (mobile phones, tablets, action cameras, professional capturing rigs) and differently experienced photographers. In such a way, our dataset has been designed and compiled to cover diversity, richness of detail and geographic extent. As default benchmark tasks, we define semantic image segmentation and instance-specific image segmentation, aiming to significantly further the development of state-of-theart methods for visual road-scene understanding.

环绕视图相机是用于自动驾驶的主要传感器，用于近场感知。它是主要用于停车可视化和自动停车的商用车中最常用的传感器之一。四个带有190 {\ deg}视场覆盖车辆周围360 {\ deg}的鱼眼相机。由于其高径向失真，标准算法不容易扩展。以前，我们发布了第一个名为Woodscape的公共鱼眼环境视图数据集。在这项工作中，我们发布了环绕视图数据集的合成版本，涵盖了其许多弱点并扩展了它。首先，不可能获得像素光流和深度的地面真相。其次，为了采样不同的框架，木景没有同时注释的所有四个相机。但是，这意味着不能设计多相机算法以在新数据集中启用的鸟眼空间中获得统一的输出。我们在Carla模拟器中实现了环绕式鱼眼的几何预测，与木观的配置相匹配并创建了Synwoodscape。

视频分析的图像分割在不同的研究领域起着重要作用，例如智能城市，医疗保健，计算机视觉和地球科学以及遥感应用。在这方面，最近致力于发展新的细分策略;最新的杰出成就之一是Panoptic细分。后者是由语义和实例分割的融合引起的。明确地，目前正在研究Panoptic细分，以帮助获得更多对视频监控，人群计数，自主驾驶，医学图像分析的图像场景的更细致的知识，以及一般对场景更深入的了解。为此，我们介绍了本文的首次全面审查现有的Panoptic分段方法，以获得作者的知识。因此，基于所采用的算法，应用场景和主要目标的性质，执行现有的Panoptic技术的明确定义分类。此外，讨论了使用伪标签注释新数据集的Panoptic分割。继续前进，进行消融研究，以了解不同观点的Panoptic方法。此外，讨论了适合于Panoptic分割的评估度量，并提供了现有解决方案性能的比较，以告知最先进的并识别其局限性和优势。最后，目前对主题技术面临的挑战和吸引不久的将来吸引相当兴趣的未来趋势，可以成为即将到来的研究研究的起点。提供代码的文件可用于：https：//github.com/elharroussomar/awesome-panoptic-egation

Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixellevel labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just 1 3 of the CamVid training set outperform models trained on the complete CamVid training set.

智能城市应用程序（例如智能交通路由或事故预防）依赖计算机视觉方法来确切的车辆定位和跟踪。由于精确标记的数据缺乏，从多个摄像机中检测和跟踪3D的车辆被证明是探索挑战的。我们提出了一个庞大的合成数据集，用于多个重叠和非重叠摄像头视图中的多个车辆跟踪和分割。与现有的数据集不同，该数据集仅为2D边界框提供跟踪地面真实，我们的数据集还包含适用于相机和世界坐标中的3D边界框的完美标签，深度估计以及实例，语义和泛型细分。该数据集由17个小时的标记视频材料组成，从64个不同的一天，雨，黎明和夜幕播放的340张摄像机录制，使其成为迄今为止多目标多型多相机跟踪的最广泛数据集。我们提供用于检测，车辆重新识别以及单摄像机跟踪的基准。代码和数据公开可用。

现有的计算机视觉系统可以与人类竞争，以理解物体的可见部分，但在描绘部分被遮挡物体的无形部分时，仍然远远远远没有达到人类。图像Amodal的完成旨在使计算机具有类似人类的Amodal完成功能，以了解完整的对象，尽管该对象被部分遮住。这项调查的主要目的是对图像Amodal完成领域的研究热点，关键技术和未来趋势提供直观的理解。首先，我们对这个新兴领域的最新文献进行了全面的评论，探讨了图像Amodal完成中的三个关键任务，包括Amodal形状完成，Amodal外观完成和订单感知。然后，我们检查了与图像Amodal完成有关的流行数据集及其共同的数据收集方法和评估指标。最后，我们讨论了现实世界中的应用程序和未来的研究方向，以实现图像的完成，从而促进了读者对现有技术和即将到来的研究趋势的挑战的理解。

不同的环境对长期自主驾驶的户外强大的视觉感知构成了巨大挑战，以及对不同环境影响的学习算法的概括仍然是一个公开问题。虽然最近单眼深度预测得到了很好的研究，但很少有很多工作，专注于不同环境的强大的基于学习的深度预测，例如，由于缺乏如此多环境的现实世界数据集和基准测试，不断变化照明和季节。为此，基于CMU Visual Location DataSet建立了第一个跨赛季单眼深度预测数据集和基准赛季。为了基准不同环境下的深度估计性能，我们使用几个新配制的指标调查来自Kitti基准的代表性和最近的最先进的开源监督，自我监督和域适应深度预测方法。通过对所提出的数据集进行广泛的实验评估，定性和定量分析了多种环境对性能和鲁棒性的影响，表明即使微调，长期单眼深度预测也仍然具有挑战性。我们进一步提供了承诺的途径，即自我监督的培训和立体声几何约束有助于提高改变环境的鲁棒性。数据集可在https://seasondepth.github.io上找到，并且在https://github.com/seasondepth/seasondepth上提供基准工具包。