增强了现实世界情景的稳健性已经被证明非常具有挑战性。一个原因是现有的鲁棒性基准是有限的，因为它们依赖于合成数据，或者它们只是将稳健性降低为数据集之间的概括，因此忽略各个滋扰因素的影响。在这项工作中，我们介绍了罗宾，是一个基准数据集，用于诊断视觉算法对现实世界中的个人滋扰的鲁棒性。罗宾在Pascal VOC 2012和Imagenet数据集中构建了10个刚性类别，并包括对象的分布示例3D姿势，形状，纹理，背景和天气状况。 Robin是丰富的注释，以实现图像分类，对象检测和3D姿势估计的基准模型。我们为许多流行的基线提供了结果，并进行了几个有趣的观察结果：1。与其他人相比，一些滋扰因素对性能有更强烈的负面影响。此外，对oodnuisance的负面影响取决于下游视觉任务。 2.利用强大数据增强的鲁棒性的目前的方法只有在现实世界的情况下只有边际效应，有时甚至会降低表现。 3.我们在鲁棒性方面，我们不会遵守卷积和变压器架构之间的任何显着差异。我们相信我们的数据集提供了丰富的试验台，以研究视觉算法的稳健性，并有助于大大推动该领域的前瞻性研究。

We introduce four new real-world distribution shift datasets consisting of changes in image style, image blurriness, geographic location, camera operation, and more. With our new datasets, we take stock of previously proposed methods for improving out-of-distribution robustness and put them to the test. We find that using larger models and artificial data augmentations can improve robustness on realworld distribution shifts, contrary to claims in prior work. We find improvements in artificial robustness benchmarks can transfer to real-world distribution shifts, contrary to claims in prior work. Motivated by our observation that data augmentations can help with real-world distribution shifts, we also introduce a new data augmentation method which advances the state-of-the-art and outperforms models pretrained with 1000× more labeled data. Overall we find that some methods consistently help with distribution shifts in texture and local image statistics, but these methods do not help with some other distribution shifts like geographic changes. Our results show that future research must study multiple distribution shifts simultaneously, as we demonstrate that no evaluated method consistently improves robustness.

我们介绍了几个新的数据集即想象的A / O和Imagenet-R以及合成环境和测试套件，我们称为CAOS。 Imagenet-A / O允许研究人员专注于想象成剩余的盲点。由于追踪稳健的表示，以特殊创建了ImageNet-R，因为表示不再简单地自然，而是包括艺术和其他演绎。 Caos Suite由Carla Simulator构建，允许包含异常物体，可以创建可重复的合成环境和用于测试稳健性的场景。所有数据集都是为测试鲁棒性和衡量鲁棒性的衡量进展而创建的。数据集已用于各种其他作品中，以衡量其具有鲁棒性的自身进步，并允许切向进展，这些进展不会完全关注自然准确性。鉴于这些数据集，我们创建了几种旨在推进鲁棒性研究的新方法。我们以最大Logit的形式和典型程度的形式构建简单的基线，并以深度的形式创建新的数据增强方法，从而提高上述基准。最大Logit考虑Logit值而不是SoftMax操作后的值，而微小的变化会产生明显的改进。典型程分将输出分布与类的后部分布进行比较。我们表明，除了分段任务之外，这将提高对基线的性能。猜测可能在像素级别，像素的语义信息比类级信息的语义信息不太有意义。最后，新的Deepaulment的新增强技术利用神经网络在彻底不同于先前使用的传统几何和相机的转换的图像上创建增强。

We introduce two challenging datasets that reliably cause machine learning model performance to substantially degrade. The datasets are collected with a simple adversarial filtration technique to create datasets with limited spurious cues. Our datasets' real-world, unmodified examples transfer to various unseen models reliably, demonstrating that computer vision models have shared weaknesses. The first dataset is called IMAGENET-A and is like the ImageNet test set, but it is far more challenging for existing models. We also curate an adversarial out-ofdistribution detection dataset called IMAGENET-O, which is the first out-of-distribution detection dataset created for ImageNet models. On IMAGENET-A a DenseNet-121 obtains around 2% accuracy, an accuracy drop of approximately 90%, and its out-of-distribution detection performance on IMAGENET-O is near random chance levels. We find that existing data augmentation techniques hardly boost performance, and using other public training datasets provides improvements that are limited. However, we find that improvements to computer vision architectures provide a promising path towards robust models.

The success of deep learning in vision can be attributed to: (a) models with high capacity; (b) increased computational power; and (c) availability of large-scale labeled data. Since 2012, there have been significant advances in representation capabilities of the models and computational capabilities of GPUs. But the size of the biggest dataset has surprisingly remained constant. What will happen if we increase the dataset size by 10× or 100×? This paper takes a step towards clearing the clouds of mystery surrounding the relationship between 'enormous data' and visual deep learning. By exploiting the JFT-300M dataset which has more than 375M noisy labels for 300M images, we investigate how the performance of current vision tasks would change if this data was used for representation learning. Our paper delivers some surprising (and some expected) findings. First, we find that the performance on vision tasks increases logarithmically based on volume of training data size. Second, we show that representation learning (or pretraining) still holds a lot of promise. One can improve performance on many vision tasks by just training a better base model. Finally, as expected, we present new state-of-theart results for different vision tasks including image classification, object detection, semantic segmentation and human pose estimation. Our sincere hope is that this inspires vision community to not undervalue the data and develop collective efforts in building larger datasets.

用于计算机视觉任务的深度神经网络在越来越安全 - 严重和社会影响的应用中部署，激励需要在各种，天然存在的成像条件下关闭模型性能的差距。在包括对抗机器学习的多种上下文中尤为色难地使用的鲁棒性，然后指在自然诱导的图像损坏或改变下保持模型性能。我们进行系统审查，以识别，分析和总结当前定义以及对计算机愿景深度学习中的非对抗鲁棒性的进展。我们发现，该研究领域已经收到了相对于对抗机器学习的不成比例地注意力，但存在显着的稳健性差距，这些差距通常表现在性能下降中与对抗条件相似。为了在上下文中提供更透明的稳健性定义，我们引入了数据生成过程的结构因果模型，并将非对抗性鲁棒性解释为模型在损坏的图像上的行为，其对应于来自未纳入数据分布的低概率样本。然后，我们确定提高神经网络鲁棒性的关键架构，数据增强和优化策略。这种稳健性的这种因果观察表明，目前文献中的常见做法，关于鲁棒性策略和评估，对应于因果概念，例如软干预导致成像条件的决定性分布。通过我们的调查结果和分析，我们提供了对未来研究如何可能介意这种明显和显着的非对抗的鲁棒性差距的观点。

尽管已经提出了几种方法来实现领域泛化的艰巨任务，但了解使这项任务挑战的原因很少受到关注。在这里，我们提出semanticdg（语义域概括）：具有15个具有相同几何形状，场景布局和摄像机参数与流行的3D Scannet数据集的基准标准，但具有照明，材料和视图点的控制域移动。使用此基准，我们独立研究了这些语义转变对概括的影响。视觉识别模型很容易推广到新颖的照明，但与材料和观点的分配变化斗争。受到人类视野的启发，我们假设场景上下文可以作为桥梁，以帮助模型跨越材料和观点域的转移，并提出上下文感知的视觉变压器，以及对材料和观点变化的对比损失，以解决这些域的变化。我们的方法（称为CDCNET）的表现优于现有域的概括方法，超过18％。作为关键的基准，我们还进行心理物理学实验，发现人类在照明，材料和观点上同样概括地概括了。此处介绍的基准和计算模型有助于了解与跨域的概括相关的挑战，并提供了向语义分布转移推断的初始步骤。我们在补充中包括所有数据和源代码。

Regional dropout strategies have been proposed to enhance the performance of convolutional neural network classifiers. They have proved to be effective for guiding the model to attend on less discriminative parts of objects (e.g. leg as opposed to head of a person), thereby letting the network generalize better and have better object localization capabilities. On the other hand, current methods for regional dropout remove informative pixels on training images by overlaying a patch of either black pixels or random noise. Such removal is not desirable because it leads to information loss and inefficiency during training. We therefore propose the CutMix augmentation strategy: patches are cut and pasted among training images where the ground truth labels are also mixed proportionally to the area of the patches. By making efficient use of training pixels and retaining the regularization effect of regional dropout, CutMix consistently outperforms the state-of-the-art augmentation strategies on CI-FAR and ImageNet classification tasks, as well as on the Im-ageNet weakly-supervised localization task. Moreover, unlike previous augmentation methods, our CutMix-trained ImageNet classifier, when used as a pretrained model, results in consistent performance gains in Pascal detection and MS-COCO image captioning benchmarks. We also show that CutMix improves the model robustness against input corruptions and its out-of-distribution detection performances. Source code and pretrained models are available at https://github.com/clovaai/CutMix-PyTorch.

本文对实例分割模型进行了全面评估，这些模型与现实世界图像损坏以及室外图像集合，例如与培训数据集不同的设置捕获的图像。室外图像评估显示了模型的概括能力，现实世界应用的一个基本方面以及广泛研究的域适应性主题。当设计用于现实世界应用程序的实例分割模型并选择现成的预期模型以直接用于手头的任务时，这些提出的鲁棒性和泛化评估很重要。具体而言，这项基准研究包括最先进的网络架构，网络骨架，标准化层，从头开始训练的模型，从头开始与预处理的网络以及多任务培训对稳健性和概括的影响。通过这项研究，我们获得了一些见解。例如，我们发现组归一化增强了跨损坏的网络的鲁棒性，其中图像内容保持不变，但损坏却添加在顶部。另一方面，分批归一化改善了图像特征统计信息在不同数据集上的概括。我们还发现，单阶段探测器比其训练大小不太概括到更大的图像分辨率。另一方面，多阶段探测器可以轻松地用于不同尺寸的图像上。我们希望我们的全面研究能够激发更强大和可靠的实例细分模型的发展。

适应不断发展的环境是所有自动驾驶系统不可避免地面临的安全挑战。但是，现有的图像和视频驾驶数据集未能捕获现实世界的可变性质。在本文中，我们介绍了最大的多任务合成数据集，用于自动驾驶，转移。它显示了云彩，雨水强度，一天中的时间以及车辆和行人密度的离散和连续变化。Shift采用全面的传感器套件和几个主流感知任务的注释，可以调查在域转移水平越来越高的感知系统性能下降，从而促进了持续适应策略的发展，以减轻此问题并评估模型的鲁棒性和一般性。我们的数据集和基准工具包可在www.vis.xyz/shift上公开获得。

对抗性训练（AT）通常被认为是防御对抗性例子的最有效的方法之一，可能会在很大程度上损害标准绩效，因此对工业规模的生产和应用的有用性有限。令人惊讶的是，这种现象在自然语言处理（NLP）任务中完全相反，在该任务中甚至可以从中受益。我们注意到NLP任务中AT的优点可能来自离散和符号输入空间。为了借用NLP风格的优势，我们提出了离散的对抗训练（DAT）。 DAT利用VQGAN改革图像数据以离散类似文本的输入，即视觉单词。然后，它可以最大程度地减少这种离散图像的最大风险，并具有符号对抗扰动。我们从分布的角度进一步提供了解释，以证明DAT的有效性。作为增强视觉表示的插件技术，DAT可以在多个任务上取得重大改进，包括图像分类，对象检测和自我监督学习。尤其是，该模型通过胶带自动编码（MAE）预先训练并由我们的DAT进行微调，而没有额外的数据可以在Imagenet-C上获得31.40 MCE，并且在Stylized-Imagenet上进行了32.77％的TOP-1准确性，建立了新的状态 - 艺术。该代码将在https://github.com/alibaba/easyrobust上找到。

We propose a technique for learning single-view 3D object pose estimation models by utilizing a new source of data -- in-the-wild videos where objects turn. Such videos are prevalent in practice (e.g., cars in roundabouts, airplanes near runways) and easy to collect. We show that classical structure-from-motion algorithms, coupled with the recent advances in instance detection and feature matching, provides surprisingly accurate relative 3D pose estimation on such videos. We propose a multi-stage training scheme that first learns a canonical pose across a collection of videos and then supervises a model for single-view pose estimation. The proposed technique achieves competitive performance with respect to existing state-of-the-art on standard benchmarks for 3D pose estimation, without requiring any pose labels during training. We also contribute an Accidental Turntables Dataset, containing a challenging set of 41,212 images of cars in cluttered backgrounds, motion blur and illumination changes that serves as a benchmark for 3D pose estimation.

人重新识别（RE-ID）在公共安全和视频监控等应用中起着重要作用。最近，从合成数据引擎的普及中获益的合成数据学习，从公众眼中引起了极大的关注。但是，现有数据集数量，多样性和变性有限，并且不能有效地用于重新ID问题。为了解决这一挑战，我们手动构造一个名为FineGPR的大型人数据集，具有细粒度的属性注释。此外，旨在充分利用FineGPR的潜力，并推广从数百万综合数据的高效培训，我们提出了一个名为AOST的属性分析流水线，它动态地学习了真实域中的属性分布，然后消除了合成和现实世界之间的差距因此，自由地部署到新场景。在基准上进行的实验表明，FineGPR具有AOST胜过（或与）现有的实际和合成数据集，这表明其对重新ID任务的可行性，并证明了众所周知的较少的原则。我们的Synthetic FineGPR数据集可公开可用于\ URL {https://github.com/jeremyxsc/finegpr}。

我们介绍了RP2K，这是一个新的大型零售产品数据集，用于细粒度图像分类。与以前的数据集专注于相对较少的产品，我们在归属于2000年不同产品的架子上收集超过500,000件零售产品图像。我们的数据集旨在推进零售对象识别的研究，该研究具有大量应用，如自动架审计和基于图像的产品信息检索。我们的数据集享受以下属性：（1）它是迄今为止产品类别的最大规模数据集。（2）所有图像在具有自然灯光的物理零售商店手动捕获，匹配真实应用的场景。（3）我们为每个对象提供丰富的注释，包括尺寸，形状和口味/气味。我们相信我们的数据集可以使计算机视觉研究和零售业受益。我们的数据集在https://www.pinlandata.com/rp2k_dataset上公开提供。

The 1$^{\text{st}}$ Workshop on Maritime Computer Vision (MaCVi) 2023 focused on maritime computer vision for Unmanned Aerial Vehicles (UAV) and Unmanned Surface Vehicle (USV), and organized several subchallenges in this domain: (i) UAV-based Maritime Object Detection, (ii) UAV-based Maritime Object Tracking, (iii) USV-based Maritime Obstacle Segmentation and (iv) USV-based Maritime Obstacle Detection. The subchallenges were based on the SeaDronesSee and MODS benchmarks. This report summarizes the main findings of the individual subchallenges and introduces a new benchmark, called SeaDronesSee Object Detection v2, which extends the previous benchmark by including more classes and footage. We provide statistical and qualitative analyses, and assess trends in the best-performing methodologies of over 130 submissions. The methods are summarized in the appendix. The datasets, evaluation code and the leaderboard are publicly available at https://seadronessee.cs.uni-tuebingen.de/macvi.

对分布（OOD）数据的概括是人类自然的能力，但对于机器而言挑战。这是因为大多数学习算法强烈依赖于i.i.d.〜对源/目标数据的假设，这在域转移导致的实践中通常会违反。域的概括（DG）旨在通过仅使用源数据进行模型学习来实现OOD的概括。在过去的十年中，DG的研究取得了长足的进步，导致了广泛的方法论，例如，基于域的一致性，元学习，数据增强或合奏学习的方法，仅举几例；还在各个应用领域进行了研究，包括计算机视觉，语音识别，自然语言处理，医学成像和强化学习。在本文中，首次提供了DG中的全面文献综述，以总结过去十年来的发展。具体而言，我们首先通过正式定义DG并将其与其他相关领域（如域适应和转移学习）联系起来来涵盖背景。然后，我们对现有方法和理论进行了彻底的审查。最后，我们通过有关未来研究方向的见解和讨论来总结这项调查。

We address the task of open-world class-agnostic object detection, i.e., detecting every object in an image by learning from a limited number of base object classes. State-of-the-art RGB-based models suffer from overfitting the training classes and often fail at detecting novel-looking objects. This is because RGB-based models primarily rely on appearance similarity to detect novel objects and are also prone to overfitting short-cut cues such as textures and discriminative parts. To address these shortcomings of RGB-based object detectors, we propose incorporating geometric cues such as depth and normals, predicted by general-purpose monocular estimators. Specifically, we use the geometric cues to train an object proposal network for pseudo-labeling unannotated novel objects in the training set. Our resulting Geometry-guided Open-world Object Detector (GOOD) significantly improves detection recall for novel object categories and already performs well with only a few training classes. Using a single "person" class for training on the COCO dataset, GOOD surpasses SOTA methods by 5.0% AR@100, a relative improvement of 24%.

自我监督学习的最新进展证明了多种视觉任务的有希望的结果。高性能自我监督方法中的一个重要成分是通过培训模型使用数据增强，以便在嵌入空间附近的相同图像的不同增强视图。然而，常用的增强管道整体地对待图像，忽略图像的部分的语义相关性-e.g。主题与背景 - 这可能导致学习杂散相关性。我们的工作通过调查一类简单但高度有效的“背景增强”来解决这个问题，这鼓励模型专注于语义相关内容，劝阻它们专注于图像背景。通过系统的调查，我们表明背景增强导致在各种任务中跨越一系列最先进的自我监督方法（MOCO-V2，BYOL，SWAV）的性能大量改进。 $ \ SIM $ + 1-2％的ImageNet收益，使得与监督基准的表现有关。此外，我们发现有限标签设置的改进甚至更大（高达4.2％）。背景技术增强还改善了许多分布换档的鲁棒性，包括天然对抗性实例，想象群-9，对抗性攻击，想象成型。我们还在产生了用于背景增强的显着掩模的过程中完全无监督的显着性检测进展。

In this paper, we introduce a new large-scale object detection dataset, Objects365, which has 365 object categories over 600K training images. More than 10 million, high-quality bounding boxes are manually labeled through a three-step, carefully designed annotation pipeline. It is the largest object detection dataset (with full annotation) so far and establishes a more challenging benchmark for the community. Objects365 can serve as a better feature learning dataset for localization-sensitive tasks like object detection and semantic segmentation. The Objects365 pre-trained models significantly outperform ImageNet pre-trained models with 5.6 points gain (42 vs 36.4) based on the standard setting of 90K iterations on COCO benchmark. Even compared with much long training time like 540K iterations, our Objects365 pretrained model with 90K iterations still have 2.7 points gain (42 vs 39.3). Meanwhile, the finetuning time can be greatly reduced (up to 10 times) when reaching the same accuracy. Better generalization ability of Object365 has also been verified on CityPersons, VOC segmentation, and ADE tasks. The dataset as well as the pretrainedmodels have been released at www.objects365.org. * indicates equal contribution.

Modern deep neural networks tend to be evaluated on static test sets. One shortcoming of this is the fact that these deep neural networks cannot be easily evaluated for robustness issues with respect to specific scene variations. For example, it is hard to study the robustness of these networks to variations of object scale, object pose, scene lighting and 3D occlusions. The main reason is that collecting real datasets with fine-grained naturalistic variations of sufficient scale can be extremely time-consuming and expensive. In this work, we present Counterfactual Simulation Testing, a counterfactual framework that allows us to study the robustness of neural networks with respect to some of these naturalistic variations by building realistic synthetic scenes that allow us to ask counterfactual questions to the models, ultimately providing answers to questions such as "Would your classification still be correct if the object were viewed from the top?" or "Would your classification still be correct if the object were partially occluded by another object?". Our method allows for a fair comparison of the robustness of recently released, state-of-the-art Convolutional Neural Networks and Vision Transformers, with respect to these naturalistic variations. We find evidence that ConvNext is more robust to pose and scale variations than Swin, that ConvNext generalizes better to our simulated domain and that Swin handles partial occlusion better than ConvNext. We also find that robustness for all networks improves with network scale and with data scale and variety. We release the Naturalistic Variation Object Dataset (NVD), a large simulated dataset of 272k images of everyday objects with naturalistic variations such as object pose, scale, viewpoint, lighting and occlusions. Project page: https://counterfactualsimulation.github.io