Data augmentation is an effective technique for improving the accuracy of modern image classifiers. However, current data augmentation implementations are manually designed. In this paper, we describe a simple procedure called AutoAugment to automatically search for improved data augmentation policies. In our implementation, we have designed a search space where a policy consists of many subpolicies, one of which is randomly chosen for each image in each mini-batch. A sub-policy consists of two operations, each operation being an image processing function such as translation, rotation, or shearing, and the probabilities and magnitudes with which the functions are applied. We use a search algorithm to find the best policy such that the neural network yields the highest validation accuracy on a target dataset. Our method achieves state-of-the-art accuracy on SVHN, and ImageNet (without additional data). On ImageNet, we attain a Top-1 accuracy of 83.5% which is 0.4% better than the previous record of 83.1%. On CIFAR-10, we achieve an error rate of 1.5%, which is 0.6% better than the previous state-of-theart. Augmentation policies we find are transferable between datasets. The policy learned on ImageNet transfers well to achieve significant improvements on other datasets, such as Oxford Flowers, Caltech-101, Oxford-IIT Pets, FGVC Aircraft, and Stanford Cars. * Work performed as a member of the Google Brain Residency Program.† Equal contribution.

Recent work has shown that data augmentation has the potential to significantly improve the generalization of deep learning models. Recently, automated augmentation strategies have led to state-of-the-art results in image classification and object detection. While these strategies were optimized for improving validation accuracy, they also led to state-of-the-art results in semi-supervised learning and improved robustness to common corruptions of images. An obstacle to a large-scale adoption of these methods is a separate search phase which increases the training complexity and may substantially increase the computational cost. Additionally, due to the separate search phase, these approaches are unable to adjust the regularization strength based on model or dataset size. Automated augmentation policies are often found by training small models on small datasets and subsequently applied to train larger models. In this work, we remove both of these obstacles. RandAugment has a significantly reduced search space which allows it to be trained on the target task with no need for a separate proxy task. Furthermore, due to the parameterization, the regularization strength may be tailored to different model and dataset sizes. RandAugment can be used uniformly across different tasks and datasets and works out of the box, matching or surpassing all previous automated augmentation approaches on CIFAR-10/100, SVHN, and ImageNet. On the ImageNet dataset we achieve 85.0% accuracy, a 0.6% increase over the previous state-of-the-art and 1.0% increase over baseline augmentation. On object detection, RandAugment leads to 1.0-1.3% improvement over baseline augmentation, and is within 0.3% mAP of AutoAugment on COCO. Finally, due to its interpretable hyperparameter, RandAugment may be used to investigate the role of data augmentation with varying model and dataset size. Code is available online. 1 * Authors contributed equally.1 github.com/tensorflow/tpu/tree/master/models/ official/efficientnet

在许多分类问题中，我们希望一个对一系列非语义转换具有强大的分类器。例如，无论其出现的方向和姿势如何，人都可以识别图片中的狗。存在实质性证据表明这种不变性可以显着提高机器学习模型的准确性和泛化。教导模型几何修正型的常用技术是通过变换输入来增加训练数据。但是，对于给定的分类任务期望需要哪种修正，并不总是已知的。确定有效的数据增强策略可以要求域专业知识或广泛的数据预处理。最近的努力，如自动化优化数据增强策略的参数化搜索空间，以自动化增强过程。虽然自动化和类似方法在几个常见的数据集上实现最先进的分类准确性，但它们仅限于学习一个数据增强策略。通常不同的类别或功能呼叫不同的几何修正。我们介绍了动态网络增强（DNA），从而了解输入条件增强策略。我们模型中的增强参数是神经网络的输出，并且随着网络权重被更新时被隐式学习。我们的模型允许动态增强策略，并在输入功能上具有几何变换的数据良好。

近年来，计算机视觉社区中最受欢迎的技术之一就是深度学习技术。作为一种数据驱动的技术，深层模型需要大量准确标记的培训数据，这在许多现实世界中通常是无法访问的。数据空间解决方案是数据增强（DA），可以人为地从原始样本中生成新图像。图像增强策略可能因数据集而有所不同，因为不同的数据类型可能需要不同的增强以促进模型培训。但是，DA策略的设计主要由具有领域知识的人类专家决定，这被认为是高度主观和错误的。为了减轻此类问题，一个新颖的方向是使用自动数据增强（AUTODA）技术自动从给定数据集中学习图像增强策略。 Autoda模型的目的是找到可以最大化模型性能提高的最佳DA策略。这项调查从图像分类的角度讨论了Autoda技术出现的根本原因。我们确定标准自动赛车模型的三个关键组件：搜索空间，搜索算法和评估功能。根据他们的架构，我们提供了现有图像AUTODA方法的系统分类法。本文介绍了Autoda领域的主要作品，讨论了他们的利弊，并提出了一些潜在的方向以进行未来的改进。

Semi-supervised learning lately has shown much promise in improving deep learning models when labeled data is scarce. Common among recent approaches is the use of consistency training on a large amount of unlabeled data to constrain model predictions to be invariant to input noise. In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning. By substituting simple noising operations with advanced data augmentation methods such as RandAugment and back-translation, our method brings substantial improvements across six language and three vision tasks under the same consistency training framework. On the IMDb text classification dataset, with only 20 labeled examples, our method achieves an error rate of 4.20, outperforming the state-of-the-art model trained on 25,000 labeled examples. On a standard semi-supervised learning benchmark, CIFAR-10, our method outperforms all previous approaches and achieves an error rate of 5.43 with only 250 examples. Our method also combines well with transfer learning, e.g., when finetuning from BERT, and yields improvements in high-data regime, such as ImageNet, whether when there is only 10% labeled data or when a full labeled set with 1.3M extra unlabeled examples is used. 1

最近，出现了许多新的半监督学习方法。随着时间的流逝，ImageNet和类似数据集的准确性提高，尚未探索自然图像分类以外的任务的性能。大多数半监督的学习方法都依赖于精心设计的数据增强管道，该数据无法转移，用于在其他域的图像上学习。在这项工作中，我们提出了一种半监督的学习方法，该方法自动为特定数据集选择了最有效的数据增强策略。我们以FixMatch方法为基础，并通过增强元学习扩展它。在分类培训之前，在额外的培训中学习了增强，并利用双层优化，以优化增强政策并最大程度地提高准确性。我们在两个特定领域的数据集上评估我们的方法，其中包含卫星图像和手绘草图，并获得最新的结果。我们在消融中进一步研究与学习增强策略相关的不同参数，并展示了如何使用策略学习将增强功能调整到ImageNet之外的数据集中。

Semi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model's performance. This domain has seen fast progress recently, at the cost of requiring more complex methods. In this paper we propose FixMatch, an algorithm that is a significant simplification of existing SSL methods. FixMatch first generates pseudo-labels using the model's predictions on weaklyaugmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a strongly-augmented version of the same image. Despite its simplicity, we show that FixMatch achieves state-of-the-art performance across a variety of standard semi-supervised learning benchmarks, including 94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 -just 4 labels per class. We carry out an extensive ablation study to tease apart the experimental factors that are most important to FixMatch's success. The code is available at https://github.com/google-research/fixmatch.

Developing neural network image classification models often requires significant architecture engineering. In this paper, we study a method to learn the model architectures directly on the dataset of interest. As this approach is expensive when the dataset is large, we propose to search for an architectural building block on a small dataset and then transfer the block to a larger dataset. The key contribution of this work is the design of a new search space (which we call the "NASNet search space") which enables transferability. In our experiments, we search for the best convolutional layer (or "cell") on the CIFAR-10 dataset and then apply this cell to the ImageNet dataset by stacking together more copies of this cell, each with their own parameters to design a convolutional architecture, which we name a "NASNet architecture". We also introduce a new regularization technique called ScheduledDropPath that significantly improves generalization in the NASNet models. On CIFAR-10 itself, a NASNet found by our method achieves 2.4% error rate, which is state-of-the-art. Although the cell is not searched for directly on ImageNet, a NASNet constructed from the best cell achieves, among the published works, state-of-the-art accuracy of 82.7% top-1 and 96.2% top-5 on ImageNet. Our model is 1.2% better in top-1 accuracy than the best human-invented architectures while having 9 billion fewer FLOPS -a reduction of 28% in computational demand from the previous state-of-the-art model. When evaluated at different levels of computational cost, accuracies of NASNets exceed those of the state-of-the-art human-designed models. For instance, a small version of NASNet also achieves 74% top-1 accuracy, which is 3.1% better than equivalently-sized, state-of-the-art models for mobile platforms. Finally, the image features learned from image classification are generically useful and can be transferred to other computer vision problems. On the task of object detection, the learned features by NASNet used with the Faster-RCNN framework surpass state-of-the-art by 4.0% achieving 43.1% mAP on the COCO dataset.

数据增强方法丰富具有增强数据的数据集以提高神经网络的性能。最近，已经出现了自动化数据增强方法，自动设计增强策略。现有工作侧重于图像分类和对象检测，而我们提供了关于语义图像分割的第一次研究，并引入了两种新方法：\ Textit {Smartaugment}和\ Textit {SmartSamplingAugment}。 Smartaugment使用贝叶斯优化来搜索增强策略的丰富空间，并在我们考虑的所有语义细分任务中实现了新的最先进的性能。 SmartSamplingAugment，一种具有固定增强策略的简单参数方法，可与现有的资源密集型方法竞争性能，并且优于廉价的最先进的数据增强方法。此外，我们分析了数据增强超参数的影响，互动和重要性，并进行了融合研究，这确认了我们的设计选择，背后是Smartaugment和SmartSamplingAugment。最后，我们将提供我们的源代码以进行再现性，并促进进一步的研究。

This work tackles the problem of semi-supervised learning of image classifiers. Our main insight is that the field of semi-supervised learning can benefit from the quickly advancing field of self-supervised visual representation learning. Unifying these two approaches, we propose the framework of self-supervised semi-supervised learning (S 4 L) and use it to derive two novel semi-supervised image classification methods. We demonstrate the effectiveness of these methods in comparison to both carefully tuned baselines, and existing semi-supervised learning methods. We then show that S 4 L and existing semi-supervised methods can be jointly trained, yielding a new state-of-the-art result on semi-supervised ILSVRC-2012 with 10% of labels.

使用卷积神经网络（CNN）已经显着改善了几种图像处理任务，例如图像分类和对象检测。与Reset和Abseralnet一样，许多架构在创建时至少在一个数据集中实现了出色的结果。培训的一个关键因素涉及网络的正规化，这可以防止结构过度装备。这项工作分析了在过去几年中开发的几种正规化方法，显示了不同CNN模型的显着改进。该作品分为三个主要区域：第一个称为“数据增强”，其中所有技术都侧重于执行输入数据的更改。第二个，命名为“内部更改”，旨在描述修改神经网络或内核生成的特征映射的过程。最后一个称为“标签”，涉及转换给定输入的标签。这项工作提出了与关于正则化的其他可用调查相比的两个主要差异：（i）第一个涉及在稿件中收集的论文并非超过五年，并第二个区别是关于可重复性，即所有作品此处推荐在公共存储库中可用的代码，或者它们已直接在某些框架中实现，例如Tensorflow或Torch。

This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive selfsupervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by Sim-CLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-ofthe-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100× fewer labels. 1

数据增强（DA）已被广泛调查，以便于多项任务中的模型优化。然而，在大多数情况下，对于具有某种概率的每个训练样本随机地对数据增强进行数据增强，这可能会产生内容破坏和视觉模糊。为了消除这一点，在本文中，我们提出了一种有效的方法，将选择，以基于样本内容和网络培训状态选择要以确定性和在线方式增强的样本。具体而言，在每批中，我们首先确定增强比，然后决定是否以这种比率增强每个训练样本。我们将此过程塑造为两步马尔可夫决策过程，并采用分层强化学习（HRL）来学习增强策略。以这种方式，可以有效地缓解选择用于增强的样品在选择样品时的负面影响，并且改善了DA的有效性。广泛的实验表明，我们所提出的选择可以适应许多常用的DA方法，例如混合，切割，自动化等，以及改善图像分类和细粒度图像识别的多个基准数据集中的性能。

一个常见的分类任务情况是，有大量数据可用于培训，但只有一小部分用类标签注释。在这种情况下，半监督培训的目的是通过利用标记数据，而且从大量未标记的数据中提高分类准确性。最近的作品通过探索不同标记和未标记数据的不同增强性数据之间的一致性约束，从而取得了重大改进。遵循这条路径，我们提出了一个新颖的无监督目标，该目标侧重于彼此相似的高置信度未标记的数据之间所研究的关系较少。新提出的对损失最大程度地减少了高置信度伪伪标签之间的统计距离，其相似性高于一定阈值。我们提出的简单算法将对损失与MixMatch家族开发的技术结合在一起，显示出比以前在CIFAR-100和MINI-IMAGENET上的算法的显着性能增长，并且与CIFAR-的最先进方法相当。 10和SVHN。此外，简单还优于传输学习设置中最新方法，其中模型是由在ImainEnet或域内实现的权重初始化的。该代码可在github.com/zijian-hu/simple上获得。

Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -from 1 example per class to 1 M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.

Semi-supervised learning (SSL) provides a powerful framework for leveraging unlabeled data when labels are limited or expensive to obtain. SSL algorithms based on deep neural networks have recently proven successful on standard benchmark tasks. However, we argue that these benchmarks fail to address many issues that SSL algorithms would face in real-world applications. After creating a unified reimplementation of various widely-used SSL techniques, we test them in a suite of experiments designed to address these issues. We find that the performance of simple baselines which do not use unlabeled data is often underreported, SSL methods differ in sensitivity to the amount of labeled and unlabeled data, and performance can degrade substantially when the unlabeled dataset contains out-ofdistribution examples. To help guide SSL research towards real-world applicability, we make our unified reimplemention and evaluation platform publicly available. 2 * Equal contribution 2 https://github.com/brain-research/realistic-ssl-evaluation 32nd Conference on Neural Information Processing Systems (NeurIPS 2018),

The performance of generative adversarial networks (GANs) heavily deteriorates given a limited amount of training data. This is mainly because the discriminator is memorizing the exact training set. To combat it, we propose Differentiable Augmentation (DiffAugment), a simple method that improves the data efficiency of GANs by imposing various types of differentiable augmentations on both real and fake samples. Previous attempts to directly augment the training data manipulate the distribution of real images, yielding little benefit; DiffAugment enables us to adopt the differentiable augmentation for the generated samples, effectively stabilizes training, and leads to better convergence. Experiments demonstrate consistent gains of our method over a variety of GAN architectures and loss functions for both unconditional and class-conditional generation. With DiffAugment, we achieve a state-of-the-art FID of 6.80 with an IS of 100.8 on ImageNet 128×128 and 2-4× reductions of FID given 1,000 images on FFHQ and LSUN. Furthermore, with only 20% training data, we can match the top performance on CIFAR-10 and CIFAR-100. Finally, our method can generate high-fidelity images using only 100 images without pre-training, while being on par with existing transfer learning algorithms. Code is available at https://github.com/mit-han-lab/data-efficient-gans.

Semi-supervised learning has proven to be a powerful paradigm for leveraging unlabeled data to mitigate the reliance on large labeled datasets. In this work, we unify the current dominant approaches for semi-supervised learning to produce a new algorithm, MixMatch, that guesses low-entropy labels for data-augmented unlabeled examples and mixes labeled and unlabeled data using MixUp. MixMatch obtains state-of-the-art results by a large margin across many datasets and labeled data amounts. For example, on CIFAR-10 with 250 labels, we reduce error rate by a factor of 4 (from 38% to 11%) and by a factor of 2 on STL-10. We also demonstrate how MixMatch can help achieve a dramatically better accuracy-privacy trade-off for differential privacy. Finally, we perform an ablation study to tease apart which components of MixMatch are most important for its success. We release all code used in our experiments. 1

在培训期间应用的图像增强对于图像分类器的泛化性能至关重要。因此，大型研究已经专注于找到给定任务的最佳增强策略。然而，randaugment [2]是一个简单的随机增强策略，最近被证明胜过现有的复杂策略。只有对抗性自动化（Advaa）[11]，一种基于对抗性培训的想法的方法，表明比争夺更好。在本文中，我们表明，与最佳的对抗方法相比，随机增强仍然是竞争力的，以及简单的课程，并猜测ADVAA的成功是由于政策控制器网络的随机性，这引入了一种温和的形式课程。

Transfer learning is a cornerstone of computer vision, yet little work has been done to evaluate the relationship between architecture and transfer. An implicit hypothesis in modern computer vision research is that models that perform better on ImageNet necessarily perform better on other vision tasks. However, this hypothesis has never been systematically tested. Here, we compare the performance of 16 classification networks on 12 image classification datasets. We find that, when networks are used as fixed feature extractors or fine-tuned, there is a strong correlation between ImageNet accuracy and transfer accuracy (r = 0.99 and 0.96, respectively). In the former setting, we find that this relationship is very sensitive to the way in which networks are trained on ImageNet; many common forms of regularization slightly improve ImageNet accuracy but yield penultimate layer features that are much worse for transfer learning. Additionally, we find that, on two small fine-grained image classification datasets, pretraining on ImageNet provides minimal benefits, indicating the learned features from Ima-geNet do not transfer well to fine-grained tasks. Together, our results show that ImageNet architectures generalize well across datasets, but ImageNet features are less general than previously suggested.