分批归一化（BN）由归一化组成部分，然后是仿射转化，并且对于训练深神经网络至关重要。网络中每个BN的标准初始化分别设置了仿射变换量表，并将其转移到1和0。但是，经过训练，我们观察到这些参数从初始化中并没有太大变化。此外，我们注意到归一化过程仍然可以产生过多的值，这对于训练是不可能的。我们重新审视BN公式，并为BN提出了一种新的初始化方法和更新方法，以解决上述问题。实验旨在强调和证明适当的BN规模初始化对性能的积极影响，并使用严格的统计显着性测试进行评估。该方法可以与现有实施方式一起使用，没有额外的计算成本。源代码可在https://github.com/osu-cvl/revisiting-bninit上获得。

神经架构的创新促进了语言建模和计算机视觉中的重大突破。不幸的是，如果网络参数未正确初始化，新颖的架构通常会导致挑战超参数选择和培训不稳定。已经提出了许多架构特定的初始化方案，但这些方案并不总是可移植到新体系结构。本文介绍了毕业，一种用于初始化神经网络的自动化和架构不可知论由方法。毕业基础是一个简单的启发式;调整每个网络层的规范，使得具有规定的超参数的SGD或ADAM的单个步骤导致可能的损耗值最小。通过在每个参数块前面引入标量乘数变量，然后使用简单的数字方案优化这些变量来完成此调整。 GradInit加速了许多卷积架构的收敛性和测试性能，无论是否有跳过连接，甚至没有归一化层。它还提高了机器翻译的原始变压器架构的稳定性，使得在广泛的学习速率和动量系数下使用ADAM或SGD来训练它而无需学习速率预热。代码可在https://github.com/zhuchen03/gradinit上获得。

Neural networks require careful weight initialization to prevent signals from exploding or vanishing. Existing initialization schemes solve this problem in specific cases by assuming that the network has a certain activation function or topology. It is difficult to derive such weight initialization strategies, and modern architectures therefore often use these same initialization schemes even though their assumptions do not hold. This paper introduces AutoInit, a weight initialization algorithm that automatically adapts to different neural network architectures. By analytically tracking the mean and variance of signals as they propagate through the network, AutoInit appropriately scales the weights at each layer to avoid exploding or vanishing signals. Experiments demonstrate that AutoInit improves performance of convolutional, residual, and transformer networks across a range of activation function, dropout, weight decay, learning rate, and normalizer settings, and does so more reliably than data-dependent initialization methods. This flexibility allows AutoInit to initialize models for everything from small tabular tasks to large datasets such as ImageNet. Such generality turns out particularly useful in neural architecture search and in activation function discovery. In these settings, AutoInit initializes each candidate appropriately, making performance evaluations more accurate. AutoInit thus serves as an automatic configuration tool that makes design of new neural network architectures more robust. The AutoInit package provides a wrapper around TensorFlow models and is available at https://github.com/cognizant-ai-labs/autoinit.

Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batchnormalized networks, we improve upon the best published result on ImageNet classification: reaching 4.9% top-5 validation error (and 4.8% test error), exceeding the accuracy of human raters.

我们调查与批量无关的归一化产生的性能降解的原因。我们发现层归一化和实例归一化的原型技术均诱导神经网络预激活中的故障模式的外观：（i）层归一化引起往复恒定函数的崩溃;（ii）实例归一化在实例统计中缺乏可变性，表现出富有症状的症状。为了缓解失败模式（i）而不加重失败模式（ii），我们介绍了使用代理分布的激活后标准化的技术“代理归一化”。当与层归一化或组归一化结合时，这种批量独立的归一化会模拟批量标准化的行为，并始终如一地匹配或超过其性能。

差异隐私（DP）提供了正式的隐私保证，以防止对手可以访问机器学习模型，从而从提取有关单个培训点的信息。最受欢迎的DP训练方法是差异私有随机梯度下降（DP-SGD），它通过在训练过程中注入噪声来实现这种保护。然而，以前的工作发现，DP-SGD通常会导致标准图像分类基准的性能显着降解。此外，一些作者假设DP-SGD在大型模型上固有地表现不佳，因为保留隐私所需的噪声规范与模型维度成正比。相反，我们证明了过度参数化模型上的DP-SGD可以比以前想象的要好得多。将仔细的超参数调整与简单技术结合起来，以确保信号传播并提高收敛速率，我们获得了新的SOTA，而没有额外数据的CIFAR-10，在81.4％的81.4％下（8，10^{ - 5}） - 使用40 -layer wide-Resnet，比以前的SOTA提高了71.7％。当对预训练的NFNET-F3进行微调时，我们在ImageNet（0.5，8*10^{ - 7}）下达到了83.8％的TOP-1精度。此外，我们还在（8，8 \ cdot 10^{ - 7}）下达到了86.7％的TOP-1精度，DP仅比当前的非私人SOTA仅4.3％。我们认为，我们的结果是缩小私人图像分类和非私有图像分类之间准确性差距的重要一步。

Transfer learning is a cornerstone of computer vision, yet little work has been done to evaluate the relationship between architecture and transfer. An implicit hypothesis in modern computer vision research is that models that perform better on ImageNet necessarily perform better on other vision tasks. However, this hypothesis has never been systematically tested. Here, we compare the performance of 16 classification networks on 12 image classification datasets. We find that, when networks are used as fixed feature extractors or fine-tuned, there is a strong correlation between ImageNet accuracy and transfer accuracy (r = 0.99 and 0.96, respectively). In the former setting, we find that this relationship is very sensitive to the way in which networks are trained on ImageNet; many common forms of regularization slightly improve ImageNet accuracy but yield penultimate layer features that are much worse for transfer learning. Additionally, we find that, on two small fine-grained image classification datasets, pretraining on ImageNet provides minimal benefits, indicating the learned features from Ima-geNet do not transfer well to fine-grained tasks. Together, our results show that ImageNet architectures generalize well across datasets, but ImageNet features are less general than previously suggested.

我们介绍Softmax梯度篡改，一种用于修改神经网络后向通过的梯度的技术，以提高其准确性。我们的方法使用基于功率的概率变换来改变预测的概率值，然后将梯度重新计算在后向通过。这种修改导致更平滑的渐变简介，我们在经验和理论上展示。我们对剩余网络进行了转换参数进行了网格搜索。我们证明修改CUMMNET中的软MAX梯度可能导致培训准确性提高，从而增加训练数据的适合，并最大限度地利用神经网络的学习能力。当与标签平滑等正则化技术相结合时，我们获得更好的测试度量和更低的泛化间隙。 Softmax渐变篡改在ImageNet DataSet上的基线上以0.52 \％$ 0.52 \％$ 0.52 \％$ 0.52 \％。我们的方法非常通用，可以跨各种不同的网络架构和数据集使用。

Image classification with small datasets has been an active research area in the recent past. However, as research in this scope is still in its infancy, two key ingredients are missing for ensuring reliable and truthful progress: a systematic and extensive overview of the state of the art, and a common benchmark to allow for objective comparisons between published methods. This article addresses both issues. First, we systematically organize and connect past studies to consolidate a community that is currently fragmented and scattered. Second, we propose a common benchmark that allows for an objective comparison of approaches. It consists of five datasets spanning various domains (e.g., natural images, medical imagery, satellite data) and data types (RGB, grayscale, multispectral). We use this benchmark to re-evaluate the standard cross-entropy baseline and ten existing methods published between 2017 and 2021 at renowned venues. Surprisingly, we find that thorough hyper-parameter tuning on held-out validation data results in a highly competitive baseline and highlights a stunted growth of performance over the years. Indeed, only a single specialized method dating back to 2019 clearly wins our benchmark and outperforms the baseline classifier.

尽管对视觉识别任务进行了显着进展，但是当培训数据稀缺或高度不平衡时，深神经网络仍然易于普遍，使他们非常容易受到现实世界的例子。在本文中，我们提出了一种令人惊讶的简单且高效的方法来缓解此限制：使用纯噪声图像作为额外的训练数据。与常见使用添加剂噪声或对抗数据的噪声不同，我们通过直接训练纯无随机噪声图像提出了完全不同的视角。我们提出了一种新的分发感知路由批量归一化层（DAR-BN），除了同一网络内的自然图像之外，还可以在纯噪声图像上训练。这鼓励泛化和抑制过度装备。我们所提出的方法显着提高了不平衡的分类性能，从而获得了最先进的导致大量的长尾图像分类数据集（Cifar-10-LT，CiFar-100-LT，想象齿 - LT，和celeba-5）。此外，我们的方法非常简单且易于使用作为一般的新增强工具（在现有增强的顶部），并且可以在任何训练方案中结合。它不需要任何专门的数据生成或培训程序，从而保持培训快速高效

Very deep convolutional networks with hundreds of layers have led to significant reductions in error on competitive benchmarks. Although the unmatched expressiveness of the many layers can be highly desirable at test time, training very deep networks comes with its own set of challenges. The gradients can vanish, the forward flow often diminishes, and the training time can be painfully slow. To address these problems, we propose stochastic depth, a training procedure that enables the seemingly contradictory setup to train short networks and use deep networks at test time. We start with very deep networks but during training, for each mini-batch, randomly drop a subset of layers and bypass them with the identity function. This simple approach complements the recent success of residual networks. It reduces training time substantially and improves the test error significantly on almost all data sets that we used for evaluation. With stochastic depth we can increase the depth of residual networks even beyond 1200 layers and still yield meaningful improvements in test error (4.91% on CIFAR-10).

Most modern convolutional neural networks (CNNs) used for object recognition are built using the same principles: Alternating convolution and max-pooling layers followed by a small number of fully connected layers. We re-evaluate the state of the art for object recognition from small images with convolutional networks, questioning the necessity of different components in the pipeline. We find that max-pooling can simply be replaced by a convolutional layer with increased stride without loss in accuracy on several image recognition benchmarks. Following this finding -and building on other recent work for finding simple network structures -we propose a new architecture that consists solely of convolutional layers and yields competitive or state of the art performance on several object recognition datasets (CIFAR-10, CIFAR-100, ImageNet). To analyze the network we introduce a new variant of the "deconvolution approach" for visualizing features learned by CNNs, which can be applied to a broader range of network structures than existing approaches.

模型量化已成为加速深度学习推理的不可或缺的技术。虽然研究人员继续推动量化算法的前沿，但是现有量化工作通常是不可否认的和不可推销的。这是因为研究人员不选择一致的训练管道并忽略硬件部署的要求。在这项工作中，我们提出了模型量化基准（MQBench），首次尝试评估，分析和基准模型量化算法的再现性和部署性。我们为实际部署选择多个不同的平台，包括CPU，GPU，ASIC，DSP，并在统一培训管道下评估广泛的最新量化算法。 MQBENCK就像一个连接算法和硬件的桥梁。我们进行全面的分析，并找到相当大的直观或反向直观的见解。通过对齐训练设置，我们发现现有的算法在传统的学术轨道上具有大致相同的性能。虽然用于硬件可部署量化，但有一个巨大的精度差距，仍然不稳定。令人惊讶的是，没有现有的算法在MQBench中赢得每一项挑战，我们希望这项工作能够激发未来的研究方向。

The deployment of deep convolutional neural networks (CNNs) in many real world applications is largely hindered by their high computational cost. In this paper, we propose a novel learning scheme for CNNs to simultaneously 1) reduce the model size; 2) decrease the run-time memory footprint; and 3) lower the number of computing operations, without compromising accuracy. This is achieved by enforcing channel-level sparsity in the network in a simple but effective way. Different from many existing approaches, the proposed method directly applies to modern CNN architectures, introduces minimum overhead to the training process, and requires no special software/hardware accelerators for the resulting models. We call our approach network slimming, which takes wide and large networks as input models, but during training insignificant channels are automatically identified and pruned afterwards, yielding thin and compact models with comparable accuracy. We empirically demonstrate the effectiveness of our approach with several state-of-the-art CNN models, including VGGNet, ResNet and DenseNet, on various image classification datasets. For VGGNet, a multi-pass version of network slimming gives a 20× reduction in model size and a 5× reduction in computing operations.

We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches 74.3% top-1 classification accuracy on ImageNet using a linear evaluation with a ResNet-50 architecture and 79.6% with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. Our implementation and pretrained models are given on GitHub. 3 * Equal contribution; the order of first authors was randomly selected.

The choice of activation functions and their motivation is a long-standing issue within the neural network community. Neuronal representations within artificial neural networks are commonly understood as logits, representing the log-odds score of presence of features within the stimulus. We derive logit-space operators equivalent to probabilistic Boolean logic-gates AND, OR, and XNOR for independent probabilities. Such theories are important to formalize more complex dendritic operations in real neurons, and these operations can be used as activation functions within a neural network, introducing probabilistic Boolean-logic as the core operation of the neural network. Since these functions involve taking multiple exponents and logarithms, they are computationally expensive and not well suited to be directly used within neural networks. Consequently, we construct efficient approximations named $\text{AND}_\text{AIL}$ (the AND operator Approximate for Independent Logits), $\text{OR}_\text{AIL}$, and $\text{XNOR}_\text{AIL}$, which utilize only comparison and addition operations, have well-behaved gradients, and can be deployed as activation functions in neural networks. Like MaxOut, $\text{AND}_\text{AIL}$ and $\text{OR}_\text{AIL}$ are generalizations of ReLU to two-dimensions. While our primary aim is to formalize dendritic computations within a logit-space probabilistic-Boolean framework, we deploy these new activation functions, both in isolation and in conjunction to demonstrate their effectiveness on a variety of tasks including image classification, transfer learning, abstract reasoning, and compositional zero-shot learning.

批量标准化是所有最先进的神经网络架构的必要组件。然而，由于引入了许多实际问题，最近的研究已经致力于设计无规范化的架构。在本文中，我们表明权重初始化是培训Reset的归一化网络的关键。特别是，我们提出了对跳过连接分支的块输出的求和操作的略微修改，从而正确初始化整个网络。我们表明，这种修改的体系结构在CiFar-10上实现了竞争结果，而无需进一步正常化，也不是算法修改。

我们通过应用更为理论证明的操作员来寻求改善神经网络中的汇集操作。我们证明Logsumexp提供了用于登录的自然或操作员。当一个人对池中汇集运算符中的元素数正确时，这将成为$ \ text {logavgexp}：= \ log（\ text {mean}（\ exp（x）））$。通过引入单个温度参数，LogavgeXP将其操作数的最大值平滑地过渡到平均值（在限制性情况下发现$ 0 ^ + $和$ t \ to + \ idty $）。在各种深度神经网络架构中，我们在实验测试的LogavgeXP，无论是没有学习的温度参数，都在电脑视觉中的各种深度神经网络架构中。

We introduce the "exponential linear unit" (ELU) which speeds up learning in deep neural networks and leads to higher classification accuracies. Like rectified linear units (ReLUs), leaky ReLUs (LReLUs) and parametrized ReLUs (PRe-LUs), ELUs alleviate the vanishing gradient problem via the identity for positive values. However ELUs have improved learning characteristics compared to the units with other activation functions. In contrast to ReLUs, ELUs have negative values which allows them to push mean unit activations closer to zero like batch normalization but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit natural gradient because of a reduced bias shift effect. While LReLUs and PReLUs have negative values, too, they do not ensure a noise-robust deactivation state. ELUs saturate to a negative value with smaller inputs and thereby decrease the forward propagated variation and information. Therefore ELUs code the degree of presence of particular phenomena in the input, while they do not quantitatively model the degree of their absence. In experiments, ELUs lead not only to faster learning, but also to significantly better generalization performance than ReLUs and LReLUs on networks with more than 5 layers. On CIFAR-100 ELUs networks significantly outperform ReLU networks with batch normalization while batch normalization does not improve ELU networks. ELU networks are among the top 10 reported CIFAR-10 results and yield the best published result on CIFAR-100, without resorting to multi-view evaluation or model averaging. On ImageNet, ELU networks considerably speed up learning compared to a ReLU network with the same architecture, obtaining less than 10% classification error for a single crop, single model network.

批量白化是一种通过转换输入特征来加速和稳定训练的技术，以具有零平均（居中）和单位方差（缩放），并且通过去除信道（去相关）之间的线性相关性。在常用的结构中，通过批量归一化经验优化，归一化层出现在卷积和激活功能之间。在批量白化研究中采用相同的结构而无需进一步分析;甚至分析了批次白化的前提，即线性层的输入变白。为了弥补差距，我们提出了一种新的卷积单元，符合该理论，我们的方法通常提高批量美白的性能。此外，我们通过调查特征的等级和相关性来展示原始卷积单元的效率。由于我们的方法是可采用的现成增白模块，我们使用迭代标准化（Iternorm），最先进的美白模块，并在五个图像分类数据集中获得显着提高的性能：CiFar-10，CiFar-100 ，幼崽200-2011，斯坦福狗和想象。值得注意的是，我们验证了我们的方法在使用大型学习率，组大小和迭代号时，提高了白化的稳定性和性能。