使用DataSet的真实标签培训而不是随机标签导致更快的优化和更好的泛化。这种差异归因于自然数据集中的输入和标签之间的对齐概念。我们发现,随机或真正标签上的具有不同架构和优化器的培训神经网络在隐藏的表示和训练标签之间强制执行相同的关系,阐明为什么神经网络表示为转移如此成功。我们首先突出显示为什么对齐的特征在经典的合成转移问题中促进转移和展示,即对齐是对相似和不同意任务的正负传输的确定因素。然后我们调查各种神经网络架构,并发现(a)在各种不同的架构和优化器中出现的对齐,并且从深度(b)对准产生的更多对准对于更接近输出的层和(c)现有的性能深度CNN表现出高级别的对准。
translated by 谷歌翻译
Recent work reported the label alignment property in a supervised learning setting: the vector of all labels in the dataset is mostly in the span of the top few singular vectors of the data matrix. Inspired by this observation, we derive a regularization method for unsupervised domain adaptation. Instead of regularizing representation learning as done by popular domain adaptation methods, we regularize the classifier so that the target domain predictions can to some extent ``align" with the top singular vectors of the unsupervised data matrix from the target domain. In a linear regression setting, we theoretically justify the label alignment property and characterize the optimality of the solution of our regularization by bounding its distance to the optimal solution. We conduct experiments to show that our method can work well on the label shift problems, where classic domain adaptation methods are known to fail. We also report mild improvement over domain adaptation baselines on a set of commonly seen MNIST-USPS domain adaptation tasks and on cross-lingual sentiment analysis tasks.
translated by 谷歌翻译
懒惰培训制度中的神经网络收敛到内核机器。在丰富的特征学习制度中可以在丰富的特征学习制度中可以使用数据依赖性内核来学习内核机器吗?我们证明,这可以是由于我们术语静音对准的现象,这可能需要网络的切线内核在特征内演变,而在小并且在损失明显降低,并且之后仅在整体尺度上生长。我们表明这种效果在具有小初始化和白化数据的同质神经网络中进行。我们在线性网络壳体提供了对这种效果的分析处理。一般来说,我们发现内核在训练的早期阶段开发了低级贡献,然后在总体上发展,产生了与最终网络的切线内核的内核回归解决方案等同的函数。内核的早期光谱学习取决于深度。我们还证明了非白化数据可以削弱无声的对准效果。
translated by 谷歌翻译
以前的工作提出了许多新的损失函数和常规程序,可提高图像分类任务的测试准确性。但是,目前尚不清楚这些损失函数是否了解下游任务的更好表示。本文研究了培训目标的选择如何影响卷积神经网络隐藏表示的可转移性,训练在想象中。我们展示了许多目标在Vanilla Softmax交叉熵上导致想象的精度有统计学意义的改进,但由此产生的固定特征提取器转移到下游任务基本较差,并且当网络完全微调时,损失的选择几乎没有效果新任务。使用居中内核对齐来测量网络隐藏表示之间的相似性,我们发现损失函数之间的差异仅在网络的最后几层中都很明显。我们深入了解倒数第二层的陈述,发现不同的目标和近奇计的组合导致大幅不同的类别分离。具有较高类别分离的表示可以在原始任务上获得更高的准确性,但它们的功能对于下游任务不太有用。我们的结果表明,用于原始任务的学习不变功能与传输任务相关的功能之间存在权衡。
translated by 谷歌翻译
在给出深层神经网络成功的理论上说明的尝试中,最近的一项工作已经确定了所谓的“懒惰”制度,在该制度中,网络可以通过其围绕初始化的线性化来很好地近似。在这里,我们根据示例的难度研究了懒惰(线性)和特征学习(非线性)制度对示例子组的比较效应。具体而言,我们表明,在功能学习模式下给出了更容易的示例,与更困难的训练相比,训练更快。换句话说,非线性动力学倾向于顺序学习增加难度的示例。我们在不同的方式上说明了这种现象,以量化示例难度,包括C得分,标签噪声以及存在虚假相关性。我们的结果揭示了对深度网络在示例难度范围内如何优先资源的新理解。
translated by 谷歌翻译
最近的工作表明,不同体系结构的卷积神经网络学会按照相同的顺序对图像进行分类。为了理解这种现象,我们重新审视了过度参数的深度线性网络模型。我们的分析表明,当隐藏层足够宽时,该模型参数的收敛速率沿数据的较大主组件的方向呈指数级数,该方向由由相应的奇异值控制的速率。我们称这种收敛模式主成分偏差(PC偏置)。从经验上讲,我们展示了PC偏差如何简化线性和非线性网络的学习顺序,在学习的早期阶段更为突出。然后,我们将结果与简单性偏见进行比较,表明可以独立看到这两个偏见,并以不同的方式影响学习顺序。最后,我们讨论了PC偏差如何解释早期停止及其与PCA的联系的一些好处,以及为什么深网与随机标签更慢地收敛。
translated by 谷歌翻译
Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the lottery ticket hypothesis: dense, randomly-initialized, feed-forward networks contain subnetworks (winning tickets) that-when trained in isolationreach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.
translated by 谷歌翻译
微调是深度学习的常见做法,使用相对较少的训练数据来实现卓越的普遍性导致下游任务。虽然在实践中广泛使用,但它缺乏强烈的理论理解。我们分析了若干架构中线性教师的回归的本方案的样本复杂性。直观地,微调的成功取决于源任务与目标任务之间的相似性,但是测量它是非微不足道的。我们表明相关措施考虑了源任务,目标任务和目标数据的协方差结构之间的关系。在线性回归的设置中,我们表明,在现实的情况下,当上述措施低时,在实际设置下,显着的样本复杂性降低是合理的。对于深线性回归,我们在用预制权重初始化网络时,我们提出了关于基于梯度训练的感应偏差的新颖结果。使用此结果,我们显示此设置的相似度量也受网络深度的影响。我们进一步在浅relu模型上显示结果,并分析了在源和目标任务中的样本复杂性的依赖性。我们经验证明了我们对合成和现实数据的结果。
translated by 谷歌翻译
在许多情况下,更简单的模型比更复杂的模型更可取,并且该模型复杂性的控制是机器学习中许多方法的目标,例如正则化,高参数调整和体系结构设计。在深度学习中,很难理解复杂性控制的潜在机制,因为许多传统措施并不适合深度神经网络。在这里,我们开发了几何复杂性的概念,该概念是使用离散的dirichlet能量计算的模型函数变异性的量度。使用理论论据和经验结果的结合,我们表明,许多常见的训练启发式方法,例如参数规范正规化,光谱规范正则化,平稳性正则化,隐式梯度正则化,噪声正则化和参数初始化的选择,都可以控制几何学复杂性,并提供一个统一的框架,以表征深度学习模型的行为。
translated by 谷歌翻译
神经网络的架构和参数通常独立优化,这需要每当修改体系结构时对参数的昂贵再次再次再次进行验证。在这项工作中,我们专注于在不需要昂贵的再培训的情况下越来越多。我们提出了一种在训练期间添加新神经元的方法,而不会影响已经学到的内容,同时改善了培训动态。我们通过最大化新重量的梯度来实现后者,并通过奇异值分解(SVD)有效地找到最佳初始化。我们称这种技术渐变最大化增长(Gradmax),并展示其各种视觉任务和架构的效力。
translated by 谷歌翻译
我们对深度学习的理论理解并没有与其经验成功保持同步。尽管已知网络体系结构至关重要,但我们尚不了解其对学习的表示和网络行为的影响,或者该体系结构如何反映任务结构。在这项工作中,我们开始通过引入门控的深层线性网络框架来解决此差距。这阐明了信息流的路径如何影响体系结构内的学习动态。至关重要的是,由于门控,这些网络可以计算其输入的非线性函数。我们得出了精确的减少,并且在某些情况下,我们可以确切解决学习动力学的方法。我们的分析表明,结构化网络中的学习动态可以概念化为具有隐性偏见的神经种族,然后控制模型的系统概括,多任务和转移的能力。我们通过自然主义数据集并使用轻松的假设来验证我们的关键见解。综上所述,我们的工作提出了将神经体系结构与学习有关的一般假设,并提供了一种数学方法,以理解更复杂的架构的设计以及模块化和组成性在解决现实世界中问题中的作用。代码和结果可在https://www.saxelab.org/gated-dln上找到。
translated by 谷歌翻译
培训深度神经网络是一项非常苛刻的任务,尤其是具有挑战性的是如何适应体系结构以提高训练有素的模型的性能。我们可以发现,有时,浅网络比深网概括得更好,并且增加更多层会导致更高的培训和测试错误。深层残留学习框架通过将跳过连接添加到几个神经网络层来解决此降解问题。最初,需要这种跳过连接才能成功地训练深层网络,因为网络的表达性会随着深度的指数增长而成功。在本文中,我们首先通过神经网络分析信息流。我们介绍和评估批处理循环,该批处理通过神经网络的每一层量化信息流。我们从经验和理论上证明,基于梯度下降的训练方法需要正面批处理融合,以成功地优化给定的损失功能。基于这些见解,我们引入了批处理凝聚正则化,以使基于梯度下降的训练算法能够单独通过每个隐藏层来优化信息流。借助批处理正则化,梯度下降优化器可以将不可吸引的网络转换为可训练的网络。我们从经验上表明,因此我们可以训练“香草”完全连接的网络和卷积神经网络 - 没有跳过连接,批处理标准化,辍学或任何其他建筑调整 - 只需将批处理 - 凝集正则术语添加到500层中损失功能。批处理 - 注入正则化的效果不仅在香草神经网络上评估,还评估了在各种计算机视觉以及自然语言处理任务上的剩余网络,自动编码器以及变压器模型上。
translated by 谷歌翻译
最近,已经观察到,转移学习解决方案可能是我们解决许多少量学习基准的全部 - 因此提出了有关何时以及如何部署元学习算法的重要问题。在本文中,我们试图通过1.提出一个新颖的指标(多样性系数)来阐明这些问题,以测量几次学习基准和2.的任务多样性。 )并在公平条件下进行学习(相同的体系结构,相同的优化器和所有经过培训的模型)。使用多样性系数,我们表明流行的迷你胶原和Cifar-fs几乎没有学习基准的多样性低。这种新颖的洞察力将转移学习解决方案比在公平比较的低多样性方面的元学习解决方案更好。具体而言,我们从经验上发现,低多样性系数与转移学习和MAML学习解决方案之间的高相似性在元测试时间和分类层相似性方面(使用基于特征的距离指标,例如SVCCA,PWCCA,CKA和OPD) )。为了进一步支持我们的主张,我们发现这种元测试的准确性仍然存在,即使模型大小变化也是如此。因此,我们得出的结论是,在低多样性制度中,MAML和转移学习在公平比较时具有等效的元检验性能。我们也希望我们的工作激发了对元学习基准测试基准的更周到的结构和定量评估。
translated by 谷歌翻译
We introduce a new representation learning approach for domain adaptation, in which data at training and test time come from similar but different distributions. Our approach is directly inspired by the theory on domain adaptation suggesting that, for effective domain transfer to be achieved, predictions must be made based on features that cannot discriminate between the training (source) and test (target) domains.The approach implements this idea in the context of neural network architectures that are trained on labeled data from the source domain and unlabeled data from the target domain (no labeled target-domain data is necessary). As the training progresses, the approach promotes the emergence of features that are (i) discriminative for the main learning task on the source domain and (ii) indiscriminate with respect to the shift between the domains. We show that this adaptation behaviour can be achieved in almost any feed-forward model by augmenting it with few standard layers and a new gradient reversal layer. The resulting augmented architecture can be trained using standard backpropagation and stochastic gradient descent, and can thus be implemented with little effort using any of the deep learning packages.We demonstrate the success of our approach for two distinct classification problems (document sentiment analysis and image classification), where state-of-the-art domain adaptation performance on standard benchmarks is achieved. We also validate the approach for descriptor learning task in the context of person re-identification application.
translated by 谷歌翻译
The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, auto-encoders, manifold learning, and deep networks. This motivates longer-term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation and manifold learning.
translated by 谷歌翻译
神经网络经常将许多无关的概念包装到一个神经元中 - 一种令人困惑的现象被称为“多疾病”,这使解释性更具挑战性。本文提供了一个玩具模型,可以完全理解多义,这是由于模型在“叠加”中存储其他稀疏特征的结果。我们证明了相变的存在,与均匀多型的几何形状的令人惊讶的联系以及与对抗性例子联系的证据。我们还讨论了对机械解释性的潜在影响。
translated by 谷歌翻译
With the ever-growing model size and the limited availability of labeled training data, transfer learning has become an increasingly popular approach in many science and engineering domains. For classification problems, this work delves into the mystery of transfer learning through an intriguing phenomenon termed neural collapse (NC), where the last-layer features and classifiers of learned deep networks satisfy: (i) the within-class variability of the features collapses to zero, and (ii) the between-class feature means are maximally and equally separated. Through the lens of NC, our findings for transfer learning are the following: (i) when pre-training models, preventing intra-class variability collapse (to a certain extent) better preserves the intrinsic structures of the input data, so that it leads to better model transferability; (ii) when fine-tuning models on downstream tasks, obtaining features with more NC on downstream data results in better test accuracy on the given task. The above results not only demystify many widely used heuristics in model pre-training (e.g., data augmentation, projection head, self-supervised learning), but also leads to more efficient and principled fine-tuning method on downstream tasks that we demonstrate through extensive experimental results.
translated by 谷歌翻译
Neural networks are known to be a class of highly expressive functions able to fit even random inputoutput mappings with 100% accuracy. In this work we present properties of neural networks that complement this aspect of expressivity. By using tools from Fourier analysis, we highlight a learning bias of deep networks towards low frequency functions -i.e. functions that vary globally without local fluctuations -which manifests itself as a frequency-dependent learning speed. Intuitively, this property is in line with the observation that over-parameterized networks prioritize learning simple patterns that generalize across data samples. We also investigate the role of the shape of the data manifold by presenting empirical and theoretical evidence that, somewhat counter-intuitively, learning higher frequencies gets easier with increasing manifold complexity.
translated by 谷歌翻译
低维歧管假设认为,在许多应用中发现的数据,例如涉及自然图像的数据(大约)位于嵌入高维欧几里得空间中的低维歧管上。在这种情况下,典型的神经网络定义了一个函数,该函数在嵌入空间中以有限数量的向量作为输入。但是,通常需要考虑在训练分布以外的点上评估优化网络。本文考虑了培训数据以$ \ mathbb r^d $的线性子空间分配的情况。我们得出对由神经网络定义的学习函数变化的估计值,沿横向子空间的方向。我们研究了数据歧管的编纂中与网络的深度和噪声相关的潜在正则化效应。由于存在噪声,我们还提出了训练中的其他副作用。
translated by 谷歌翻译
Recent work has sought to understand the behavior of neural networks by comparing representations between layers and between different trained models. We examine methods for comparing neural network representations based on canonical correlation analysis (CCA). We show that CCA belongs to a family of statistics for measuring multivariate similarity, but that neither CCA nor any other statistic that is invariant to invertible linear transformation can measure meaningful similarities between representations of higher dimension than the number of data points. We introduce a similarity index that measures the relationship between representational similarity matrices and does not suffer from this limitation. This similarity index is equivalent to centered kernel alignment (CKA) and is also closely connected to CCA. Unlike CCA, CKA can reliably identify correspondences between representations in networks trained from different initializations.
translated by 谷歌翻译