我们解决了对追踪预测的影响函数的高效计算，回到培训数据。我们提出并分析了一种新方法来加速基于Arnoldi迭代的反向Hessian计算。通过这种改进，我们实现了我们的知识，首次成功实施了影响功能，该函数的尺寸为全尺寸（语言和愿景）变压器模型，具有数十亿个参数。我们评估我们对图像分类和序列任务的方法，以百万次训练示例。我们的代码将在https://github.com/google-research/jax-influence上获得。

Good models require good training data. For overparameterized deep models, the causal relationship between training data and model predictions is increasingly opaque and poorly understood. Influence analysis partially demystifies training's underlying interactions by quantifying the amount each training instance alters the final model. Measuring the training data's influence exactly can be provably hard in the worst case; this has led to the development and use of influence estimators, which only approximate the true influence. This paper provides the first comprehensive survey of training data influence analysis and estimation. We begin by formalizing the various, and in places orthogonal, definitions of training data influence. We then organize state-of-the-art influence analysis methods into a taxonomy; we describe each of these methods in detail and compare their underlying assumptions, asymptotic complexities, and overall strengths and weaknesses. Finally, we propose future research directions to make influence analysis more useful in practice as well as more theoretically and empirically sound. A curated, up-to-date list of resources related to influence analysis is available at https://github.com/ZaydH/influence_analysis_papers.

我们研究了使用尖刺，现场依赖的随机矩阵理论研究迷你批次对深神经网络损失景观的影响。我们表明，批量黑森州的极值值的大小大于经验丰富的黑森州。我们还获得了类似的结果对Hessian的概括高斯牛顿矩阵近似。由于我们的定理，我们推导出作为批量大小的最大学习速率的分析表达式，为随机梯度下降（线性缩放）和自适应算法（例如ADAM（Square Root Scaling）提供了通知实际培训方案，例如光滑，非凸深神经网络。虽然随机梯度下降的线性缩放是在我们概括的更多限制性条件下导出的，但是适应优化者的平方根缩放规则是我们的知识，完全小说。随机二阶方法和自适应方法的百分比，我们得出了最小阻尼系数与学习率与批量尺寸的比率成比例。我们在Cifar-$ 100 $和ImageNet数据集上验证了我们的VGG / WimerEsnet架构上的索赔。根据我们对象检的调查，我们基于飞行学习率和动量学习者开发了一个随机兰齐齐竞争，这避免了对这些关键的超参数进行昂贵的多重评估的需求，并在预残留的情况下显示出良好的初步结果Cifar的architecure - $ 100 $。

最近的立法导致对机器学习的兴趣，即从预测模型中删除特定的培训样本，就好像它们在培训数据集中从未存在。由于损坏/对抗性数据或仅仅是用户的更新隐私要求，也可能需要进行学习。对于不需要培训的模型（K-NN），只需删除最近的原始样品即可有效。但是，这个想法不适合学习更丰富的表示的模型。由于模型维度D的趋势，最新的想法利用了基于优化的更新，因为损失函数的Hessian颠倒了。我们使用新的条件独立系数L-CODEC的变体来识别模型参数的子集，其语义重叠在单个样本级别上。我们的方法完全避免了将（可能）巨大矩阵倒置的必要性。通过利用马尔可夫毯子的选择，我们前提是l-codec也适合深度学习以及视觉中的其他应用。与替代方案相比，L-Codec在原本是不可行的设置中可以实现近似学习，包括用于面部识别的视觉模型，人重新识别和可能需要未经学习的样品进行排除的NLP模型。代码可以在https://github.com/vsingh-group/lcodec-deep-unlearning/

深度学习在广泛的AI应用方面取得了有希望的结果。较大的数据集和模型一致地产生更好的性能。但是，我们一般花费更长的培训时间，以更多的计算和沟通。在本调查中，我们的目标是在模型精度和模型效率方面提供关于大规模深度学习优化的清晰草图。我们调查最常用于优化的算法，详细阐述了大批量培训中出现的泛化差距的可辩论主题，并审查了解决通信开销并减少内存足迹的SOTA策略。

有针对性的训练集攻击将恶意实例注入训练集中，以导致训练有素的模型错误地标记一个或多个特定的测试实例。这项工作提出了目标识别的任务，该任务决定了特定的测试实例是否是训练集攻击的目标。目标识别可以与对抗性识别相结合，以查找（并删除）攻击实例，从而减轻对其他预测的影响，从而减轻攻击。我们没有专注于单个攻击方法或数据模式，而是基于影响力估计，这量化了每个培训实例对模型预测的贡献。我们表明，现有的影响估计量的不良实际表现通常来自于他们对训练实例和迭代次数的过度依赖。我们重新归一化的影响估计器解决了这一弱点。他们的表现远远超过了原始估计量，可以在对抗和非对抗环境中识别有影响力的训练示例群体，甚至发现多达100％的对抗训练实例，没有清洁数据误报。然后，目标识别简化以检测具有异常影响值的测试实例。我们证明了我们的方法对各种数据域的后门和中毒攻击的有效性，包括文本，视觉和语音，以及针对灰色盒子的自适应攻击者，该攻击者专门优化了逃避我们方法的对抗性实例。我们的源代码可在https://github.com/zaydh/target_indistification中找到。

我们开发了一种新的原则性算法，用于估计培训数据点对深度学习模型的行为的贡献，例如它做出的特定预测。我们的算法估计了AME，该数量量衡量了将数据点添加到训练数据子集中的预期（平均）边际效应，并从给定的分布中采样。当从均匀分布中采样子集时，AME将还原为众所周知的Shapley值。我们的方法受因果推断和随机实验的启发：我们采样了训练数据的不同子集以训练多个子模型，并评估每个子模型的行为。然后，我们使用套索回归来基于子集组成共同估计每个数据点的AME。在稀疏假设（$ k \ ll n $数据点具有较大的AME）下，我们的估计器仅需要$ O（k \ log n）$随机的子模型培训，从而改善了最佳先前的Shapley值估算器。

有效地近似损失函数的局部曲率信息是用于深神经网络的优化和压缩的关键工具。然而，大多数现有方法近似二阶信息具有高计算或存储成本，这可以限制其实用性。在这项工作中，我们调查矩阵，用于估计逆象征的矢量产品（IHVPS）的矩阵线性时间方法，因为当Hessian可以近似为乘语 - 一个矩阵的总和时，如Hessian的经典近似由经验丰富的Fisher矩阵。我们提出了两个新的算法作为称为M-FAC的框架的一部分：第一个算法朝着网络压缩量身定制，如果Hessian给出了M $等级的总和，则可以计算Dimension $ D $的IHVP。 ，使用$ O（DM ^ 2）$预压制，$ O（DM）$代价计算IHVP，并查询逆Hessian的任何单个元素的费用$ O（m）$。第二算法针对优化设置，我们希望在反向Hessian之间计算产品，估计在优化步骤的滑动窗口和给定梯度方向上，根据预先说明的SGD所需的梯度方向。我们为计算IHVP和OHVP和O（DM + M ^ 3）$ of $ o（dm + m ^ 2）$提供算法，以便从滑动窗口添加或删除任何渐变。这两种算法产生最先进的结果，用于网络修剪和相对于现有二阶方法的计算开销的优化。在[9]和[17]可用实现。

Transfer learning, where a model is first pre-trained on a data-rich task before being finetuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

How can we explain the predictions of a blackbox model? In this paper, we use influence functions -a classic technique from robust statistics -to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. To scale up influence functions to modern machine learning settings, we develop a simple, efficient implementation that requires only oracle access to gradients and Hessian-vector products. We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visuallyindistinguishable training-set attacks.

We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network's Fisher information matrix which is neither diagonal nor low-rank, and in some cases is completely non-sparse. It is derived by approximating various large blocks of the Fisher (corresponding to entire layers) as being the Kronecker product of two much smaller matrices. While only several times more expensive to compute than the plain stochastic gradient, the updates produced by K-FAC make much more progress optimizing the objective, which results in an algorithm that can be much faster than stochastic gradient descent with momentum in practice. And unlike some previously proposed approximate natural-gradient/Newton methods which use high-quality non-diagonal curvature matrices (such as Hessian-free optimization), K-FAC works very well in highly stochastic optimization regimes. This is because the cost of storing and inverting K-FAC's approximation to the curvature matrix does not depend on the amount of data used to estimate it, which is a feature typically associated only with diagonal or low-rank approximations to the curvature matrix.

我们呈现本地合奏，一种检测欠缺的方法 - 当许多可能的预测因子与训练数据和模型类一致 - 在预先训练的模型中的测试时间。我们的方法使用本地二阶信息来近似于来自同一类的模型的集合中的预测方差。我们通过估计与Hessian的低曲率方向对准的测试点梯度的组件的规范来计算该近似值，并提供一种估计该数量的易诊方法。在实验上，我们表明我们的方法能够检测到预先训练的模型在测试数据上产生了缺点，应用于分布外检测，检测杂散相关性和主动学习。

The behaviors of deep neural networks (DNNs) are notoriously resistant to human interpretations. In this paper, we propose Hypergradient Data Relevance Analysis, or HYDRA, which interprets the predictions made by DNNs as effects of their training data. Existing approaches generally estimate data contributions around the final model parameters and ignore how the training data shape the optimization trajectory. By unrolling the hypergradient of test loss w.r.t. the weights of training data, HYDRA assesses the contribution of training data toward test data points throughout the training trajectory. In order to accelerate computation, we remove the Hessian from the calculation and prove that, under moderate conditions, the approximation error is bounded. Corroborating this theoretical claim, empirical results indicate the error is indeed small. In addition, we quantitatively demonstrate that HYDRA outperforms influence functions in accurately estimating data contribution and detecting noisy data labels. The source code is available at https://github.com/cyyever/aaai_hydra_8686.

Continual Learning (CL) is a field dedicated to devise algorithms able to achieve lifelong learning. Overcoming the knowledge disruption of previously acquired concepts, a drawback affecting deep learning models and that goes by the name of catastrophic forgetting, is a hard challenge. Currently, deep learning methods can attain impressive results when the data modeled does not undergo a considerable distributional shift in subsequent learning sessions, but whenever we expose such systems to this incremental setting, performance drop very quickly. Overcoming this limitation is fundamental as it would allow us to build truly intelligent systems showing stability and plasticity. Secondly, it would allow us to overcome the onerous limitation of retraining these architectures from scratch with the new updated data. In this thesis, we tackle the problem from multiple directions. In a first study, we show that in rehearsal-based techniques (systems that use memory buffer), the quantity of data stored in the rehearsal buffer is a more important factor over the quality of the data. Secondly, we propose one of the early works of incremental learning on ViTs architectures, comparing functional, weight and attention regularization approaches and propose effective novel a novel asymmetric loss. At the end we conclude with a study on pretraining and how it affects the performance in Continual Learning, raising some questions about the effective progression of the field. We then conclude with some future directions and closing remarks.

用于估计模型不确定性的线性拉普拉斯方法在贝叶斯深度学习社区中引起了人们的重新关注。该方法提供了可靠的误差线，并接受模型证据的封闭式表达式，从而可以选择模型超参数。在这项工作中，我们检查了这种方法背后的假设，尤其是与模型选择结合在一起。我们表明，这些与一些深度学习的标准工具（构成近似方法和归一化层）相互作用，并为如何更好地适应这种经典方法对现代环境提出建议。我们为我们的建议提供理论支持，并在MLP，经典CNN，具有正常化层，生成性自动编码器和变压器的剩余网络上进行经验验证它们。

本文评价用机器学习问题的数值优化方法。由于机器学习模型是高度参数化的，我们专注于适合高维优化的方法。我们在二次模型上构建直觉，以确定哪种方法适用于非凸优化，并在凸函数上开发用于这种方法的凸起函数。随着随机梯度下降和动量方法的这种理论基础，我们试图解释为什么机器学习领域通常使用的方法非常成功。除了解释成功的启发式之外，最后一章还提供了对更多理论方法的广泛审查，这在实践中并不像惯例。所以在某些情况下，这项工作试图回答这个问题：为什么默认值中包含的默认TensorFlow优化器？

影响功能有效地估计了删除单个训练数据点对模型学习参数的影响。尽管影响估计值与线性模型的剩余重新进行了良好的重新对齐，但最近的作品表明，在神经网络中，这种比对通常很差。在这项工作中，我们通过将其分解为五个单独的术语来研究导致这种差异的特定因素。我们研究每个术语对各种架构和数据集的贡献，以及它们如何随网络宽度和培训时间等因素而变化。尽管实际影响函数估计值可能是非线性网络中保留对方的重新培训的差异，但我们表明它们通常是对不同对象的良好近似值，我们称其为近端Bregman响应函数（PBRF）。由于PBRF仍然可以用来回答许多激励影响功能的问题，例如识别有影响力或标记的示例，因此我们的结果表明，影响功能估计的当前算法比以前的错误分析所暗示的更有用的结果。

深度神经网络（DNN）的计算要求增加导致获得稀疏，且准确的DNN模型的兴趣。最近的工作已经调查了稀疏训练的更加困难的情况，其中DNN重量尽可能稀少，以减少训练期间的计算成本。现有的稀疏训练方法通常是经验的，并且可以具有相对于致密基线的准确性较低。在本文中，我们介绍了一种称为交替压缩/解压缩（AC / DC）训练DNN的一般方法，证明了算法变体的收敛，并表明AC / DC在类似的计算预算中准确地表现出现有的稀疏训练方法;在高稀疏水平下，AC / DC甚至优于现有的现有方法，依赖于准确的预训练密集模型。 AC / DC的一个重要属性是它允许联合培训密集和稀疏的型号，在训练过程结束时产生精确的稀疏密集模型对。这在实践中是有用的，其中压缩变体可能是为了在资源受限的设置中进行部署而不重新执行整个训练流，并且还为我们提供了深入和压缩模型之间的精度差距的见解。代码可在：https://github.com/ist-daslab/acdc。

二阶优化器被认为具有加快神经网络训练的潜力，但是由于曲率矩阵的尺寸巨大，它们通常需要近似值才能计算。最成功的近似家庭是Kronecker因块状曲率估计值（KFAC）。在这里，我们结合了先前工作的工具，以评估确切的二阶更新和仔细消融以建立令人惊讶的结果：由于其近似值，KFAC与二阶更新无关，尤其是，它极大地胜过真实的第二阶段更新。订单更新。这一挑战广泛地相信，并立即提出了为什么KFAC表现如此出色的问题。为了回答这个问题，我们提出了强烈的证据，表明KFAC近似于一阶算法，该算法在神经元上执行梯度下降而不是权重。最后，我们表明，这种优化器通常会在计算成本和数据效率方面改善KFAC。

Multilayer Neural Networks trained with the backpropagation algorithm constitute the best example of a successful Gradient-Based Learning technique. Given an appropriate network architecture, Gradient-Based Learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional Neural Networks, that are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques.Real-life document recognition systems are composed of multiple modules including eld extraction, segmentation, recognition, and language modeling. A new learning paradigm, called Graph Transformer Networks (GTN), allows such multi-module systems to be trained globally using Gradient-Based methods so as to minimize an overall performance measure.Two systems for on-line handwriting recognition are described. Experiments demonstrate the advantage of global training, and the exibility of Graph Transformer Networks.A Graph Transformer Network for reading bank check is also described. It uses Convolutional Neural Network character recognizers combined with global training techniques to provides record accuracy on business and personal checks. It is deployed commercially and reads several million checks per day.