We propose a notation for tensors with named axes, which relieves the author, reader, and future implementers of machine learning models from the burden of keeping track of the order of axes and the purpose of each. The notation makes it easy to lift operations on low-order tensors to higher order ones, for example, from images to minibatches of images, or from an attention mechanism to multiple attention heads. After a brief overview and formal definition of the notation, we illustrate it through several examples from modern machine learning, from building blocks like attention and convolution to full models like Transformers and LeNet. We then discuss differential calculus in our notation and compare with some alternative notations. Our proposals build on ideas from many previous papers and software libraries. We hope that our notation will encourage more authors to use named tensors, resulting in clearer papers and more precise implementations.
translated by 谷歌翻译
我们基于功能分析中的分类结构开发了一种自动和符号分化的组成方法,其中衍生物是抽象向量上的线性函数,而不是限于标量,向量,矩阵或张力器,表示为多维阵列。我们表明,可以使用差分计算来实现符号和自动分化,以生成基于原始,恒定,线性和双线性函数的规则以及其顺序和并行组成的线性函数。线性函数以组合域特异性语言表示。最后,我们提供了一个微积分,用于象征性地计算衍生物的伴随,而无需使用矩阵,而矩阵过于效率低,无法在高维空间上使用。衍生物的最终符号表示保留了输入程序的数据并行操作。组合分化和计算形式的伴随的组合在行为上等同于反向模式自动分化。特别是,它为矩阵过于效率而无法表示线性功能的优化提供了机会。
translated by 谷歌翻译
我们在真实和复杂的环境中介绍了前向和向后模式广告的经典无坐标形式主义。我们展示了如何从基本原理开始的许多矩阵函数正式得出前向和向后公式。
translated by 谷歌翻译
We present a new algorithm for automatically bounding the Taylor remainder series. In the special case of a scalar function $f: \mathbb{R} \mapsto \mathbb{R}$, our algorithm takes as input a reference point $x_0$, trust region $[a, b]$, and integer $k \ge 0$, and returns an interval $I$ such that $f(x) - \sum_{i=0}^k \frac {f^{(i)}(x_0)} {i!} (x - x_0)^i \in I (x - x_0)^{k+1}$ for all $x \in [a, b]$. As in automatic differentiation, the function $f$ is provided to the algorithm in symbolic form, and must be composed of known elementary functions. At a high level, our algorithm has two steps. First, for a variety of commonly-used elementary functions (e.g., $\exp$, $\log$), we derive sharp polynomial upper and lower bounds on the Taylor remainder series. We then recursively combine the bounds for the elementary functions using an interval arithmetic variant of Taylor-mode automatic differentiation. Our algorithm can make efficient use of machine learning hardware accelerators, and we provide an open source implementation in JAX. We then turn our attention to applications. Most notably, we use our new machinery to create the first universal majorization-minimization optimization algorithms: algorithms that iteratively minimize an arbitrary loss using a majorizer that is derived automatically, rather than by hand. Applied to machine learning, this leads to architecture-specific optimizers for training deep networks that converge from any starting point, without hyperparameter tuning. Our experiments show that for some optimization problems, these hyperparameter-free optimizers outperform tuned versions of gradient descent, Adam, and AdaGrad. We also show that our automatically-derived bounds can be used for verified global optimization and numerical integration, and to prove sharper versions of Jensen's inequality.
translated by 谷歌翻译
These notes were compiled as lecture notes for a course developed and taught at the University of the Southern California. They should be accessible to a typical engineering graduate student with a strong background in Applied Mathematics. The main objective of these notes is to introduce a student who is familiar with concepts in linear algebra and partial differential equations to select topics in deep learning. These lecture notes exploit the strong connections between deep learning algorithms and the more conventional techniques of computational physics to achieve two goals. First, they use concepts from computational physics to develop an understanding of deep learning algorithms. Not surprisingly, many concepts in deep learning can be connected to similar concepts in computational physics, and one can utilize this connection to better understand these algorithms. Second, several novel deep learning algorithms can be used to solve challenging problems in computational physics. Thus, they offer someone who is interested in modeling a physical phenomena with a complementary set of tools.
translated by 谷歌翻译
十年自2010年以来,人工智能成功一直处于计算机科学和技术的最前沿,传染媒介空间模型已经巩固了人工智能最前沿的位置。与此同时,量子计算机已经变得更加强大,主要进步的公告经常在新闻中。这些区域的基础的数学技术比有时意识到更多的共同之处。传染媒介空间在20世纪30年代的量子力学的公理心脏上采取了位置,这一采用是从矢量空间的线性几何形状推导逻辑和概率的关键动机。粒子之间的量子相互作用是使用张量产品进行建模的,其也用于表达人工神经网络中的物体和操作。本文介绍了这些常见的数学区域中的一些,包括如何在人工智能(AI)中使用的示例,特别是在自动推理和自然语言处理(NLP)中。讨论的技术包括矢量空间,标量产品,子空间和含义,正交投影和否定,双向矩阵,密度矩阵,正算子和张量产品。应用领域包括信息检索,分类和含义,建模字传感和歧义,知识库的推断和语义构成。其中一些方法可能会在量子硬件上实现。该实施中的许多实际步骤都处于早期阶段,其中一些已经实现了。解释一些常见的数学工具可以帮助AI和量子计算中的研究人员进一步利用这些重叠,识别和沿途探索新方向。
translated by 谷歌翻译
这项正在进行的工作旨在为统计学习提供统一的介绍,从诸如GMM和HMM等经典模型到现代神经网络(如VAE和扩散模型)缓慢地构建。如今,有许多互联网资源可以孤立地解释这一点或新的机器学习算法,但是它们并没有(也不能在如此简短的空间中)将这些算法彼此连接起来,或者与统计模型的经典文献相连现代算法出现了。同样明显缺乏的是一个单一的符号系统,尽管对那些已经熟悉材料的人(如这些帖子的作者)不满意,但对新手的入境造成了重大障碍。同样,我的目的是将各种模型(尽可能)吸收到一个用于推理和学习的框架上,表明(以及为什么)如何以最小的变化将一个模型更改为另一个模型(其中一些是新颖的,另一些是文献中的)。某些背景当然是必要的。我以为读者熟悉基本的多变量计算,概率和统计以及线性代数。这本书的目标当然不是​​完整性,而是从基本知识到过去十年中极强大的新模型的直线路径或多或少。然后,目标是补充而不是替换,诸如Bishop的\ emph {模式识别和机器学习}之类的综合文本,该文本现在已经15岁了。
translated by 谷歌翻译
在过去十年中,已经开发出新的深度学习(DL)算法,工作负载和硬件来解决各种问题。尽管工作量和硬件生态系统的进步,DL系统的编程方法是停滞不前的。 DL工作负载从DL库中的高度优化,特定于平台和不灵活的内核,或者在新颖的操作员的情况下,通过具有强大性能的DL框架基元建立参考实现。这项工作介绍了Tensor加工基元(TPP),一个编程抽象,用于高效的DL工作负载的高效,便携式实现。 TPPS定义了一组紧凑而多才多艺的2D张镜操作员(或虚拟张量ISA),随后可以用作构建块,以在高维张量上构建复杂的运算符。 TPP规范是平台 - 不可行的,因此通过TPPS表示的代码是便携式的,而TPP实现是高度优化的,并且特定于平台。我们展示了我们使用独立内核和端到端DL&HPC工作负载完全通过TPPS表达的方法的效力和生存性,这在多个平台上优于最先进的实现。
translated by 谷歌翻译
神经切线核(NTK),定义为$ \ theta_ \ theta^f(x_1,x_2)= \ left [\ partial f(\ theta,x_1)\ big/\ big/\ partial \ partial \ theta \ theta \ the f(\ theta,x_2)\ big/\ partial \ theta \ right]^t $ where $ \ weft [\ partial f(\ theta,\ cdot)\ big/\ big/\ partial \ theta \ right] $是一个神经网络(nn)雅各布(Jacobian)已成为深度学习研究的核心研究对象。在无限宽度极限中,有时可以通过分析计算NTK,对于理解NN体系结构的训练和概括很有用。在有限的宽度下,NTK还用于更好地初始化NN,比较跨模型,执行体系结构搜索并进行元学习。不幸的是,众所周知,有限的宽度NTK计算昂贵,这严重限制了其实际实用程序。我们对有限宽度网络中NTK计算的计算和内存需求进行了第一个深入分析。利用神经网络的结构,我们进一步提出了两种新颖的算法,这些算法改变了有限宽度NTK的计算和内存要求的指数,从而极大地提高了效率。我们的算法可以以黑匣子方式应用于任何可区分功能,包括实现神经网络的功能。我们在https://github.com/google/neural-tangents的神经切线包(ARXIV:1912.02803)中开放我们的实现。
translated by 谷歌翻译
对称性一直是探索广泛复杂系统的基本工具。在机器学习中,在模型和数据中都探索了对称性。在本文中,我们试图将模型家族架构引起的对称性与该家族的内部数据表示的对称性联系起来。我们通过计算一组基本的对称组来做到这一点,我们称它们称为模型的\ emph {Intertwiner组}。这些中的每一个都来自模型的特定非线性层,不同的非线性导致不同的对称组。这些组以模型的权重更改模型的权重,使模型所代表的基础函数保持恒定,但模型内部数据的内部表示可能会改变。我们通过一系列实验将Intertwiner组连接到模型的数据内部表示,这些实验在具有相同体系结构的模型之间探测隐藏状态之间的相似性。我们的工作表明,网络的对称性在该网络的数据表示中传播到对称性中,从而使我们更好地了解架构如何影响学习和预测过程。最后,我们推测,对于Relu网络,交织组可能会为在隐藏层而不是任意线性组合的激活基础上集中模型可解释性探索的共同实践提供理由。
translated by 谷歌翻译
We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness. To approximate softmax attentionkernels, Performers use a novel Fast Attention Via positive Orthogonal Random features approach (FAVOR+), which may be of independent interest for scalable kernel methods. FAVOR+ can also be used to efficiently model kernelizable attention mechanisms beyond softmax. This representational power is crucial to accurately compare softmax with other kernels for the first time on large-scale tasks, beyond the reach of regular Transformers, and investigate optimal attention-kernels. Performers are linear architectures fully compatible with regular Transformers and with strong theoretical guarantees: unbiased or nearly-unbiased estimation of the attention matrix, uniform convergence and low estimation variance. We tested Performers on a rich set of tasks stretching from pixel-prediction through text models to protein sequence modeling. We demonstrate competitive results with other examined efficient sparse and dense attention methods, showcasing effectiveness of the novel attention-learning paradigm leveraged by Performers.
translated by 谷歌翻译
Automatic differentiation (AD) is a technique for computing the derivative of a function represented by a program. This technique is considered as the de-facto standard for computing the differentiation in many machine learning and optimisation software tools. Despite the practicality of this technique, the performance of the differentiated programs, especially for functional languages and in the presence of vectors, is suboptimal. We present an AD system for a higher-order functional array-processing language. The core functional language underlying this system simultaneously supports both source-to-source forward-mode AD and global optimisations such as loop transformations. In combination, gradient computation with forward-mode AD can be as efficient as reverse mode, and the Jacobian matrices required for numerical algorithms such as Gauss-Newton and Levenberg-Marquardt can be efficiently computed.
translated by 谷歌翻译
Pre-publication draft of a book to be published byMorgan & Claypool publishers. Unedited version released with permission. All relevant copyrights held by the author and publisher extend to this pre-publication draft.
translated by 谷歌翻译
Recent trends of incorporating attention mechanisms in vision have led researchers to reconsider the supremacy of convolutional layers as a primary building block. Beyond helping CNNs to handle long-range dependencies, Ramachandran et al. (2019) showed that attention can completely replace convolution and achieve state-of-the-art performance on vision tasks. This raises the question: do learned attention layers operate similarly to convolutional layers? This work provides evidence that attention layers can perform convolution and, indeed, they often learn to do so in practice. Specifically, we prove that a multi-head self-attention layer with sufficient number of heads is at least as expressive as any convolutional layer. Our numerical experiments then show that self-attention layers attend to pixel-grid patterns similarly to CNN layers, corroborating our analysis. Our code is publicly available 1 .
translated by 谷歌翻译
变形金刚是今天最重要的机器学习工作负载之一。培训是一个非常计算密集的任务,通常需要几天或几周,并且对优化变压器进行了重大关注。尽管如此,现有的实现不会有效地利用GPU。我们发现数据移动是培训时的关键瓶颈。由于Amdahl的法律和大规模改进的计算性能,培训现已成为记忆束缚。此外,现有框架使用次优数据布局。使用这些洞察力,我们提供了一个用于全局优化变压器数据移动的配方。我们将数据移动降低到22.91%,总体上实现了在训练伯特编码器层和1.19x的整个伯特的最先进框架上的1.30倍的性能改进。我们的方法更广泛地适用于优化深神经网络,并深入了解如何解决新兴的性能瓶颈。
translated by 谷歌翻译
由于其宽度趋于无穷大,如果梯度下降下的深度神经网络的行为可以简化和可预测(例如,如果神经切线核(NTK)给出,则如果适当地进行了参数化(例如,NTK参数化)。但是,我们表明,神经网络的标准和NTK参数化不接受可以学习特征的无限宽度限制,这对于训练和转移学习至关重要。我们对标准参数化提出了简单的修改,以允许在极限内进行特征学习。使用 * Tensor程序 *技术,我们为此类限制提供了明确的公式。在Word2Vec和Omniglot上通过MAML进行的几杆学习,这是两个依赖特征学习的规范任务,我们准确地计算了这些限制。我们发现它们的表现都优于NTK基准和有限宽度网络,后者接近无限宽度的特征学习表现,随着宽度的增加。更普遍地,我们对神经网络参数化的自然空间进行分类,该空间概括了标准,NTK和平均场参数化。我们显示1)该空间中的任何参数化都可以接受特征学习或具有内核梯度下降给出的无限宽度训练动力学,但并非两者兼而有之; 2)可以使用Tensor程序技术计算任何此类无限宽度限制。可以在github.com/edwardjhu/tp4上找到我们的实验代码。
translated by 谷歌翻译
即使机器学习算法已经在数据科学中发挥了重要作用,但许多当前方法对输入数据提出了不现实的假设。由于不兼容的数据格式,或数据集中的异质,分层或完全缺少的数据片段,因此很难应用此类方法。作为解决方案,我们提出了一个用于样本表示,模型定义和培训的多功能,统一的框架,称为“ Hmill”。我们深入审查框架构建和扩展的机器学习的多个范围范式。从理论上讲,为HMILL的关键组件的设计合理,我们将通用近似定理的扩展显示到框架中实现的模型所实现的所有功能的集合。本文还包含有关我们实施中技术和绩效改进的详细讨论,该讨论将在MIT许可下发布供下载。该框架的主要资产是其灵活性,它可以通过相同的工具对不同的现实世界数据源进行建模。除了单独观察到每个对象的一组属性的标准设置外,我们解释了如何在框架中实现表示整个对象系统的图表中的消息推断。为了支持我们的主张,我们使用框架解决了网络安全域的三个不同问题。第一种用例涉及来自原始网络观察结果的IoT设备识别。在第二个问题中,我们研究了如何使用以有向图表示的操作系统的快照可以对恶意二进制文件进行分类。最后提供的示例是通过网络中实体之间建模域黑名单扩展的任务。在所有三个问题中,基于建议的框架的解决方案可实现与专业方法相当的性能。
translated by 谷歌翻译
每个已知的人工深神经网络(DNN)都对应于规范Grothendieck的拓扑中的一个物体。它的学习动态对应于此拓扑中的形态流动。层中的不变结构(例如CNNS或LSTMS)对应于Giraud的堆栈。这种不变性应该是对概括属性的原因,即从约束下的学习数据中推断出来。纤维代表语义前类别(Culioli,Thom),在该类别上定义了人工语言,内部逻辑,直觉主义者,古典或线性(Girard)。网络的语义功能是其能够用这种语言表达理论的能力,以回答输出数据中有关输出的问题。语义信息的数量和空间是通过类比与2015年香农和D.Bennequin的Shannon熵的同源解释来定义的。他们概括了Carnap和Bar-Hillel(1952)发现的措施。令人惊讶的是,上述语义结构通过封闭模型类别的几何纤维对象进行了分类,然后它们产生了DNNS及其语义功能的同位不变。故意类型的理论(Martin-Loef)组织了这些物体和它们之间的纤维。 Grothendieck的导数分析了信息内容和交流。
translated by 谷歌翻译
Graph neural networks (GNNs) are widely used for modeling complex interactions between entities represented as vertices of a graph. Despite recent efforts to theoretically analyze the expressive power of GNNs, a formal characterization of their ability to model interactions is lacking. The current paper aims to address this gap. Formalizing strength of interactions through an established measure known as separation rank, we quantify the ability of certain GNNs to model interaction between a given subset of vertices and its complement, i.e. between sides of a given partition of input vertices. Our results reveal that the ability to model interaction is primarily determined by the partition's walk index -- a graph-theoretical characteristic that we define by the number of walks originating from the boundary of the partition. Experiments with common GNN architectures corroborate this finding. As a practical application of our theory, we design an edge sparsification algorithm named Walk Index Sparsification (WIS), which preserves the ability of a GNN to model interactions when input edges are removed. WIS is simple, computationally efficient, and markedly outperforms alternative methods in terms of induced prediction accuracy. More broadly, it showcases the potential of improving GNNs by theoretically analyzing the interactions they can model.
translated by 谷歌翻译
神经网络是强大的功能估计器,导致其作为建模结构化数据的首选范式的地位。但是,与其他强调问题模块化的结构化表示不同,例如因子图 - 神经网络通常是从输入到输出的单片映射,并具有固定的计算顺序。这种限制阻止他们捕获模型变量之间的不同计算方向和相互作用。在本文中,我们结合了因子图和神经网络的代表性强度,提出了无向神经网络(UNNS):一个灵活的框架,用于指定可以按任何顺序执行的计算。对于特定的选择,我们提出的模型集成并扩展了许多现有的架构:带有隐式层的馈电,经常性,自我发项网络,自动编码器和网络。我们在一系列任务上展示了无方向性的神经体系结构的有效性:受树约束依赖性解析,卷积图像分类和序列完成。通过改变计算顺序,我们展示了如何同时将单个UNN用作分类器和原型发生器,以及它如何填充输入序列的缺失部分,从而使它们成为进一步研究的有希望的领域。
translated by 谷歌翻译