最近已证明状态空间模型（SSM）是深度学习层非常有效的，它是序列模型（例如RNN，CNN或变压器）的有前途替代方案。第一个显示这种潜力的版本是S4模型，它通过使用称为HIPPO矩阵的规定状态矩阵对涉及长期依赖性的任务特别有效。尽管这具有可解释的数学机制来建模长期依赖性，但它引入了一种自定义表示和算法，可能难以实施。另一方面，最新的S4变体称为DSS，表明将状态矩阵完全对角线限制在使用基于近似S4矩阵的特定初始化时，仍然可以保留原始模型的性能。这项工作旨在系统地了解如何参数化和初始化此类对角线状态空间模型。虽然从经典的结果来看，几乎所有SSM都具有等效的对角线形式，但我们表明初始化对于性能至关重要。我们通过证明S4矩阵的对角线限制出人意料地在无限状态尺寸的极限中恢复了相同的内核来解释为什么DSS在数学上起作用。我们还系统地描述了参数化和计算对角线SSM的各种设计选择，并执行对这些选择的影响的受控经验研究。我们的最终型号S4D是S4的简单对角线版本，其内核计算仅需要2行代码，并且几乎在所有设置中都与S4相当地执行，并具有最新的图像，音频和医疗时间序列域的结果，在远程竞技场基准中平均为85％。

序列建模的一个中心目标是设计一个单个原则模型，该模型可以解决各种方式和任务，尤其是在远程依赖方面的序列数据。尽管包括RNN，CNN和Transformers在内的传统模型具有用于捕获长期依赖性的专业变体，但它们仍然很难扩展到长时间的10000美元或更多步骤。通过模拟基本状态空间模型（SSM）\（x'（t）= ax（t）= ax（t） + bu（t），y（t）= cx（t） + du（t） + du（t）\ ), and showed that for appropriate choices of the state matrix \( A \), this system could handle long-range dependencies mathematically and empirically.但是，该方法具有过度的计算和内存需求，使其无法作为一般序列建模解决方案。我们根据SSM的新参数化提出了结构化状态空间序列模型（S4），并表明它可以比以前的方法更有效地计算出其理论强度。我们的技术涉及对\（a \）进行低级校正的调节，从而使其对角度稳定，并将SSM降低到库奇内核的精心研究的计算中。 S4在各种既定的基准测试范围内取得了强劲的经验结果，包括（i）在顺序CIFAR-10上的91 \％精度，没有数据增强或辅助损失，与较大的2-D Resnet相当，（ii）实质上关闭。在图像和语言建模任务上与变形金刚的差距，同时在远程竞技场基准的每个任务上执行每一代$ 60 \ times $ $（iii）sota，包括求解所有先前工作的挑战性path-x任务，而所有先前工作的长度为16K，同时与所有竞争对手一样高效。

有效地对远程依赖性建模是序列建模的重要目标。最近，使用结构化状态空间序列（S4）层的模型在许多远程任务上实现了最先进的性能。 S4层将线性状态空间模型（SSM）与深度学习技术结合在一起，并利用HIPPO框架进行在线功能近似以实现高性能。但是，该框架导致了架构约束和计算困难，使S4方法变得复杂，可以理解和实施。我们重新审视这样的想法，即遵循河马框架对于高性能是必要的。具体而言，我们替换了许多独立的单输入单输出（SISO）SSM的库S4层与一个多输入的多输出（MIMO）SSM一起使用，并具有降低的潜在尺寸。 MIMO系统的缩小潜在维度允许使用有效的并行扫描，从而简化了将S5层应用于序列到序列转换所需的计算。此外，我们将S5 SSM的状态矩阵初始化，其近似与S4 SSMS使用的河马级矩阵近似，并表明这是MIMO设置的有效初始化。 S5与S4在远程任务上的表现相匹配，包括在远程竞技场基准的套件中平均达到82.46％，而S4的80.48％和最佳的变压器变体的61.41％。

线性时间不变的状态空间模型（SSM）是工程和统计数据的经典模型，最近通过结构化状态空间序列模型（S4）证明，在机器学习中非常有前途。 S4的核心成分涉及将SSM状态矩阵初始化为称为HIPPO矩阵的特定矩阵，这对于S4处理长序列的能力在经验上很重要。但是，S4使用的特定矩阵实际上是在特定时间变化的动态系统中得出的，并且将此矩阵用作时间不变的SSM没有已知的数学解释。因此，S4模拟远程依赖性的理论机制实际上仍无法解释。我们得出了河马框架的更一般和直观的公式，该框架将S4作为对指数型的Legendre多项式的分解提供了简单的数学解释，解释了其捕获长依赖性的能力。我们的概括引入了理论上丰富的SSM类，还使我们能够为其他碱基（例如傅立叶基础）得出更直观的S4变体，并解释了训练S4的其他方面，例如如何初始化重要的时间表参数。这些见解将S4的性能提高到远程竞技场基准的86％，在最困难的Path-X任务中，S4的性能为96％。

线性状态空间模型（SSM）的状态过渡矩阵的适当参数化，然后是标准非线性，使他们能够从顺序数据中有效地学习表示形式，从。在本文中，我们表明，当线性液体时恒定（LTC）状态空间模型给出诸如S4之类的结构SSM时，我们可以进一步改善。 LTC神经网络是带有输入依赖性状态过渡模块的因果连续神经网络，这使他们学会在推理时适应传入的输入。我们表明，通过使用对角和S4中引入的状态过渡矩阵的对角线加低级分解以及一些简化的基于LTC的结构状态空间模型（称为Liquid-S4）实现了新的最新最先进的最先进跨序列建模任务具有长期依赖性（例如图像，文本，音频和医疗时间序列）的艺术概括，在远程竞技场基准中的平均性能为87.32％。在完整的原始语音命令识别中，数据集Liquid-S4的精度达到96.78％，与S4相比，参数计数降低了30％。性能的额外增益是液体-S4的核结构的直接结果，该结构考虑了训练和推理过程中输入序列样本的相似性。

状态空间模型已显示在建模远距离依赖性方面有效，特别是序列分类任务。在这项工作中，我们着重于对英语书籍，GitHub源代码和Arxiv数学文章的自回旋序列建模。基于围绕封闭激活功能的有效性的最新发展，我们提出了一个名为“封闭状态空间（GSS）”的新层，并表明它的训练速度明显快于TPU的S4（即DSS）的对角线版本，具有相当竞争力 - 基于变压器的基线，并表现出零击向更长的输入，同时直接实施。最后，我们表明，利用自我意见来建模局部依赖性，可以进一步提高GSS的性能。

神经网络的经典发展主要集中在有限维欧基德空间或有限组之间的学习映射。我们提出了神经网络的概括，以学习映射无限尺寸函数空间之间的运算符。我们通过一类线性积分运算符和非线性激活函数的组成制定运营商的近似，使得组合的操作员可以近似复杂的非线性运算符。我们证明了我们建筑的普遍近似定理。此外，我们介绍了四类运算符参数化：基于图形的运算符，低秩运算符，基于多极图形的运算符和傅里叶运算符，并描述了每个用于用每个计算的高效算法。所提出的神经运营商是决议不变的：它们在底层函数空间的不同离散化之间共享相同的网络参数，并且可以用于零击超分辨率。在数值上，与现有的基于机器学习的方法，达西流程和Navier-Stokes方程相比，所提出的模型显示出卓越的性能，而与传统的PDE求解器相比，与现有的基于机器学习的方法有关的基于机器学习的方法。

Sequence models based on linear state spaces (SSMs) have recently emerged as a promising choice of architecture for modeling long range dependencies across various modalities. However, they invariably rely on discretization of a continuous state space, which complicates their presentation and understanding. In this work, we dispose of the discretization step, and propose a model based on vanilla Diagonal Linear RNNs ($\mathrm{DLR}$). We empirically show that $\mathrm{DLR}$ is as performant as previously-proposed SSMs in the presence of strong supervision, despite being conceptually much simpler. Moreover, we characterize the expressivity of SSMs (including $\mathrm{DLR}$) and attention-based models via a suite of $13$ synthetic sequence-to-sequence tasks involving interactions over tens of thousands of tokens, ranging from simple operations, such as shifting an input sequence, to detecting co-dependent visual features over long spatial ranges in flattened images. We find that while SSMs report near-perfect performance on tasks that can be modeled via $\textit{few}$ convolutional kernels, they struggle on tasks requiring $\textit{many}$ such kernels and especially when the desired sequence manipulation is $\textit{context-dependent}$. For example, $\mathrm{DLR}$ learns to perfectly shift a $0.5M$-long input by an arbitrary number of positions but fails when the shift size depends on context. Despite these limitations, $\mathrm{DLR}$ reaches high performance on two higher-order reasoning tasks $\mathrm{ListOpsSubTrees}$ and $\mathrm{PathfinderSegmentation}\text{-}\mathrm{256}$ with input lengths $8K$ and $65K$ respectively, and gives encouraging performance on $\mathrm{PathfinderSegmentation}\text{-}\mathrm{512}$ with input length $262K$ for which attention is not a viable choice.

过度分辨的神经网络概括井，但训练昂贵。理想情况下，人们希望减少其计算成本，同时保留其概括的益处。稀疏的模型培训是实现这一目标的简单和有希望的方法，但随着现有方法与准确性损失，慢速训练运行时的困难或困难，仍然存在挑战，仍然存在困难的挑战。核心问题是，在离散的一组稀疏矩阵上搜索稀疏性掩模是困难和昂贵的。为了解决此问题，我们的主要见解是通过具有称为蝴蝶矩阵产品的固定结构的固定结构来优化优化稀疏矩阵的连续超集。随着蝴蝶矩阵不是硬件效率，我们提出了简单的蝴蝶（块和平坦）的变体来利用现代硬件。我们的方法（像素化蝴蝶）使用基于扁平块蝴蝶和低秩矩阵的简单固定稀疏模式，以缩小大多数网络层（例如，注意，MLP）。我们经验验证了像素化蝴蝶比蝴蝶快3倍，加快培训，以实现有利的准确性效率权衡。在ImageNet分类和Wikitext-103语言建模任务中，我们的稀疏模型训练比致密的MLP - 混频器，视觉变压器和GPT-2媒体更快地训练高达2.5倍，没有精确下降。

Low-rank matrix approximations, such as the truncated singular value decomposition and the rank-revealing QR decomposition, play a central role in data analysis and scientific computing. This work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low-rank matrix approximation. These techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets.This paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions. These methods use random sampling to identify a subspace that captures most of the action of a matrix. The input matrix is then compressed-either explicitly or implicitly-to this subspace, and the reduced matrix is manipulated deterministically to obtain the desired low-rank factorization. In many cases, this approach beats its classical competitors in terms of accuracy, speed, and robustness. These claims are supported by extensive numerical experiments and a detailed error analysis.The specific benefits of randomized techniques depend on the computational environment. Consider the model problem of finding the k dominant components of the singular value decomposition of an m × n matrix. (i) For a dense input matrix, randomized algorithms require O(mn log(k)) floating-point operations (flops) in contrast with O(mnk) for classical algorithms. (ii) For a sparse input matrix, the flop count matches classical Krylov subspace methods, but the randomized approach is more robust and can easily be reorganized to exploit multi-processor architectures. (iii) For a matrix that is too large to fit in fast memory, the randomized techniques require only a constant number of passes over the data, as opposed to O(k) passes for classical algorithms. In fact, it is sometimes possible to perform matrix approximation with a single pass over the data.

在许多学科中，动态系统的数据信息预测模型的开发引起了广泛的兴趣。我们提出了一个统一的框架，用于混合机械和机器学习方法，以从嘈杂和部分观察到的数据中识别动态系统。我们将纯数据驱动的学习与混合模型进行比较，这些学习结合了不完善的域知识。我们的公式与所选的机器学习模型不可知，在连续和离散的时间设置中都呈现，并且与表现出很大的内存和错误的模型误差兼容。首先，我们从学习理论的角度研究无内存线性（W.R.T.参数依赖性）模型误差，从而定义了过多的风险和概括误差。对于沿阵行的连续时间系统，我们证明，多余的风险和泛化误差都通过与T的正方形介于T的术语（指定训练数据的时间间隔）的术语界定。其次，我们研究了通过记忆建模而受益的方案，证明了两类连续时间复发性神经网络（RNN）的通用近似定理：两者都可以学习与内存有关的模型误差。此外，我们将一类RNN连接到储层计算，从而将学习依赖性错误的学习与使用随机特征在Banach空间之间进行监督学习的最新工作联系起来。给出了数值结果（Lorenz '63，Lorenz '96多尺度系统），以比较纯粹的数据驱动和混合方法，发现混合方法较少，渴望数据较少，并且更有效。最后，我们从数值上证明了如何利用数据同化来从嘈杂，部分观察到的数据中学习隐藏的动态，并说明了通过这种方法和培训此类模型来表示记忆的挑战。

We introduce a novel gated recurrent unit (GRU) with a weighted time-delay feedback mechanism in order to improve the modeling of long-term dependencies in sequential data. This model is a discretized version of a continuous-time formulation of a recurrent unit, where the dynamics are governed by delay differential equations (DDEs). By considering a suitable time-discretization scheme, we propose $\tau$-GRU, a discrete-time gated recurrent unit with delay. We prove the existence and uniqueness of solutions for the continuous-time model, and we demonstrate that the proposed feedback mechanism can help improve the modeling of long-term dependencies. Our empirical results show that $\tau$-GRU can converge faster and generalize better than state-of-the-art recurrent units and gated recurrent architectures on a range of tasks, including time-series classification, human activity recognition, and speech recognition.

State space models (SSMs) have demonstrated state-of-the-art sequence modeling performance in some modalities, but underperform attention in language modeling. Moreover, despite scaling nearly linearly in sequence length instead of quadratically, SSMs are still slower than Transformers due to poor hardware utilization. In this paper, we make progress on understanding the expressivity gap between SSMs and attention in language modeling, and on reducing the hardware barrier between SSMs and attention. First, we use synthetic language modeling tasks to understand the gap between SSMs and attention. We find that existing SSMs struggle with two capabilities: recalling earlier tokens in the sequence and comparing tokens across the sequence. To understand the impact on language modeling, we propose a new SSM layer, H3, that is explicitly designed for these abilities. H3 matches attention on the synthetic languages and comes within 0.4 PPL of Transformers on OpenWebText. Furthermore, a hybrid 125M-parameter H3-attention model that retains two attention layers surprisingly outperforms Transformers on OpenWebText by 1.0 PPL. Next, to improve the efficiency of training SSMs on modern hardware, we propose FlashConv. FlashConv uses a fused block FFT algorithm to improve efficiency on sequences up to 8K, and introduces a novel state passing algorithm that exploits the recurrent properties of SSMs to scale to longer sequences. FlashConv yields 2$\times$ speedup on the long-range arena benchmark and allows hybrid language models to generate text 1.6$\times$ faster than Transformers. Using FlashConv, we scale hybrid H3-attention language models up to 1.3B parameters on the Pile and find promising initial results, achieving lower perplexity than Transformers and outperforming Transformers in zero- and few-shot learning on a majority of tasks in the SuperGLUE benchmark.

Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expanding their application. We believe the field has now matured and is in need of a unified perspective. In this review, we attempt to provide such a perspective by describing flows through the lens of probabilistic modeling and inference. We place special emphasis on the fundamental principles of flow design, and discuss foundational topics such as expressive power and computational trade-offs. We also broaden the conceptual framing of flows by relating them to more general probability transformations. Lastly, we summarize the use of flows for tasks such as generative modeling, approximate inference, and supervised learning.

基于签名的技术使数学洞察力洞悉不断发展的数据的复杂流之间的相互作用。这些见解可以自然地转化为理解流数据的数值方法，也许是由于它们的数学精度，已被证明在数据不规则而不是固定的情况下分析流的数据以及数据和数据的尺寸很有用样本量均为中等。了解流的多模式数据是指数的：$ d $ d $的字母中的$ n $字母中的一个单词可以是$ d^n $消息之一。签名消除了通过采样不规则性引起的指数级噪声，但仍然存在指数量的信息。这项调查旨在留在可以直接管理指数缩放的域中。在许多问题中，可伸缩性问题是一个重要的挑战，但需要另一篇调查文章和进一步的想法。这项调查描述了一系列环境集足够小以消除大规模机器学习的可能性，并且可以有效地使用一小部分免费上下文和原则性功能。工具的数学性质可以使他们对非数学家的使用恐吓。本文中介绍的示例旨在弥合此通信差距，并提供从机器学习环境中绘制的可进行的工作示例。笔记本可以在线提供这些示例中的一些。这项调查是基于伊利亚·雪佛兰（Ilya Chevryev）和安德烈·科米利津（Andrey Kormilitzin）的早期论文，它们在这种机械开发的较早时刻大致相似。本文说明了签名提供的理论见解是如何在对应用程序数据的分析中简单地实现的，这种方式在很大程度上对数据类型不可知。

We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network's Fisher information matrix which is neither diagonal nor low-rank, and in some cases is completely non-sparse. It is derived by approximating various large blocks of the Fisher (corresponding to entire layers) as being the Kronecker product of two much smaller matrices. While only several times more expensive to compute than the plain stochastic gradient, the updates produced by K-FAC make much more progress optimizing the objective, which results in an algorithm that can be much faster than stochastic gradient descent with momentum in practice. And unlike some previously proposed approximate natural-gradient/Newton methods which use high-quality non-diagonal curvature matrices (such as Hessian-free optimization), K-FAC works very well in highly stochastic optimization regimes. This is because the cost of storing and inverting K-FAC's approximation to the curvature matrix does not depend on the amount of data used to estimate it, which is a feature typically associated only with diagonal or low-rank approximations to the curvature matrix.

Accurate uncertainty quantification is a major challenge in deep learning, as neural networks can make overconfident errors and assign high confidence predictions to out-of-distribution (OOD) inputs. The most popular approaches to estimate predictive uncertainty in deep learning are methods that combine predictions from multiple neural networks, such as Bayesian neural networks (BNNs) and deep ensembles. However their practicality in real-time, industrial-scale applications are limited due to the high memory and computational cost. Furthermore, ensembles and BNNs do not necessarily fix all the issues with the underlying member networks. In this work, we study principled approaches to improve uncertainty property of a single network, based on a single, deterministic representation. By formalizing the uncertainty quantification as a minimax learning problem, we first identify distance awareness, i.e., the model's ability to quantify the distance of a testing example from the training data, as a necessary condition for a DNN to achieve high-quality (i.e., minimax optimal) uncertainty estimation. We then propose Spectral-normalized Neural Gaussian Process (SNGP), a simple method that improves the distance-awareness ability of modern DNNs with two simple changes: (1) applying spectral normalization to hidden weights to enforce bi-Lipschitz smoothness in representations and (2) replacing the last output layer with a Gaussian process layer. On a suite of vision and language understanding benchmarks, SNGP outperforms other single-model approaches in prediction, calibration and out-of-domain detection. Furthermore, SNGP provides complementary benefits to popular techniques such as deep ensembles and data augmentation, making it a simple and scalable building block for probabilistic deep learning. Code is open-sourced at https://github.com/google/uncertainty-baselines

重型模型引起了神经网络现代发展的关注。深度平衡模型（DEQ）代表具有重量趋势的无限深度神经网络，最近的研究表明了这种方法的潜力。需要迭代解决训练中的根发现问题，并建立在模型确定的基础动力学基础上，需要DEQ。在本文中，我们介绍了稳定的不变模型（SIM），这是一种新的深层模型，原理在稳定性下近似DEQ，并将动力学扩展到更一般的动力学，从而收敛到不变的集合（不受固定点的限制）。得出SIMS的关键要素是用Koopman和Perron--Frobenius操作员的光谱表示动力学的代表。该视角大致揭示了用DEQS揭示稳定的动力学，然后衍生了两个SIMS的变体。我们还提出了可以以与前馈模型相同的方式学习的SIMS的实现。我们通过实验说明了SIMS的经验表现，并证明SIMS在几个学习任务中对DEQ实现了比较或出色的表现。

Pre-publication draft of a book to be published byMorgan & Claypool publishers. Unedited version released with permission. All relevant copyrights held by the author and publisher extend to this pre-publication draft.

结构分布，即组合空间的分布，通常用于学习观察到数据的潜在概率表示。然而，缩放这些模型是由高计算和内存复杂度相对于潜在表示的大小的瓶颈。诸如隐藏的马尔可夫模型（HMMS）和概率的无内容语法（PCFG）的常见模型在隐藏状态的数量中需要时间和空间二次和立方。这项工作展示了一种简单的方法来降低大类结构化模型的计算和内存复杂性。我们展示通过将中央推理步骤视为矩阵 - 矢量产品，并使用低秩约束，我们可以通过等级进行模型表达性和速度。用神经参数化结构化模型进行语言建模，复音音乐建模，无监督语法诱导和视频建模的实验表明，我们的方法在提供实用加速度的同时匹配大状态空间的标准模型的准确性。