在本文中，据我们所知，我们提供了将各种掩盖机制纳入变形金刚以可扩展方式融入变形金刚结构的第一种综合方法。我们表明，有关线性因果关注的最新结果（Choromanski等，2021）和对数线性RPE注意力（Luo等，2021）是这种一般机制的特殊情况。但是，通过将问题作为对未掩盖注意力的拓扑调制（基于图的）调制，我们以前获得了几个未知结果，包括有效的D维RPE掩盖和图形内掩蔽。我们利用许多数学技术，从光谱分析到动态编程和随机步行到新算法，以求解图形上的马尔可夫过程。我们提供相应的经验评估。

We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness. To approximate softmax attentionkernels, Performers use a novel Fast Attention Via positive Orthogonal Random features approach (FAVOR+), which may be of independent interest for scalable kernel methods. FAVOR+ can also be used to efficiently model kernelizable attention mechanisms beyond softmax. This representational power is crucial to accurately compare softmax with other kernels for the first time on large-scale tasks, beyond the reach of regular Transformers, and investigate optimal attention-kernels. Performers are linear architectures fully compatible with regular Transformers and with strong theoretical guarantees: unbiased or nearly-unbiased estimation of the attention matrix, uniform convergence and low estimation variance. We tested Performers on a rich set of tasks stretching from pixel-prediction through text models to protein sequence modeling. We demonstrate competitive results with other examined efficient sparse and dense attention methods, showcasing effectiveness of the novel attention-learning paradigm leveraged by Performers.

Graph classification is an important area in both modern research and industry. Multiple applications, especially in chemistry and novel drug discovery, encourage rapid development of machine learning models in this area. To keep up with the pace of new research, proper experimental design, fair evaluation, and independent benchmarks are essential. Design of strong baselines is an indispensable element of such works. In this thesis, we explore multiple approaches to graph classification. We focus on Graph Neural Networks (GNNs), which emerged as a de facto standard deep learning technique for graph representation learning. Classical approaches, such as graph descriptors and molecular fingerprints, are also addressed. We design fair evaluation experimental protocol and choose proper datasets collection. This allows us to perform numerous experiments and rigorously analyze modern approaches. We arrive to many conclusions, which shed new light on performance and quality of novel algorithms. We investigate application of Jumping Knowledge GNN architecture to graph classification, which proves to be an efficient tool for improving base graph neural network architectures. Multiple improvements to baseline models are also proposed and experimentally verified, which constitutes an important contribution to the field of fair model comparison.

Kernel matrices, as well as weighted graphs represented by them, are ubiquitous objects in machine learning, statistics and other related fields. The main drawback of using kernel methods (learning and inference using kernel matrices) is efficiency -- given $n$ input points, most kernel-based algorithms need to materialize the full $n \times n$ kernel matrix before performing any subsequent computation, thus incurring $\Omega(n^2)$ runtime. Breaking this quadratic barrier for various problems has therefore, been a subject of extensive research efforts. We break the quadratic barrier and obtain $\textit{subquadratic}$ time algorithms for several fundamental linear-algebraic and graph processing primitives, including approximating the top eigenvalue and eigenvector, spectral sparsification, solving linear systems, local clustering, low-rank approximation, arboricity estimation and counting weighted triangles. We build on the recent Kernel Density Estimation framework, which (after preprocessing in time subquadratic in $n$) can return estimates of row/column sums of the kernel matrix. In particular, we develop efficient reductions from $\textit{weighted vertex}$ and $\textit{weighted edge sampling}$ on kernel graphs, $\textit{simulating random walks}$ on kernel graphs, and $\textit{importance sampling}$ on matrices to Kernel Density Estimation and show that we can generate samples from these distributions in $\textit{sublinear}$ (in the support of the distribution) time. Our reductions are the central ingredient in each of our applications and we believe they may be of independent interest. We empirically demonstrate the efficacy of our algorithms on low-rank approximation (LRA) and spectral sparsification, where we observe a $\textbf{9x}$ decrease in the number of kernel evaluations over baselines for LRA and a $\textbf{41x}$ reduction in the graph size for spectral sparsification.

在过去十年中，图形内核引起了很多关注，并在结构化数据上发展成为一种快速发展的学习分支。在过去的20年中，该领域发生的相当大的研究活动导致开发数十个图形内核，每个图形内核都对焦于图形的特定结构性质。图形内核已成功地成功地在广泛的域中，从社交网络到生物信息学。本调查的目标是提供图形内核的文献的统一视图。特别是，我们概述了各种图形内核。此外，我们对公共数据集的几个内核进行了实验评估，并提供了比较研究。最后，我们讨论图形内核的关键应用，并概述了一些仍有待解决的挑战。

图形内核是历史上最广泛使用的图形分类任务的技术。然而，由于图的手工制作的组合特征，这些方法具有有限的性能。近年来，由于其性能卓越，图形神经网络（GNNS）已成为与下游图形相关任务的最先进的方法。大多数GNN基于消息传递神经网络（MPNN）框架。然而，最近的研究表明，MPNN不能超过Weisfeiler-Lehman（WL）算法在图形同构术中的力量。为了解决现有图形内核和GNN方法的限制，在本文中，我们提出了一种新的GNN框架，称为\ Texit {内核图形神经网络}（Kernnns），该框架将图形内核集成到GNN的消息传递过程中。通过卷积神经网络（CNNS）中的卷积滤波器的启发，KERGNNS采用可训练的隐藏图作为绘图过滤器，该绘图过滤器与子图组合以使用图形内核更新节点嵌入式。此外，我们表明MPNN可以被视为Kergnns的特殊情况。我们将Kergnns应用于多个与图形相关的任务，并使用交叉验证来与基准进行公平比较。我们表明，与现有的现有方法相比，我们的方法达到了竞争性能，证明了增加GNN的表现能力的可能性。我们还表明，KERGNNS中的训练有素的图形过滤器可以揭示数据集的本地图形结构，与传统GNN模型相比，显着提高了模型解释性。

Pre-publication draft of a book to be published byMorgan & Claypool publishers. Unedited version released with permission. All relevant copyrights held by the author and publisher extend to this pre-publication draft.

Deep learning has revolutionized many machine learning tasks in recent years, ranging from image classification and video processing to speech recognition and natural language understanding. The data in these tasks are typically represented in the Euclidean space. However, there is an increasing number of applications where data are generated from non-Euclidean domains and are represented as graphs with complex relationships and interdependency between objects. The complexity of graph data has imposed significant challenges on existing machine learning algorithms. Recently, many studies on extending deep learning approaches for graph data have emerged. In this survey, we provide a comprehensive overview of graph neural networks (GNNs) in data mining and machine learning fields. We propose a new taxonomy to divide the state-of-the-art graph neural networks into four categories, namely recurrent graph neural networks, convolutional graph neural networks, graph autoencoders, and spatial-temporal graph neural networks. We further discuss the applications of graph neural networks across various domains and summarize the open source codes, benchmark data sets, and model evaluation of graph neural networks. Finally, we propose potential research directions in this rapidly growing field.

最新提出的基于变压器的图形模型的作品证明了香草变压器用于图形表示学习的不足。要了解这种不足，需要研究变压器的光谱分析是否会揭示其对其表现力的见解。类似的研究已经确定，图神经网络（GNN）的光谱分析为其表现力提供了额外的观点。在这项工作中，我们系统地研究并建立了变压器领域中的空间和光谱域之间的联系。我们进一步提供了理论分析，并证明了变压器中的空间注意机制无法有效捕获所需的频率响应，因此，固有地限制了其在光谱空间中的表现力。因此，我们提出了feta，该框架旨在在整个图形频谱（即图形的实际频率成分）上进行注意力类似于空间空间中的注意力。经验结果表明，FETA在标准基准的所有任务中为香草变压器提供均匀的性能增益，并且可以轻松地扩展到具有低通特性的基于GNN的模型（例如GAT）。

In the last few years, graph neural networks (GNNs) have become the standard toolkit for analyzing and learning from data on graphs. This emerging field has witnessed an extensive growth of promising techniques that have been applied with success to computer science, mathematics, biology, physics and chemistry. But for any successful field to become mainstream and reliable, benchmarks must be developed to quantify progress. This led us in March 2020 to release a benchmark framework that i) comprises of a diverse collection of mathematical and real-world graphs, ii) enables fair model comparison with the same parameter budget to identify key architectures, iii) has an open-source, easy-to-use and reproducible code infrastructure, and iv) is flexible for researchers to experiment with new theoretical ideas. As of December 2022, the GitHub repository has reached 2,000 stars and 380 forks, which demonstrates the utility of the proposed open-source framework through the wide usage by the GNN community. In this paper, we present an updated version of our benchmark with a concise presentation of the aforementioned framework characteristics, an additional medium-sized molecular dataset AQSOL, similar to the popular ZINC, but with a real-world measured chemical target, and discuss how this framework can be leveraged to explore new GNN designs and insights. As a proof of value of our benchmark, we study the case of graph positional encoding (PE) in GNNs, which was introduced with this benchmark and has since spurred interest of exploring more powerful PE for Transformers and GNNs in a robust experimental setting.

近年来，基于Weisfeiler-Leman算法的算法和神经架构，是一个众所周知的Graph同构问题的启发式问题，它成为具有图形和关系数据的机器学习的强大工具。在这里，我们全面概述了机器学习设置中的算法的使用，专注于监督的制度。我们讨论了理论背景，展示了如何将其用于监督的图形和节点表示学习，讨论最近的扩展，并概述算法的连接（置换 - ）方面的神经结构。此外，我们概述了当前的应用和未来方向，以刺激进一步的研究。

几何深度学习取得了长足的进步，旨在概括从传统领域到非欧几里得群岛的结构感知神经网络的设计，从而引起图形神经网络（GNN），这些神经网络（GNN）可以应用于形成的图形结构数据，例如社会，例如，网络，生物化学和材料科学。尤其是受欧几里得对应物的启发，尤其是图形卷积网络（GCN）通过提取结构感知功能来成功处理图形数据。但是，当前的GNN模型通常受到各种现象的限制，这些现象限制了其表达能力和推广到更复杂的图形数据集的能力。大多数模型基本上依赖于通过本地平均操作对图形信号的低通滤波，从而导致过度平滑。此外，为了避免严重的过度厚度，大多数流行的GCN式网络往往是较浅的，并且具有狭窄的接收场，导致侵犯。在这里，我们提出了一个混合GNN框架，该框架将传统的GCN过滤器与通过几何散射定义的带通滤波器相结合。我们进一步介绍了一个注意框架，该框架允许该模型在节点级别上从不同过滤器的组合信息进行本地参与。我们的理论结果确定了散射过滤器的互补益处，以利用图表中的结构信息，而我们的实验显示了我们方法对各种学习任务的好处。

图表神经网络（GNNS）在各种机器学习任务中获得了表示学习的提高。然而，应用邻域聚合的大多数现有GNN通常在图中的图表上执行不良，其中相邻的节点属于不同的类。在本文中，我们示出了在典型的异界图中，边缘可以被引导，以及是否像是处理边缘，也可以使它们过度地影响到GNN模型的性能。此外，由于异常的限制，节点对来自本地邻域之外的类似节点的消息非常有益。这些激励我们开发一个自适应地学习图表的方向性的模型，并利用潜在的长距离相关性节点之间。我们首先将图拉普拉斯概括为基于所提出的特征感知PageRank算法向数字化，该算法同时考虑节点之间的图形方向性和长距离特征相似性。然后，Digraph Laplacian定义了一个图形传播矩阵，导致一个名为{\ em diglaciangcn}的模型。基于此，我们进一步利用节点之间的通勤时间测量的节点接近度，以便在拓扑级别上保留节点的远距离相关性。具有不同级别的10个数据集的广泛实验，同意级别展示了我们在节点分类任务任务中对现有解决方案的有效性。

Deep learning has been shown to be successful in a number of domains, ranging from acoustics, images, to natural language processing. However, applying deep learning to the ubiquitous graph data is non-trivial because of the unique characteristics of graphs. Recently, substantial research efforts have been devoted to applying deep learning methods to graphs, resulting in beneficial advances in graph analysis techniques. In this survey, we comprehensively review the different types of deep learning methods on graphs. We divide the existing methods into five categories based on their model architectures and training strategies: graph recurrent neural networks, graph convolutional networks, graph autoencoders, graph reinforcement learning, and graph adversarial methods. We then provide a comprehensive overview of these methods in a systematic manner mainly by following their development history. We also analyze the differences and compositions of different methods. Finally, we briefly outline the applications in which they have been used and discuss potential future research directions.

作为建模复杂关系的强大工具，HyperGraphs从图表学习社区中获得了流行。但是，深度刻画学习中的常用框架专注于具有边缘独立的顶点权重（EIVW）的超图，而无需考虑具有具有更多建模功率的边缘依赖性顶点权重（EDVWS）的超图。为了弥补这一点，我们提出了一般的超图光谱卷积（GHSC），这是一个通用学习框架，不仅可以处理EDVW和EIVW HyperGraphs，而且更重要的是，理论上可以明确地利用现有强大的图形卷积神经网络（GCNN）明确说明，从而很大程度上可以释放。超图神经网络的设计。在此框架中，给定的无向GCNN的图形拉普拉斯被统一的HyperGraph Laplacian替换，该统一的HyperGraph Laplacian通过将我们所定义的广义超透明牌与简单的无向图等同起来，从随机的步行角度将顶点权重信息替换。来自各个领域的广泛实验，包括社交网络分析，视觉目标分类和蛋白质学习，证明了拟议框架的最新性能。

消息传递神经网络（MPNNS）是由于其简单性和可扩展性而大部分地进行图形结构数据的深度学习的领先架构。不幸的是，有人认为这些架构的表现力有限。本文提出了一种名为Comifariant Subgraph聚合网络（ESAN）的新颖框架来解决这个问题。我们的主要观察是，虽然两个图可能无法通过MPNN可区分，但它们通常包含可区分的子图。因此，我们建议将每个图形作为由某些预定义策略导出的一组子图，并使用合适的等分性架构来处理它。我们为图同构同构同构造的1立维Weisfeiler-Leman（1-WL）测试的新型变体，并在这些新的WL变体方面证明了ESAN的表达性下限。我们进一步证明，我们的方法增加了MPNNS和更具表现力的架构的表现力。此外，我们提供了理论结果，描述了设计选择诸如子图选择政策和等效性神经结构的设计方式如何影响我们的架构的表现力。要处理增加的计算成本，我们提出了一种子图采样方案，可以将其视为我们框架的随机版本。关于真实和合成数据集的一套全面的实验表明，我们的框架提高了流行的GNN架构的表现力和整体性能。

Graph neural networks (GNNs) are widely used for modeling complex interactions between entities represented as vertices of a graph. Despite recent efforts to theoretically analyze the expressive power of GNNs, a formal characterization of their ability to model interactions is lacking. The current paper aims to address this gap. Formalizing strength of interactions through an established measure known as separation rank, we quantify the ability of certain GNNs to model interaction between a given subset of vertices and its complement, i.e. between sides of a given partition of input vertices. Our results reveal that the ability to model interaction is primarily determined by the partition's walk index -- a graph-theoretical characteristic that we define by the number of walks originating from the boundary of the partition. Experiments with common GNN architectures corroborate this finding. As a practical application of our theory, we design an edge sparsification algorithm named Walk Index Sparsification (WIS), which preserves the ability of a GNN to model interactions when input edges are removed. WIS is simple, computationally efficient, and markedly outperforms alternative methods in terms of induced prediction accuracy. More broadly, it showcases the potential of improving GNNs by theoretically analyzing the interactions they can model.

A current goal in the graph neural network literature is to enable transformers to operate on graph-structured data, given their success on language and vision tasks. Since the transformer's original sinusoidal positional encodings (PEs) are not applicable to graphs, recent work has focused on developing graph PEs, rooted in spectral graph theory or various spatial features of a graph. In this work, we introduce a new graph PE, Graph Automaton PE (GAPE), based on weighted graph-walking automata (a novel extension of graph-walking automata). We compare the performance of GAPE with other PE schemes on both machine translation and graph-structured tasks, and we show that it generalizes several other PEs. An additional contribution of this study is a theoretical and controlled experimental comparison of many recent PEs in graph transformers, independent of the use of edge features.

这篇综述的目的是将读者介绍到图表内，以将其应用于化学信息学中的分类问题。图内核是使我们能够推断分子的化学特性的功能，可以帮助您完成诸如寻找适合药物设计的化合物等任务。内核方法的使用只是一种特殊的两种方式量化了图之间的相似性。我们将讨论限制在这种方法上，尽管近年来已经出现了流行的替代方法，但最著名的是图形神经网络。

在处理大规模网络和关系数据时，降低图是基本的。它们可以通过在粗糙的结构中求解它们来缩小高度计算影响的尺寸。同时，图减少起着在图神经网络中合并层的作用，从结构中提取多分辨率表示。在这些情况下，还原机制保留距离关系和拓扑特性的能力似乎是基本的，以及可扩展性，使其能够应用于实际大小的问题。在本文中，我们基于最大重量$ k $独立的集合的图理论概念引入了图形粗化机制，从而提供了一种贪婪的算法，该算法允许在GPU上有效地并行实现。我们的方法是常规数据（图像，序列）中的第一个图形结构化对应物。我们证明了在路径长度上的失真界限的理论保证，以及在污垢图中保留关键拓扑特性的能力。我们利用这些概念来定义我们在图形分类任务中经验评估的图表合并机制，表明它与文献中的合并方法进行了比较。