过度平滑是一个具有挑战性的问题,这会降低深图卷积网络(GCNS)的性能。然而,用于缓解过度平滑问题的现有研究缺乏一般性或有效性。在本文中,我们分析了过度平滑问题背后的潜在问题,即特征 - 多样性退化,梯度消失和模型重量衰减。灵感来自于此,我们提出了一个简单而有效的即插即用模块,速度,缓解过度平滑。具体地,对于GCN模型的每个中间层,随机地(或基于节点度)选择节点以通过直接向非线性函数馈送它们的输入特征来跳过卷积操作。分析,1)跳过卷积操作可以防止特征失去多样性; 2)“跳过”节点使能梯度直接传递回来,从而减轻梯度消失和模型权重过腐蚀问题。为了展示Skipnode的优越性,我们对九个流行的数据集进行了广泛的实验,包括同性恋和异化图,在两个典型的任务上具有不同的图表大小:节点分类和链路预测。具体而言,1)SkipNode具有适应不同数据集和任务的各种基于GCN的模型的普遍性。 2)Skipnode优于最近最先进的反平滑插头 - 播放模块,即DropEdge和Dropnode,在不同的设置中。代码将在GitHub上公开提供。
translated by 谷歌翻译
提高GCN的深度(预计将允许更多表达性)显示出损害性能,尤其是在节点分类上。原因的主要原因在于过度平滑。过度平滑的问题将GCN的输出驱动到一个在节点之间包含有限的区别信息的空间,从而导致表现不佳。已经提出了一些有关完善GCN架构的作品,但理论上仍然未知这些改进是否能够缓解过度平衡。在本文中,我们首先从理论上分析了通用GCN如何与深度增加的作用,包括通用GCN,GCN,具有偏见,RESGCN和APPNP。我们发现所有这些模型都以通用过程为特征:所有节点融合到Cuboid。在该定理下,我们建议通过在每个训练时期随机去除一定数量的边缘来减轻过度光滑的状态。从理论上讲,Dropedge可以降低过度平滑的收敛速度,或者可以减轻尺寸崩溃引起的信息损失。对模拟数据集的实验评估已可视化不同GCN之间过度平滑的差异。此外,对几个真正的基准支持的广泛实验,这些实验始终如一地改善各种浅GCN和深度GCN的性能。
translated by 谷歌翻译
Over-fitting and over-smoothing are two main obstacles of developing deep Graph Convolutional Networks (GCNs) for node classification. In particular, over-fitting weakens the generalization ability on small dataset, while over-smoothing impedes model training by isolating output representations from the input features with the increase in network depth. This paper proposes DropEdge, a novel and flexible technique to alleviate both issues. At its core, DropEdge randomly removes a certain number of edges from the input graph at each training epoch, acting like a data augmenter and also a message passing reducer. Furthermore, we theoretically demonstrate that DropEdge either reduces the convergence speed of over-smoothing or relieves the information loss caused by it. More importantly, our DropEdge is a general skill that can be equipped with many other backbone models (e.g. GCN, ResGCN, GraphSAGE, and JKNet) for enhanced performance. Extensive experiments on several benchmarks verify that DropEdge consistently improves the performance on a variety of both shallow and deep GCNs. The effect of DropEdge on preventing over-smoothing is empirically visualized and validated as well. Codes are released on https://github.com/DropEdge/DropEdge.
translated by 谷歌翻译
Graph convolutional networks (GCNs) are a powerful deep learning approach for graph-structured data. Recently, GCNs and subsequent variants have shown superior performance in various application areas on real-world datasets. Despite their success, most of the current GCN models are shallow, due to the over-smoothing problem.In this paper, we study the problem of designing and analyzing deep graph convolutional networks. We propose the GCNII, an extension of the vanilla GCN model with two simple yet effective techniques: Initial residual and Identity mapping. We provide theoretical and empirical evidence that the two techniques effectively relieves the problem of over-smoothing. Our experiments show that the deep GCNII model outperforms the state-of-the-art methods on various semi-and fullsupervised tasks. Code is available at https: //github.com/chennnM/GCNII.
translated by 谷歌翻译
图形神经网络已成为从图形结构数据学习的不可缺少的工具之一,并且它们的实用性已在各种各样的任务中显示。近年来,建筑设计的巨大改进,导致各种预测任务的性能更好。通常,这些神经架构在同一层中使用可知的权重矩阵组合节点特征聚合和特征转换。这使得分析从各种跳过的节点特征和神经网络层的富有效力来挑战。由于不同的图形数据集显示在特征和类标签分布中的不同级别和异常级别,因此必须了解哪些特征对于没有任何先前信息的预测任务是重要的。在这项工作中,我们将节点特征聚合步骤和深度与图形神经网络分离,并经验分析了不同的聚合特征在预测性能中发挥作用。我们表明,并非通过聚合步骤生成的所有功能都很有用,并且通常使用这些较少的信息特征可能对GNN模型的性能有害。通过我们的实验,我们表明学习这些功能的某些子集可能会导致各种数据集的性能更好。我们建议使用Softmax作为常规器,并从不同跳距的邻居聚合的功能的“软选择器”;和L2 - GNN层的标准化。结合这些技术,我们呈现了一个简单浅的模型,特征选择图神经网络(FSGNN),并经验展示所提出的模型比九个基准数据集中的最先进的GNN模型实现了可比或甚至更高的准确性节点分类任务,具有显着的改进,可达51.1%。
translated by 谷歌翻译
Graph neural networks have shown significant success in the field of graph representation learning. Graph convolutions perform neighborhood aggregation and represent one of the most important graph operations. Nevertheless, one layer of these neighborhood aggregation methods only consider immediate neighbors, and the performance decreases when going deeper to enable larger receptive fields. Several recent studies attribute this performance deterioration to the over-smoothing issue, which states that repeated propagation makes node representations of different classes indistinguishable. In this work, we study this observation systematically and develop new insights towards deeper graph neural networks. First, we provide a systematical analysis on this issue and argue that the key factor compromising the performance significantly is the entanglement of representation transformation and propagation in current graph convolution operations. After decoupling these two operations, deeper graph neural networks can be used to learn graph node representations from larger receptive fields. We further provide a theoretical analysis of the above observation when building very deep models, which can serve as a rigorous and gentle description of the over-smoothing issue. Based on our theoretical and empirical analysis, we propose Deep Adaptive Graph Neural Network (DAGNN) to adaptively incorporate information from large receptive fields. A set of experiments on citation, coauthorship, and co-purchase datasets have confirmed our analysis and insights and demonstrated the superiority of our proposed methods. CCS CONCEPTS• Mathematics of computing → Graph algorithms; • Computing methodologies → Artificial intelligence; Neural networks.
translated by 谷歌翻译
Graph Neural Networks (GNNs) are powerful tools for graph representation learning. Despite their rapid development, GNNs also face some challenges, such as over-fitting, over-smoothing, and non-robustness. Previous works indicate that these problems can be alleviated by random dropping methods, which integrate augmented data into models by randomly masking parts of the input. However, some open problems of random dropping on GNNs remain to be solved. First, it is challenging to find a universal method that are suitable for all cases considering the divergence of different datasets and models. Second, augmented data introduced to GNNs causes the incomplete coverage of parameters and unstable training process. Third, there is no theoretical analysis on the effectiveness of random dropping methods on GNNs. In this paper, we propose a novel random dropping method called DropMessage, which performs dropping operations directly on the propagated messages during the message-passing process. More importantly, we find that DropMessage provides a unified framework for most existing random dropping methods, based on which we give theoretical analysis of their effectiveness. Furthermore, we elaborate the superiority of DropMessage: it stabilizes the training process by reducing sample variance; it keeps information diversity from the perspective of information theory, enabling it become a theoretical upper bound of other methods. To evaluate our proposed method, we conduct experiments that aims for multiple tasks on five public datasets and two industrial datasets with various backbone models. The experimental results show that DropMessage has the advantages of both effectiveness and generalization, and can significantly alleviate the problems mentioned above.
translated by 谷歌翻译
图形神经网络(GNN)在各种图挖掘任务中取得了巨大的成功。但是,当GNN堆叠着许多层时,总是观察到急剧性能降解。结果,大多数GNN仅具有浅层建筑,这限制了它们的表现力和对深社区的开发。最近的研究将深度GNN的性能降低归因于\ textit {过度平滑}的问题。在本文中,我们将传统的图形卷积操作分为两个独立操作:\ textit {passagation}(\ textbf {p})和\ textit {transformation}(\ textbf {t})。可以分为传播深度($ d_p $)和转换深度($ d_t $)。通过广泛的实验,我们发现深度GNNS性能下降的主要原因是\ textit {model dygradation}问题是由大$ d_t $而不是\ textit {过度平滑}问题引起的,主要是由大$ d_p $引起的。 。此外,我们提出\ textIt {自适应初始残留}(air),一个与各种GNN架构兼容的插件模块,以减轻\ textit {model {model dradation degradation}问题和\ textit {textit {过度敏感}问题同时。六个现实世界数据集的实验结果表明,配备空气的GNN胜过大多数具有浅层建筑的GNN,这是由于大型$ d_p $和$ d_t $的好处,而与空气相关的时间成本则可以忽略。
translated by 谷歌翻译
图形神经网络(GNN)由于从图形结构数据中学习表示能力而引起了很多关注。尽管GNN在许多域中成功地应用了,但GNN的优化程度较低,并且在节点分类的性能很大程度上受到了长尾节点学位分布的影响。本文着重于通过归一化提高GNN的性能。详细说明,通过研究图中的节点度的长尾巴分布,我们提出了一种新颖的GNN归一化方法,该方法称为RESNORM(\ textbf {res}将长尾巴分布纳入正常分布,通过\ textbf {norm} alization)。 RESNOR的$比例$操作重塑节点标准偏差(NSTD)分布,以提高尾部节点的准确性(\ textit {i}。\ textit {e}。,低度节点)。我们提供了理论解释和经验证据,以理解上述$ scale $的机制。除了长期的分销问题外,过度光滑也是困扰社区的基本问题。为此,我们分析了标准偏移的行为,并证明了标准移位是重量矩阵上的预处理,从而增加了过度平滑的风险。考虑到过度光滑的问题,我们为Resnorm设计了一个$ Shift $操作,以低成本的方式模拟了特定于学位的参数策略。广泛的实验验证了重新分类对几个节点分类基准数据集的有效性。
translated by 谷歌翻译
由于学习节点表示的优越性,图形神经网络(GNNS)受到了巨大的关注。这些模型依赖于消息传递和特征转换功能来从邻居编码结构和功能信息。然而,堆叠更多的卷积层显着降低了GNN的性能。大多数最近的研究将此限制属于过平滑问题,其中节点嵌入式会聚到无法区分的向量。通过许多实验观察,我们认为,主要因素降低性能是不稳定的正向标准化和后向梯度因特征变换的不当设计而导致的,尤其是对于未发生过平滑的浅GNN。因此,我们提出了一个名为Ortho-GConv的新型正交特征转换,这通常可以增加现有的GNN骨干,以稳定模型训练并改善模型的泛化性能。具体地,我们从三个视角综合地维持特征变换的正交性,即混合权重初始化,正交变换和正交正规。通过用ortho-gconv配备现有的GNN(例如GCN,JKNET,GCNII),我们展示了正交特征变换的一般性以实现稳定训练,并显示其对节点和图形分类任务的有效性。
translated by 谷歌翻译
最近,图形神经网络(GNN)通过利用图形结构和节点特征的知识来表现出图表表示的显着性能。但是,他们中的大多数都有两个主要限制。首先,GNN可以通过堆叠更多的层来学习高阶结构信息,但由于过度光滑的问题,无法处理较大的深度。其次,由于昂贵的计算成本和高内存使用情况,在大图上应用这些方法并不容易。在本文中,我们提出了节点自适应特征平滑(NAFS),这是一种简单的非参数方法,该方法构建了没有参数学习的节点表示。 NAFS首先通过特征平滑提取每个节点及其不同啤酒花的邻居的特征,然后自适应地结合了平滑的特征。此外,通过不同的平滑策略提取的平滑特征的合奏可以进一步增强构建的节点表示形式。我们在两个不同的应用程序方案上对四个基准数据集进行实验:节点群集和链接预测。值得注意的是,具有功能合奏的NAFS优于这些任务上最先进的GNN,并减轻上述大多数基于学习的GNN对应物的两个限制。
translated by 谷歌翻译
Graph Neural Networks (graph NNs) are a promising deep learning approach for analyzing graph-structured data. However, it is known that they do not improve (or sometimes worsen) their predictive performance as we pile up many layers and add non-lineality. To tackle this problem, we investigate the expressive power of graph NNs via their asymptotic behaviors as the layer size tends to infinity. Our strategy is to generalize the forward propagation of a Graph Convolutional Network (GCN), which is a popular graph NN variant, as a specific dynamical system. In the case of a GCN, we show that when its weights satisfy the conditions determined by the spectra of the (augmented) normalized Laplacian, its output exponentially approaches the set of signals that carry information of the connected components and node degrees only for distinguishing nodes. Our theory enables us to relate the expressive power of GCNs with the topological information of the underlying graphs inherent in the graph spectra. To demonstrate this, we characterize the asymptotic behavior of GCNs on the Erdős -Rényi graph. We show that when the Erdős -Rényi graph is sufficiently dense and large, a broad range of GCNs on it suffers from the "information loss" in the limit of infinite layers with high probability. Based on the theory, we provide a principled guideline for weight normalization of graph NNs. We experimentally confirm that the proposed weight scaling enhances the predictive performance of GCNs in real data 1 .
translated by 谷歌翻译
图表神经网络(GNNS)在各种机器学习任务中获得了表示学习的提高。然而,应用邻域聚合的大多数现有GNN通常在图中的图表上执行不良,其中相邻的节点属于不同的类。在本文中,我们示出了在典型的异界图中,边缘可以被引导,以及是否像是处理边缘,也可以使它们过度地影响到GNN模型的性能。此外,由于异常的限制,节点对来自本地邻域之外的类似节点的消息非常有益。这些激励我们开发一个自适应地学习图表的方向性的模型,并利用潜在的长距离相关性节点之间。我们首先将图拉普拉斯概括为基于所提出的特征感知PageRank算法向数字化,该算法同时考虑节点之间的图形方向性和长距离特征相似性。然后,Digraph Laplacian定义了一个图形传播矩阵,导致一个名为{\ em diglaciangcn}的模型。基于此,我们进一步利用节点之间的通勤时间测量的节点接近度,以便在拓扑级别上保留节点的远距离相关性。具有不同级别的10个数据集的广泛实验,同意级别展示了我们在节点分类任务任务中对现有解决方案的有效性。
translated by 谷歌翻译
作为建模复杂关系的强大工具,HyperGraphs从图表学习社区中获得了流行。但是,深度刻画学习中的常用框架专注于具有边缘独立的顶点权重(EIVW)的超图,而无需考虑具有具有更多建模功率的边缘依赖性顶点权重(EDVWS)的超图。为了弥补这一点,我们提出了一般的超图光谱卷积(GHSC),这是一个通用学习框架,不仅可以处理EDVW和EIVW HyperGraphs,而且更重要的是,理论上可以明确地利用现有强大的图形卷积神经网络(GCNN)明确说明,从而很大程度上可以释放。超图神经网络的设计。在此框架中,给定的无向GCNN的图形拉普拉斯被统一的HyperGraph Laplacian替换,该统一的HyperGraph Laplacian通过将我们所定义的广义超透明牌与简单的无向图等同起来,从随机的步行角度将顶点权重信息替换。来自各个领域的广泛实验,包括社交网络分析,视觉目标分类和蛋白质学习,证明了拟议框架的最新性能。
translated by 谷歌翻译
图神经网络(GNN)已证明其在各种应用中的表现出色。然而,其背后的工作机制仍然神秘。 GNN模型旨在学习图形结构数据的有效表示,该数据本质上与图形信号denoising(GSD)的原理相吻合。算法展开是一种“学习优化”技术的算法,由于其在构建高效和可解释的神经网络体系结构方面的前景,人们引起了人们的关注。在本文中,我们引入了基于GSD问题的截断优化算法(例如梯度下降和近端梯度下降)构建的一类展开网络。它们被证明与许多流行的GNN模型紧密相连,因为这些GNN中的正向传播实际上是为特定GSD提供服务的展开网络。此外,可以将GNN模型的训练过程视为解决了较低级别的GSD问题的双重优化问题。这种连接带来了GNN的新景,因为我们可以尝试从GSD对应物中理解它们的实际功能,并且还可以激励设计新的GNN模型。基于算法展开的观点,一种名为UGDGNN的表达模型,即展开的梯度下降GNN,进一步提出了继承具有吸引力的理论属性的。七个基准数据集上的大量数值模拟表明,UGDGNN可以比最新模型实现卓越或竞争性的性能。
translated by 谷歌翻译
图形神经网络(GNNS)由于图形数据的规模和模型参数的数量呈指数增长,因此限制了它们在实际应用中的效用,因此往往会遭受高计算成本。为此,最近的一些作品着重于用彩票假设(LTH)稀疏GNN,以降低推理成本,同时保持绩效水平。但是,基于LTH的方法具有两个主要缺点:1)它们需要对密集模型进行详尽且迭代的训练,从而产生了极大的训练计算成本,2)它们仅修剪图形结构和模型参数,但忽略了节点功能维度,存在大量冗余。为了克服上述局限性,我们提出了一个综合的图形渐进修剪框架,称为CGP。这是通过在一个训练过程中设计在训练图周期修剪范式上进行动态修剪GNN来实现的。与基于LTH的方法不同,提出的CGP方法不需要重新训练,这大大降低了计算成本。此外,我们设计了一个共同策略,以全面地修剪GNN的所有三个核心元素:图形结构,节点特征和模型参数。同时,旨在完善修剪操作,我们将重生过程引入我们的CGP框架,以重新建立修剪但重要的连接。提出的CGP通过在6个GNN体系结构中使用节点分类任务进行评估,包括浅层模型(GCN和GAT),浅但深度散发模型(SGC和APPNP)以及Deep Models(GCNII和RESGCN),总共有14个真实图形数据集,包括来自挑战性开放图基准的大规模图数据集。实验表明,我们提出的策略在匹配时大大提高了训练和推理效率,甚至超过了现有方法的准确性。
translated by 谷歌翻译
近年来,图形神经网络(GNNS)在许多现实世界中的应用(例如建议和药物发现)中取得了巨大的成功。尽管取得了成功,但已将过度厚度确定为限制GNN绩效的关键问题之一。这表明由于堆叠聚合器,学到的节点表示是无法区分的。在本文中,我们提出了一种新的观点,以研究深度GNN的性能降低,即特征过度相关。通过有关此问题的经验和理论研究,我们证明了更深层次的GNN中的特征过度相关的存在,并揭示了导致该问题的潜在原因。为了减少功能相关性,我们提出了一个通用框架,可以鼓励GNN编码较少的冗余信息。广泛的实验表明,Decorr可以帮助实现更深入的GNN,并与现有的技术相辅相成。
translated by 谷歌翻译
Designing spectral convolutional networks is a challenging problem in graph learning. ChebNet, one of the early attempts, approximates the spectral graph convolutions using Chebyshev polynomials. GCN simplifies ChebNet by utilizing only the first two Chebyshev polynomials while still outperforming it on real-world datasets. GPR-GNN and BernNet demonstrate that the Monomial and Bernstein bases also outperform the Chebyshev basis in terms of learning the spectral graph convolutions. Such conclusions are counter-intuitive in the field of approximation theory, where it is established that the Chebyshev polynomial achieves the optimum convergent rate for approximating a function. In this paper, we revisit the problem of approximating the spectral graph convolutions with Chebyshev polynomials. We show that ChebNet's inferior performance is primarily due to illegal coefficients learnt by ChebNet approximating analytic filter functions, which leads to over-fitting. We then propose ChebNetII, a new GNN model based on Chebyshev interpolation, which enhances the original Chebyshev polynomial approximation while reducing the Runge phenomenon. We conducted an extensive experimental study to demonstrate that ChebNetII can learn arbitrary graph convolutions and achieve superior performance in both full- and semi-supervised node classification tasks. Most notably, we scale ChebNetII to a billion graph ogbn-papers100M, showing that spectral-based GNNs have superior performance. Our code is available at https://github.com/ivam-he/ChebNetII.
translated by 谷歌翻译
图表神经网络(GNNS)在图形结构数据的表现中表现出巨大的成功。在捕获图形拓扑中,GNN中的层展图表卷积显示为强大。在此过程中,GNN通常由预定义的内核引导,例如拉普拉斯矩阵,邻接矩阵或其变体。但是,预定义的内核的采用可能会限制不同图形的必要性:图形和内核之间的不匹配将导致次优性能。例如,当高频信息对于图表具有重要意义时,聚焦在低频信息上的GNN可能无法实现令人满意的性能,反之亦然。为了解决这个问题,在本文中,我们提出了一种新颖的框架 - 即,即Adaptive Kernel图神经网络(AKGNN) - 这将在第一次尝试时以统一的方式适应最佳图形内核。在所提出的AKGNN中,我们首先设计一种数据驱动的图形内核学习机制,它通过修改图拉普拉斯的最大特征值来自适应地调制全通过和低通滤波器之间的平衡。通过此过程,AKGNN了解高频信号之间的最佳阈值以减轻通用问题。稍后,我们通过参数化技巧进一步减少参数的数量,并通过全局读出功能增强富有表现力。在确认的基准数据集中进行了广泛的实验,并且有希望的结果通过与最先进的GNNS比较,展示了我们所提出的Akgnn的出色表现。源代码在公开上可用:https://github.com/jumxglhf/akgnn。
translated by 谷歌翻译
图形神经网络(GNN)在学习强大的节点表示中显示了令人信服的性能,这些表现在保留节点属性和图形结构信息的强大节点表示中。然而,许多GNNS在设计有更深的网络结构或手柄大小的图形时遇到有效性和效率的问题。已经提出了几种采样算法来改善和加速GNN的培训,但他们忽略了解GNN性能增益的来源。图表数据中的信息的测量可以帮助采样算法来保持高价值信息,同时消除冗余信息甚至噪声。在本文中,我们提出了一种用于GNN的公制引导(MEGUIDE)子图学习框架。 MEGUIDE采用两种新颖的度量:功能平滑和连接失效距离,以指导子图采样和迷你批次的培训。功能平滑度专为分析节点的特征而才能保留最有价值的信息,而连接失败距离可以测量结构信息以控制子图的大小。我们展示了MEGUIDE在多个数据集上培训各种GNN的有效性和效率。
translated by 谷歌翻译