近年来,自我监督学习(SSL)已广泛探索。特别是,生成的SSL在自然语言处理和其他AI领域(例如BERT和GPT的广泛采用)中获得了新的成功。尽管如此,对比度学习 - 严重依赖结构数据的增强和复杂的培训策略,这是图SSL的主要方法,而迄今为止,生成SSL在图形上的进度(尤其是GAES)尚未达到潜在的潜力。正如其他领域所承诺的。在本文中,我们确定并检查对GAE的发展产生负面影响的问题,包括其重建目标,训练鲁棒性和错误指标。我们提出了一个蒙版的图形自动编码器Graphmae,该图可以减轻这些问题,以预处理生成性自我监督图。我们建议没有重建图形结构,而是提议通过掩盖策略和缩放余弦误差将重点放在特征重建上,从而使GraphMae的强大训练受益。我们在21个公共数据集上进行了大量实验,以实现三个不同的图形学习任务。结果表明,Graphmae-A简单的图形自动编码器具有仔细的设计-CAN始终在对比度和生成性最新基准相比,始终产生优于性的表现。这项研究提供了对图自动编码器的理解,并证明了在图上的生成自我监督预训练的潜力。
translated by 谷歌翻译
关于图表的深度学习最近吸引了重要的兴趣。然而,大多数作品都侧重于(半)监督学习,导致缺点包括重标签依赖,普遍性差和弱势稳健性。为了解决这些问题,通过良好设计的借口任务在不依赖于手动标签的情况下提取信息知识的自我监督学习(SSL)已成为图形数据的有希望和趋势的学习范例。与计算机视觉和自然语言处理等其他域的SSL不同,图表上的SSL具有独家背景,设计理念和分类。在图表的伞下自我监督学习,我们对采用图表数据采用SSL技术的现有方法及时及全面的审查。我们构建一个统一的框架,数学上正式地规范图表SSL的范例。根据借口任务的目标,我们将这些方法分为四类:基于生成的,基于辅助性的,基于对比的和混合方法。我们进一步描述了曲线图SSL在各种研究领域的应用,并总结了绘图SSL的常用数据集,评估基准,性能比较和开源代码。最后,我们讨论了该研究领域的剩余挑战和潜在的未来方向。
translated by 谷歌翻译
我们介绍了一种新颖的屏蔽图AutoEncoder(MGAE)框架,以在图形结构数据上执行有效的学习。从自我监督学习中欣识见,我们随机掩盖了大部分边缘,并在训练期间尝试重建这些缺失的边缘。 Mgae有两个核心设计。首先,我们发现掩蔽了输入图结构的高比率,例如70 \%$,产生一个非凡和有意义的自我监督任务,使下游应用程序受益。其次,我们使用图形神经网络(GNN)作为编码器,以在部分掩蔽的图表上执行消息传播。为了重建大量掩模边缘,提出了一种定制的互相关解码器。它可以捕获多粒度的锚边的头部和尾部节点之间的互相关。耦合这两种设计使MGAE能够有效且有效地培训。在多个开放数据集(Planetoid和OGB基准测试)上进行了广泛的实验,证明MGAE通常比链接预测和节点分类更好地表现优于最先进的无监督竞争对手。
translated by 谷歌翻译
图神经网络的自我监督学习(SSL)正在成为利用未标记数据的有前途的方式。当前,大多数方法基于从图像域改编的对比度学习,该学习需要视图生成和足够数量的负样本。相比之下,现有的预测模型不需要负面抽样,但缺乏关于借口训练任务设计的理论指导。在这项工作中,我们提出了lagraph,这是基于潜在图预测的理论基础的预测SSL框架。 lagraph的学习目标被推导为自我监督的上限,以预测未观察到的潜在图。除了改进的性能外,Lagraph还为包括基于不变性目标的预测模型的最新成功提供了解释。我们提供了比较毛发与不同领域中相关方法的理论分析。我们的实验结果表明,劳拉在性能方面的优势和鲁棒性对于训练样本量减少了图形级别和节点级任务。
translated by 谷歌翻译
数据增强已广泛用于图像数据和语言数据,但仍然探索图形神经网络(GNN)。现有方法专注于从全局视角增强图表数据,并大大属于两个类型:具有特征噪声注入的结构操纵和对抗训练。但是,最近的图表数据增强方法忽略了GNNS“消息传递机制的本地信息的重要性。在这项工作中,我们介绍了本地增强,这通过其子图结构增强了节点表示的局部。具体而言,我们将数据增强模拟为特征生成过程。鉴于节点的功能,我们的本地增强方法了解其邻居功能的条件分布,并生成更多邻居功能,以提高下游任务的性能。基于本地增强,我们进一步设计了一个新颖的框架:La-GNN,可以以即插即用的方式应用于任何GNN模型。广泛的实验和分析表明,局部增强一致地对各种基准的各种GNN架构始终如一地产生性能改进。
translated by 谷歌翻译
生成的自我监督学习(SSL),尤其是蒙面自动编码器,已成为最令人兴奋的学习范式之一,并且在处理图形数据方面表现出了巨大的潜力。但是,现实世界图总是异质的,它提出了现有方法忽略的三个关键挑战:1)如何捕获复杂的图形结构? 2)如何合并各种节点属性? 3)如何编码不同的节点位置?鉴于此,我们研究了异质图上生成SSL的问题,并提出了HGMAE,这是一种新型的异质图掩盖自动编码器模型,以应对这些挑战。 HGMAE通过两种创新的掩蔽技术和三种独特的培训策略捕获了全面的图形信息。特别是,我们首先使用动态掩模速率开发Metapath掩盖和自适应属性掩蔽,以实现在异质图上有效和稳定的学习。然后,我们设计了几种培训策略,包括基于Metapath的边缘重建,以采用复杂的结构信息,目标属性恢复以结合各种节点属性,以及位置特征预测以编码节点位置信息。广泛的实验表明,HGMAE在多个数据集上的几个任务上均优于对比度和生成的最新基准。
translated by 谷歌翻译
Graph representation learning has emerged as a powerful technique for addressing real-world problems. Various downstream graph learning tasks have benefited from its recent developments, such as node classification, similarity search, and graph classification. However, prior arts on graph representation learning focus on domain specific problems and train a dedicated model for each graph dataset, which is usually non-transferable to out-of-domain data. Inspired by the recent advances in pre-training from natural language processing and computer vision, we design Graph Contrastive Coding (GCC) 1 -a self-supervised graph neural network pre-training framework-to capture the universal network topological properties across multiple networks. We design GCC's pre-training task as subgraph instance discrimination in and across networks and leverage contrastive learning to empower graph neural networks to learn the intrinsic and transferable structural representations. We conduct extensive experiments on three graph learning tasks and ten graph datasets. The results show that GCC pre-trained on a collection of diverse datasets can achieve competitive or better performance to its task-specific and trained-from-scratch counterparts. This suggests that the pre-training and fine-tuning paradigm presents great potential for graph representation learning.
translated by 谷歌翻译
近年来,异构图形神经网络(HGNNS)一直在开花,但每个工作所使用的独特数据处理和评估设置会让他们的进步完全了解。在这项工作中,我们通过使用其官方代码,数据集,设置和超参数来展示12个最近的HGNN的系统再现,揭示了关于HGNN的进展的令人惊讶的结果。我们发现,由于设置不当,简单的均匀GNN,例如GCN和GAT在很大程度上低估了。具有适当输入的GAT通常可以匹配或优于各种场景的所有现有HGNN。为了促进稳健和可重复的HGNN研究,我们构建异构图形基准(HGB),由具有三个任务的11个不同数据集组成。 HGB标准化异构图数据分割,特征处理和性能评估的过程。最后,我们介绍了一个简单但非常强大的基线简单 - HGN - 这显着优于HGB上以前的所有模型 - 以加速未来HGNN的进步。
translated by 谷歌翻译
图表自我监督学习已被极大地用于从未标记的图表中学习表示形式。现有方法可以大致分为预测性学习和对比度学习,在这种学习中,后者通过更好的经验表现吸引了更多的研究注意力。我们认为,与对比模型相比,具有潜在增强和强大的解码器武器的预测模型可以实现可比较甚至更好的表示能力。在这项工作中,我们将数据增强引入潜在空间,以进行卓越的概括和提高效率。一个名为Wiener Graph DeonStolutional网络的新型图解码器相应地设计为从增强潜伏表示的信息重建。理论分析证明了图形滤波器的出色重建能力。各种数据集的广泛实验结果证明了我们方法的有效性。
translated by 谷歌翻译
图对比度学习已被证明是图形神经网络(GNN)预训练的有效任务。但是,一个关键问题可能会严重阻碍现有作品中的代表权:当前方法创建的积极实例通常会错过图表的关键信息,甚至会错过非法实例(例如分子生成中的非化学意识图)。为了解决此问题,我们建议直接从训练集中的现有图中选择正图实例,该实例最终保持与目标图的合法性和相似性。我们的选择基于某些特定于域的成对相似性测量以及从层次图编码图中的相似性关系的采样。此外,我们开发了一种自适应节点级预训练方法,以动态掩盖节点在图中均匀分布。我们对来自各个域的$ 13 $图形分类和节点分类基准数据集进行了广泛的实验。结果表明,通过我们的策略预先培训的GNN模型可以胜过那些训练有素的从划痕模型以及通过现有方法获得的变体。
translated by 谷歌翻译
已经提出了图形神经网络(GNN)预训练方法来增强GNN的能力。具体而言,首先在大规模的未标记图上预先训练GNN,然后在单独的小标记图上进行微调,以用于下游应用程序,例如节点分类。一种流行的预训练方法是掩盖一部分边缘,并接受了GNN的培训以恢复它们。但是,这种生成方法遭受了图不匹配。也就是说,输入到GNN偏离原始图的蒙版图。为了减轻此问题,我们提出了DIP-GNN(图神经网络的歧视性预训练)。具体来说,我们训练一个发电机以恢复蒙版边缘的身份,同时,我们训练一个判别器,以区分生成的边缘与原始图的边缘。在我们的框架中,鉴别器看到的图形更好地匹配原始图,因为生成器可以恢复蒙版边缘的一部分。大规模同质和异质图的广泛实验证明了该框架的有效性。
translated by 谷歌翻译
Generalizable, transferrable, and robust representation learning on graph-structured data remains a challenge for current graph neural networks (GNNs). Unlike what has been developed for convolutional neural networks (CNNs) for image data, self-supervised learning and pre-training are less explored for GNNs. In this paper, we propose a graph contrastive learning (GraphCL) framework for learning unsupervised representations of graph data. We first design four types of graph augmentations to incorporate various priors. We then systematically study the impact of various combinations of graph augmentations on multiple datasets, in four different settings: semi-supervised, unsupervised, and transfer learning as well as adversarial attacks. The results show that, even without tuning augmentation extents nor using sophisticated GNN architectures, our GraphCL framework can produce graph representations of similar or better generalizability, transferrability, and robustness compared to state-of-the-art methods. We also investigate the impact of parameterized graph augmentation extents and patterns, and observe further performance gains in preliminary experiments. Our codes are available at: https://github.com/Shen-Lab/GraphCL.
translated by 谷歌翻译
Existing graph contrastive learning methods rely on augmentation techniques based on random perturbations (e.g., randomly adding or dropping edges and nodes). Nevertheless, altering certain edges or nodes can unexpectedly change the graph characteristics, and choosing the optimal perturbing ratio for each dataset requires onerous manual tuning. In this paper, we introduce Implicit Graph Contrastive Learning (iGCL), which utilizes augmentations in the latent space learned from a Variational Graph Auto-Encoder by reconstructing graph topological structure. Importantly, instead of explicitly sampling augmentations from latent distributions, we further propose an upper bound for the expected contrastive loss to improve the efficiency of our learning algorithm. Thus, graph semantics can be preserved within the augmentations in an intelligent way without arbitrary manual design or prior human knowledge. Experimental results on both graph-level and node-level tasks show that the proposed method achieves state-of-the-art performance compared to other benchmarks, where ablation studies in the end demonstrate the effectiveness of modules in iGCL.
translated by 谷歌翻译
Inspired by the impressive success of contrastive learning (CL), a variety of graph augmentation strategies have been employed to learn node representations in a self-supervised manner. Existing methods construct the contrastive samples by adding perturbations to the graph structure or node attributes. Although impressive results are achieved, it is rather blind to the wealth of prior information assumed: with the increase of the perturbation degree applied on the original graph, 1) the similarity between the original graph and the generated augmented graph gradually decreases; 2) the discrimination between all nodes within each augmented view gradually increases. In this paper, we argue that both such prior information can be incorporated (differently) into the contrastive learning paradigm following our general ranking framework. In particular, we first interpret CL as a special case of learning to rank (L2R), which inspires us to leverage the ranking order among positive augmented views. Meanwhile, we introduce a self-ranking paradigm to ensure that the discriminative information among different nodes can be maintained and also be less altered to the perturbations of different degrees. Experiment results on various benchmark datasets verify the effectiveness of our algorithm compared with the supervised and unsupervised models.
translated by 谷歌翻译
Many applications of machine learning require a model to make accurate predictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and then fine-tune it on a downstream task of interest. While pre-training has been effective in many language and vision domains, it remains an open question how to effectively use pre-training on graph datasets. In this paper, we develop a new strategy and self-supervised methods for pre-training Graph Neural Networks (GNNs). The key to the success of our strategy is to pre-train an expressive GNN at the level of individual nodes as well as entire graphs so that the GNN can learn useful local and global representations simultaneously. We systematically study pre-training on multiple graph classification datasets. We find that naïve strategies, which pre-train GNNs at the level of either entire graphs or individual nodes, give limited improvement and can even lead to negative transfer on many downstream tasks. In contrast, our strategy avoids negative transfer and improves generalization significantly across downstream tasks, leading up to 9.4% absolute improvements in ROC-AUC over non-pre-trained models and achieving state-of-the-art performance for molecular property prediction and protein function prediction.However, pre-training on graph datasets remains a hard challenge. Several key studies (
translated by 谷歌翻译
无监督的图形表示学习是图形数据的非琐碎主题。在结构化数据的无监督代表学习中对比学习和自我监督学习的成功激发了图表上的类似尝试。使用对比损耗的当前无监督的图形表示学习和预培训主要基于手工增强图数据之间的对比度。但是,由于不可预测的不变性,图数据增强仍然没有很好地探索。在本文中,我们提出了一种新颖的协作图形神经网络对比学习框架(CGCL),它使用多个图形编码器来观察图形。不同视图观察的特征充当了图形编码器之间对比学习的图表增强,避免了任何扰动以保证不变性。 CGCL能够处理图形级和节点级表示学习。广泛的实验表明CGCL在无监督的图表表示学习中的优势以及图形表示学习的手工数据增强组合的非必要性。
translated by 谷歌翻译
蒙面的自动编码器是可扩展的视觉学习者,因为Mae \ Cite {He2022masked}的标题表明,视觉中的自我监督学习(SSL)可能会采用与NLP中类似的轨迹。具体而言,具有蒙版预测(例如BERT)的生成借口任务已成为NLP中的事实上的标准SSL实践。相比之下,他们的歧视性对应物(例如对比度学习)掩埋了视力中的生成方法的早期尝试;但是,蒙版图像建模的成功已恢复了屏蔽自动编码器(过去通常被称为DeNosing AutoCoder)。作为在NLP中与Bert弥合差距的一个里程碑,蒙面自动编码器吸引了对SSL在视觉及其他方面的前所未有的关注。这项工作对蒙面自动编码器进行了全面的调查,以洞悉SSL的有希望的方向。作为第一个使用蒙版自动编码器审查SSL的人,这项工作通过讨论其历史发展,最新进度以及对不同应用的影响,重点介绍其在视觉中的应用。
translated by 谷歌翻译
Although substantial efforts have been made using graph neural networks (GNNs) for AI-driven drug discovery (AIDD), effective molecular representation learning remains an open challenge, especially in the case of insufficient labeled molecules. Recent studies suggest that big GNN models pre-trained by self-supervised learning on unlabeled datasets enable better transfer performance in downstream molecular property prediction tasks. However, they often require large-scale datasets and considerable computational resources, which is time-consuming, computationally expensive, and environmentally unfriendly. To alleviate these limitations, we propose a novel pre-training model for molecular representation learning, Bi-branch Masked Graph Transformer Autoencoder (BatmanNet). BatmanNet features two tailored and complementary graph autoencoders to reconstruct the missing nodes and edges from a masked molecular graph. To our surprise, BatmanNet discovered that the highly masked proportion (60%) of the atoms and bonds achieved the best performance. We further propose an asymmetric graph-based encoder-decoder architecture for either nodes and edges, where a transformer-based encoder only takes the visible subset of nodes or edges, and a lightweight decoder reconstructs the original molecule from the latent representation and mask tokens. With this simple yet effective asymmetrical design, our BatmanNet can learn efficiently even from a much smaller-scale unlabeled molecular dataset to capture the underlying structural and semantic information, overcoming a major limitation of current deep neural networks for molecular representation learning. For instance, using only 250K unlabelled molecules as pre-training data, our BatmanNet with 2.575M parameters achieves a 0.5% improvement on the average AUC compared with the current state-of-the-art method with 100M parameters pre-trained on 11M molecules.
translated by 谷歌翻译
使用图神经网络(GNN)提取分子的信息表示,对于AI驱动的药物发现至关重要。最近,图形研究界一直在试图复制自然语言处理预处理的成功,并获得了一些成功。但是,我们发现在许多情况下,自我监督预审计对分子数据的益处可以忽略不计。我们对GNN预处理的关键组成部分进行了彻底的消融研究,包括预处理目标,数据拆分方法,输入特征,预处理数据集量表和GNN体系结构,以决定下游任务的准确性。我们的第一个重要发现是,在许多情况下,自我监督的图表预处理没有统计学上的显着优势。其次,尽管可以通过额外的监督预处理可以观察到改进,但通过更丰富或更平衡的数据拆分,改进可能会减少。第三,实验性超参数对下游任务的准确性具有更大的影响,而不是训练训练的任务。我们假设对分子进行预训练的复杂性不足,从而导致下游任务的可转移知识较低。
translated by 谷歌翻译
自我监督的学习提供了一个有希望的途径,消除了在图形上的代表学习中的昂贵标签信息的需求。然而,为了实现最先进的性能,方法通常需要大量的负例,并依赖于复杂的增强。这可能是昂贵的,特别是对于大图。为了解决这些挑战,我们介绍了引导的图形潜伏(BGRL) - 通过预测输入的替代增强来学习图表表示学习方法。 BGRL仅使用简单的增强,并减轻了对否定例子对比的需求,因此通过设计可扩展。 BGRL胜过或匹配现有的几种建立的基准,同时降低了内存成本的2-10倍。此外,我们表明,BGR1可以缩放到半监督方案中的数亿个节点的极大的图表 - 实现最先进的性能并改善监督基线,其中表示仅通过标签信息而塑造。特别是,我们的解决方案以BGRL为中心,将kdd杯2021的开放图基准的大规模挑战组成了一个获奖条目,在比所有先前可用的基准更大的级别的图形订单上,从而展示了我们方法的可扩展性和有效性。
translated by 谷歌翻译