Out-of-distribution (OOD) generalization on graphs is drawing widespread attention. However, existing efforts mainly focus on the OOD issue of correlation shift. While another type, covariate shift, remains largely unexplored but is the focus of this work. From a data generation view, causal features are stable substructures in data, which play key roles in OOD generalization. While their complementary parts, environments, are unstable features that often lead to various distribution shifts. Correlation shift establishes spurious statistical correlations between environments and labels. In contrast, covariate shift means that there exist unseen environmental features in test data. Existing strategies of graph invariant learning and data augmentation suffer from limited environments or unstable causal features, which greatly limits their generalization ability on covariate shift. In view of that, we propose a novel graph augmentation strategy: Adversarial Causal Augmentation (AdvCA), to alleviate the covariate shift. Specifically, it adversarially augments the data to explore diverse distributions of the environments. Meanwhile, it keeps the causal features invariant across diverse environments. It maintains the environmental diversity while ensuring the invariance of the causal features, thereby effectively alleviating the covariate shift. Extensive experimental results with in-depth analyses demonstrate that AdvCA can outperform 14 baselines on synthetic and real-world datasets with various covariate shifts.
translated by 谷歌翻译
分数(OOD)学习涉及培训和测试数据遵循不同分布的方案。尽管在机器学习中已经深入研究了一般的OOD问题,但图形OOD只是一个新兴领域。目前,缺少针对图形OOD方法评估的系统基准。在这项工作中,我们旨在为图表开发一个被称为GOOD的OOD基准。我们明确地在协变量和概念变化和设计数据拆分之间进行了区分,以准确反映不同的变化。我们考虑图形和节点预测任务,因为在设计变化时存在关键差异。总体而言,Good包含8个具有14个域选择的数据集。当与协变量,概念和无移位结合使用时,我们获得了42个不同的分裂。我们在7种常见的基线方法上提供了10种随机运行的性能结果。这总共导致294个数据集模型组合。我们的结果表明,分布和OOD设置之间的性能差距很大。我们的结果还阐明了通过不同方法的协变量和概念转移之间的不同性能趋势。我们的良好基准是一个不断增长的项目,并希望随着该地区的发展,数量和种类繁多。可以通过$ \ href {https://github.com/divelab/good/} {\ text {https://github.com/divelab/good/good/}} $访问良好基准。
translated by 谷歌翻译
学习强大的表示是图形神经网络(GNN)的一个中心主题。它需要从输入图中炼制关键信息,而不是琐碎的模式,以丰富表示。为此,图表注意力和汇集方法占上风。他们主要遵循“学会参加”的范式。它最大限度地提高了上述子图和地面真理标签之间的相互信息。然而,这种训练范例易于捕获微级子图和标签之间的虚假相关性。这种杂散的相关性对分布(ID)测试评估有益,但在分布外(OOD)测试数据中引起差的概括。在这项工作中,我们从因果角度重新审视GNN建模。在我们的因果假设之上,琐碎的信息是关键信息和标签之间的混淆,它在它们之间打开了一个后门路径,使它们保持虚拟相关。因此,我们提出了一个新的解压缩训练范式(DTP),更好地减轻了批评信息的混淆效果并锁存,以提高表示和泛化能力。具体而言,我们采用注意模块解开关键的子图和微不足道的子图。然后我们使每个关键的子图相当与不同的琐碎子图相互作用,以实现稳定的预测。它允许GNN捕获一个更可靠的子图,其与标签的关系跨越不同的分布。我们对综合和现实世界数据集进行了广泛的实验,以证明有效性。
translated by 谷歌翻译
Graph machine learning has been extensively studied in both academia and industry. Although booming with a vast number of emerging methods and techniques, most of the literature is built on the in-distribution hypothesis, i.e., testing and training graph data are identically distributed. However, this in-distribution hypothesis can hardly be satisfied in many real-world graph scenarios where the model performance substantially degrades when there exist distribution shifts between testing and training graph data. To solve this critical problem, out-of-distribution (OOD) generalization on graphs, which goes beyond the in-distribution hypothesis, has made great progress and attracted ever-increasing attention from the research community. In this paper, we comprehensively survey OOD generalization on graphs and present a detailed review of recent advances in this area. First, we provide a formal problem definition of OOD generalization on graphs. Second, we categorize existing methods into three classes from conceptually different perspectives, i.e., data, model, and learning strategy, based on their positions in the graph machine learning pipeline, followed by detailed discussions for each category. We also review the theories related to OOD generalization on graphs and introduce the commonly used graph datasets for thorough evaluations. Finally, we share our insights on future research directions. This paper is the first systematic and comprehensive review of OOD generalization on graphs, to the best of our knowledge.
translated by 谷歌翻译
尽管最近在欧几里得数据(例如图像)上使用不变性原理(OOD)概括(例如图像),但有关图数据的研究仍然受到限制。与图像不同,图形的复杂性质给采用不变性原理带来了独特的挑战。特别是,图表上的分布变化可以以多种形式出现,例如属性和结构,因此很难识别不变性。此外,在欧几里得数据上通常需要的域或环境分区通常需要的图形可能非常昂贵。为了弥合这一差距,我们提出了一个新的框架,以捕获图形的不变性,以在各种分配变化下进行保证的OOD概括。具体而言,我们表征了具有因果模型的图形上的潜在分布变化,得出结论,当模型仅关注包含有关标签原因最多信息的子图时,可以实现图形上的OOD概括。因此,我们提出了一个信息理论目标,以提取最大地保留不变的阶级信息的所需子图。用这些子图学习不受分配变化的影响。对合成和现实世界数据集进行的广泛实验,包括在AI ADED药物发现中充满挑战的环境,验证了我们方法的上等OOD概括能力。
translated by 谷歌翻译
图表神经网络(GNNS)在测试和训练图数据来自相同分布时取得了令人印象深刻的性能。然而,现有的GNN缺乏分发的泛化能力,使得它们的性能在测试和训练图数据之间存在分布时显着降低。为了解决这个问题,在这项工作中,我们提出了一个用于在具有训练图的不同分布的看不见的分布的看不见的令人满意的令人满意的令人满意的通用图形神经网络(OOD-GNN)。我们所提出的OOD-GNN采用新颖的非线性图形表示去序方法,利用随机傅里叶特征,这鼓励模型通过迭代优化样本图权重和图形编码器来消除相关和无关的图表表示之间的统计依赖性。我们进一步设计了一个全局重量估计器,以学习训练图的权重,使得图形表示中的变量被迫独立。学习权重有助于图形编码器摆脱虚假相关性,并且反过来,更集中学习鉴别图形表示与地面真理标签之间的真实连接。我们进行广泛的实验,以验证两个合成和12个现实世界数据集的分发外概括能力,分配换档。结果表明,我们所提出的OOD-GNN显着优于最先进的基线。
translated by 谷歌翻译
建议图表神经网络(GNNS)在不考虑训练和测试图之间的不可知分布的情况下,诱导GNN的泛化能力退化在分布外(OOD)设置。这种退化的根本原因是大多数GNN是基于I.I.D假设开发的。在这种设置中,GNN倾向于利用在培训中存在的微妙统计相关性用于预测,即使它是杂散的相关性。然而,这种杂散的相关性可能在测试环境中改变,导致GNN的失败。因此,消除了杂散相关的影响对于稳定的GNN来说是至关重要的。为此,我们提出了一个普遍的因果代表框架,称为稳定凝球。主要思想是首先从图数据中提取高级表示,并诉诸因因果推理的显着能力,以帮助模型摆脱虚假相关性。特别是,我们利用图形池化层以提取基于子图的表示作为高级表示。此外,我们提出了一种因果变量区别,以纠正偏置训练分布。因此,GNN将更多地集中在稳定的相关性上。对合成和现实世界ood图数据集的广泛实验良好地验证了所提出的框架的有效性,灵活性和可解释性。
translated by 谷歌翻译
Despite impressive success in many tasks, deep learning models are shown to rely on spurious features, which will catastrophically fail when generalized to out-of-distribution (OOD) data. Invariant Risk Minimization (IRM) is proposed to alleviate this issue by extracting domain-invariant features for OOD generalization. Nevertheless, recent work shows that IRM is only effective for a certain type of distribution shift (e.g., correlation shift) while it fails for other cases (e.g., diversity shift). Meanwhile, another thread of method, Adversarial Training (AT), has shown better domain transfer performance, suggesting that it has the potential to be an effective candidate for extracting domain-invariant features. This paper investigates this possibility by exploring the similarity between the IRM and AT objectives. Inspired by this connection, we propose Domainwise Adversarial Training (DAT), an AT-inspired method for alleviating distribution shift by domain-specific perturbations. Extensive experiments show that our proposed DAT can effectively remove domain-varying features and improve OOD generalization under both correlation shift and diversity shift.
translated by 谷歌翻译
研究兴趣大大增加了将数据驱动方法应用于力学问题的问题。尽管传统的机器学习(ML)方法已经实现了许多突破,但它们依赖于以下假设:培训(观察到的)数据和测试(看不见)数据是独立的且分布相同的(i.i.d)。因此,当应用于未知的测试环境和数据分布转移的现实世界力学问题时,传统的ML方法通常会崩溃。相反,分布(OOD)的概括假定测试数据可能会发生变化(即违反I.I.D.假设)。迄今为止,已经提出了多种方法来改善ML方法的OOD概括。但是,由于缺乏针对OOD回归问题的基准数据集,因此这些OOD方法在主导力学领域的回归问题上的效率仍然未知。为了解决这个问题,我们研究了机械回归问题的OOD泛化方法的性能。具体而言,我们确定了三个OOD问题:协变量移位,机制移位和采样偏差。对于每个问题,我们创建了两个基准示例,以扩展机械MNIST数据集收集,并研究了流行的OOD泛化方法在这些机械特定的回归问题上的性能。我们的数值实验表明,在大多数情况下,与传统的ML方法相比,在大多数情况下,在这些OOD问题上的传统ML方法的性能更好,但迫切需要开发更强大的OOD概括方法,这些方法在多个OOD场景中有效。总体而言,我们希望这项研究以及相关的开放访问基准数据集将进一步开发用于机械特定回归问题的OOD泛化方法。
translated by 谷歌翻译
从观察数据中学习因果结构是机器学习的基本挑战。但是,大多数常用的可区分因果发现方法是不可识别的,这将此问题变成了容易发生数据偏差的连续优化任务。在许多现实生活中,数据是从不同环境中收集的,在不同的环境中,功能关系在整个环境中保持一致,而添加噪声的分布可能会有所不同。本文提出了可区分的因果发现(DICD),利用基于可区分框架的多环境信息,以避免学习虚假边缘和错误的因果方向。具体而言,DICD旨在在消除环境依赖性相关性的同时发现环境不变的因果关系。我们进一步制定了强制执行目标结构方程模型的约束,以在整个环境中保持最佳状态。在温和条件下提供了足够的环境,提供了针对拟议DICD的可识别性的理论保证。关于合成和现实世界数据集的广泛实验验证了DICD优于最先进的因果发现方法,而SHD中最高36%。我们的代码将是开源的。
translated by 谷歌翻译
需要解释的图表学习是需要的,因为许多科学应用都取决于学习模型来从图形结构数据中收集见解。先前的工作主要集中在使用事后方法来解释预训练的模型(尤其是图形神经网络模型)。他们反对固有的可解释模型,因为对这些模型的良好解释通常是以其预测准确性为代价。而且,广泛使用的固有解释的注意力机制通常无法在图形学习任务中提供忠实的解释。在这项工作中,我们通过提出图形随机关注(GSAT)来解决这两个问题,这是一种来自信息瓶颈原理的注意机制。 GSAT利用随机关注来阻止从任务 - 核定图组件中的信息,同时学习降低随机性的注意力以选择与任务相关的子图以进行解释。 GSAT也可以通过随机注意机制应用于微调和解释预训练的模型。八个数据集的广泛实验表明,GSAT在解释AUC中的最高最高为20%$ \ uparrow $,而预测准确性则高于最高的最高$ \ uparrow $。
translated by 谷歌翻译
尽管机器学习模型迅速推进了各种现实世界任务的最先进,但鉴于这些模型对虚假相关性的脆弱性,跨域(OOD)的概括仍然是一个挑战性的问题。尽管当前的域概括方法通常着重于通过新的损耗函数设计在不同域上实施某些不变性属性,但我们提出了一种平衡的迷你批次采样策略,以减少观察到的训练分布中域特异性的虚假相关性。更具体地说,我们提出了一种两步方法,该方法1)识别虚假相关性的来源,以及2)通过在确定的来源上匹配,构建平衡的迷你批次而没有虚假相关性。我们提供了伪造来源的可识别性保证,并表明我们提出的方法是从所有培训环境中平衡,无虚拟分布的样本。实验是在三个具有伪造相关性的计算机视觉数据集上进行的,从经验上证明,与随机的迷你批次采样策略相比,我们平衡的微型批次采样策略可改善四个不同建立的域泛化模型基线的性能。
translated by 谷歌翻译
域泛化(DG)的主要挑战是克服多个训练域和看不见的测试域之间的潜在分布偏移。一类流行的DG算法旨在学习在训练域中具有不变因果关系的表示。但是,某些特征,称为\ emph {伪不变特征},可能是培训域中的不变性,但不是测试域,并且可以大大降低现有算法的性能。为了解决这个问题,我们提出了一种新颖的算法,称为不变信息瓶颈(IIB),该算法学习跨越训练和测试域的最小值的最小值。通过最大限度地减少表示和输入之间的相互信息,IIB可以减轻其对伪不变特征的依赖,这对于DG是期望的。为了验证IIB原则的有效性,我们对大型DG基准进行了广泛的实验。结果表明,在两个评估度量标准中,IIB的IIIb平均超过2.8 \%和3.8 \%的准确性。
translated by 谷歌翻译
理由定义为最能解释或支持机器学习模型预测的输入功能的子集。基本原理识别改善了神经网络在视觉和语言数据上的普遍性和解释性。在诸如分子和聚合物属性预测之类的图应用中,识别称为图理由的代表性子图结构在图神经网络的性能中起着至关重要的作用。现有的图形合并和/或分发干预方法缺乏示例,无法学习确定最佳图理由。在这项工作中,我们介绍了一个名为“环境替代”的新的增强操作,该操作自动创建虚拟数据示例以改善基本原理识别。我们提出了一个有效的框架,该框架在潜在空间中对真实和增强的示例进行基本环境分离和表示学习,以避免显式图解码和编码的高复杂性。与最近的技术相比,对七个分子和四个聚合物实际数据集进行的实验证明了拟议的基于增强的图形合理化框架的有效性和效率。
translated by 谷歌翻译
Machine learning models rely on various assumptions to attain high accuracy. One of the preliminary assumptions of these models is the independent and identical distribution, which suggests that the train and test data are sampled from the same distribution. However, this assumption seldom holds in the real world due to distribution shifts. As a result models that rely on this assumption exhibit poor generalization capabilities. Over the recent years, dedicated efforts have been made to improve the generalization capabilities of these models collectively known as -- \textit{domain generalization methods}. The primary idea behind these methods is to identify stable features or mechanisms that remain invariant across the different distributions. Many generalization approaches employ causal theories to describe invariance since causality and invariance are inextricably intertwined. However, current surveys deal with the causality-aware domain generalization methods on a very high-level. Furthermore, we argue that it is possible to categorize the methods based on how causality is leveraged in that method and in which part of the model pipeline is it used. To this end, we categorize the causal domain generalization methods into three categories, namely, (i) Invariance via Causal Data Augmentation methods which are applied during the data pre-processing stage, (ii) Invariance via Causal representation learning methods that are utilized during the representation learning stage, and (iii) Invariance via Transferring Causal mechanisms methods that are applied during the classification stage of the pipeline. Furthermore, this survey includes in-depth insights into benchmark datasets and code repositories for domain generalization methods. We conclude the survey with insights and discussions on future directions.
translated by 谷歌翻译
Distributional shift is one of the major obstacles when transferring machine learning prediction systems from the lab to the real world. To tackle this problem, we assume that variation across training domains is representative of the variation we might encounter at test time, but also that shifts at test time may be more extreme in magnitude. In particular, we show that reducing differences in risk across training domains can reduce a model's sensitivity to a wide range of extreme distributional shifts, including the challenging setting where the input contains both causal and anticausal elements. We motivate this approach, Risk Extrapolation (REx), as a form of robust optimization over a perturbation set of extrapolated domains (MM-REx), and propose a penalty on the variance of training risks (V-REx) as a simpler variant. We prove that variants of REx can recover the causal mechanisms of the targets, while also providing some robustness to changes in the input distribution ("covariate shift"). By tradingoff robustness to causally induced distributional shifts and covariate shift, REx is able to outperform alternative methods such as Invariant Risk Minimization in situations where these types of shift co-occur.
translated by 谷歌翻译
域的概括(DG)通过利用来自多个相关分布或域的标记培训数据在看不见的测试分布上表现良好的预测因子。为了实现这一目标,标准公式优化了所有可能域的最差性能。但是,由于最糟糕的转变在实践中的转变极不可能,这通常会导致过度保守的解决方案。实际上,最近的一项研究发现,没有DG算法在平均性能方面优于经验风险最小化。在这项工作中,我们认为DG既不是最坏的问题,也不是一个普通的问题,而是概率问题。为此,我们为DG提出了一个概率框架,我们称之为可能的域概括,其中我们的关键想法是在训练期间看到的分配变化应在测试时告诉我们可能的变化。为了实现这一目标,我们将培训和测试域明确关联为从同一基础元分布中获取的,并提出了一个新的优化问题 - 分数风险最小化(QRM) - 要求该预测因子以很高的概率概括。然后,我们证明了QRM:(i)产生的预测因子,这些预测因素将具有所需概率的新域(给定足够多的域和样本); (ii)随着概括的所需概率接近一个,恢复因果预测因子。在我们的实验中,我们引入了针对DG的更全面的以分位数评估协议,并表明我们的算法在真实和合成数据上的最先进基准都优于最先进的基准。
translated by 谷歌翻译
本文着重于由于看不见的分布变化而导致性能下降的图表上的分布概括。以前的图形域概括始终诉诸于不同源域之间的不变预测因子。但是,他们假设在培训期间提供了足够的源域,为现实应用带来了巨大挑战。相比之下,我们通过从源域中构造多个种群来提出一个新的图形域概括框架,称为DPS。具体而言,DPS旨在发现多个\ textbf {d} iverse和\ textbf {p}可redictable \ textbf {s}带有一组发电机的ubgraphs,即,子图是彼此不同的,但它们彼此不同,但所有这些都与相同的语义共享输入图。这些生成的源域被利用以学习跨域的\ textIt {Equi-Prestivical}图神经网络(GNN),这有望很好地概括到看不见的目标域。通常,DPS是模型不合时宜的,可以与各种GNN骨架合并。节点级别和图形基准测试的广泛实验表明,所提出的DPS为各种图形域概括任务实现了令人印象深刻的性能。
translated by 谷歌翻译
流行的图神经网络模型在图表学习方面取得了重大进展。但是,在本文中,我们发现了一个不断被忽视的现象:用完整图测试的预训练的图表学习模型的表现不佳,该模型用良好的图表测试。该观察结果表明,图中存在混杂因素,这可能会干扰模型学习语义信息,而当前的图表表示方法并未消除其影响。为了解决这个问题,我们建议强大的因果图表示学习(RCGRL)学习可靠的图形表示,以防止混杂效应。 RCGRL引入了一种主动方法,可以在无条件的力矩限制下生成仪器变量,该方法使图表学习模型能够消除混杂因素,从而捕获与下游预测有因果关系的歧视性信息。我们提供定理和证明,以保证拟议方法的理论有效性。从经验上讲,我们对合成数据集和多个基准数据集进行了广泛的实验。结果表明,与最先进的方法相比,RCGRL实现了更好的预测性能和泛化能力。
translated by 谷歌翻译
The problem of covariate-shift generalization has attracted intensive research attention. Previous stable learning algorithms employ sample reweighting schemes to decorrelate the covariates when there is no explicit domain information about training data. However, with finite samples, it is difficult to achieve the desirable weights that ensure perfect independence to get rid of the unstable variables. Besides, decorrelating within stable variables may bring about high variance of learned models because of the over-reduced effective sample size. A tremendous sample size is required for these algorithms to work. In this paper, with theoretical justification, we propose SVI (Sparse Variable Independence) for the covariate-shift generalization problem. We introduce sparsity constraint to compensate for the imperfectness of sample reweighting under the finite-sample setting in previous methods. Furthermore, we organically combine independence-based sample reweighting and sparsity-based variable selection in an iterative way to avoid decorrelating within stable variables, increasing the effective sample size to alleviate variance inflation. Experiments on both synthetic and real-world datasets demonstrate the improvement of covariate-shift generalization performance brought by SVI.
translated by 谷歌翻译