Graph Neural Networks (GNNs) have shown satisfying performance on various graph learning tasks. To achieve better fitting capability, most GNNs are with a large number of parameters, which makes these GNNs computationally expensive. Therefore, it is difficult to deploy them onto edge devices with scarce computational resources, e.g., mobile phones and wearable smart devices. Knowledge Distillation (KD) is a common solution to compress GNNs, where a light-weighted model (i.e., the student model) is encouraged to mimic the behavior of a computationally expensive GNN (i.e., the teacher GNN model). Nevertheless, most existing GNN-based KD methods lack fairness consideration. As a consequence, the student model usually inherits and even exaggerates the bias from the teacher GNN. To handle such a problem, we take initial steps towards fair knowledge distillation for GNNs. Specifically, we first formulate a novel problem of fair knowledge distillation for GNN-based teacher-student frameworks. Then we propose a principled framework named RELIANT to mitigate the bias exhibited by the student model. Notably, the design of RELIANT is decoupled from any specific teacher and student model structures, and thus can be easily adapted to various GNN-based KD frameworks. We perform extensive experiments on multiple real-world datasets, which corroborates that RELIANT achieves less biased GNN knowledge distillation while maintaining high prediction utility.
translated by 谷歌翻译
图形神经网络(GNN)表现出令人满意的各种图分析问题的性能。因此,在各种决策方案中,它们已成为\ emph {de exto}解决方案。但是,GNN可以针对某些人口亚组产生偏差的结果。最近的一些作品在经验上表明,输入网络的偏见结构是GNN的重要来源。然而,没有系统仔细检查输入网络结构的哪一部分会导致对任何给定节点的偏见预测。对输入网络的结构如何影响GNN结果的偏见的透明度很大,在很大程度上限制了在各种决策方案中的安全采用GNN。在本文中,我们研究了GNN中偏见的结构解释的新研究问题。具体而言,我们提出了一个新颖的事后解释框架,以识别可以最大程度地解释出偏见的两个边缘集,并最大程度地促进任何给定节点的GNN预测的公平水平。这种解释不仅提供了对GNN预测的偏见/公平性的全面理解,而且在建立有效但公平的GNN模型方面具有实际意义。对现实世界数据集的广泛实验验证了拟议框架在为GNN偏见提供有效的结构解释方面的有效性。可以在https://github.com/yushundong/referee上找到开源代码。
translated by 谷歌翻译
由于学术和工业领域的异质图无处不在,研究人员最近提出了许多异质图神经网络(HGNN)。在本文中,我们不再采用更强大的HGNN模型,而是有兴趣设计一个多功能的插件模块,该模块解释了从预先训练的HGNN中提取的关系知识。据我们所知,我们是第一个在异质图上提出高阶(雇用)知识蒸馏框架的人,无论HGNN的模型体系结构如何,它都可以显着提高预测性能。具体而言,我们的雇用框架最初执行一阶节点级知识蒸馏,该蒸馏曲线及其预测逻辑编码了老师HGNN的语义。同时,二阶关系级知识蒸馏模仿了教师HGNN生成的不同类型的节点嵌入之间的关系相关性。在各种流行的HGNN模型和三个现实世界的异质图上进行了广泛的实验表明,我们的方法获得了一致且相当大的性能增强,证明了其有效性和泛化能力。
translated by 谷歌翻译
知识蒸馏最近成为一种流行的技术,以改善卷积神经网络的模型泛化能力。然而,它对图形神经网络的影响小于令人满意的,因为图形拓扑和节点属性可能以动态方式改变,并且在这种情况下,静态教师模型引导学生培训不足。在本文中,我们通过在在线蒸馏时期同时培训一组图形神经网络来解决这一挑战,其中组知识发挥作用作为动态虚拟教师,并且有效地捕获了图形神经网络的结构变化。为了提高蒸馏性能,在学生之间转移两种知识,以增强彼此:在图形拓扑和节点属性中反映信息的本地知识,以及反映课程预测的全局知识。随着香草知识蒸馏等,在利用有效的对抗性循环学习框架,将全球知识与KL分歧转移。广泛的实验验证了我们提出的在线对抗蒸馏方法的有效性。
translated by 谷歌翻译
图形神经网络(GNN)已被广泛用于建模图形结构化数据,这是由于其在广泛的实用应用中令人印象深刻的性能。最近,GNNS的知识蒸馏(KD)在图形模型压缩和知识转移方面取得了显着进步。但是,大多数现有的KD方法都需要大量的真实数据,这些数据在实践中不容易获得,并且可能排除其在教师模型对稀有或难以获取数据集培训的情况下的适用性。为了解决这个问题,我们提出了第一个用于图形结构化数据(DFAD-GNN)的无数据对抗知识蒸馏的端到端框架。具体而言,我们的DFAD-GNN采用生成性对抗网络,主要由三个组成部分组成:预先训练的教师模型和学生模型被视为两个歧视者,并利用生成器来衍生训练图来从教师模型进入学生模型。在各种基准模型和六个代表性数据集上进行的广泛实验表明,我们的DFAD-GNN在图形分类任务中显着超过了最新的无数据基线。
translated by 谷歌翻译
Graph Neural Networks (GNNs) have been a prevailing technique for tackling various analysis tasks on graph data. A key premise for the remarkable performance of GNNs relies on complete and trustworthy initial graph descriptions (i.e., node features and graph structure), which is often not satisfied since real-world graphs are often incomplete due to various unavoidable factors. In particular, GNNs face greater challenges when both node features and graph structure are incomplete at the same time. The existing methods either focus on feature completion or structure completion. They usually rely on the matching relationship between features and structure, or employ joint learning of node representation and feature (or structure) completion in the hope of achieving mutual benefit. However, recent studies confirm that the mutual interference between features and structure leads to the degradation of GNN performance. When both features and structure are incomplete, the mismatch between features and structure caused by the missing randomness exacerbates the interference between the two, which may trigger incorrect completions that negatively affect node representation. To this end, in this paper we propose a general GNN framework based on teacher-student distillation to improve the performance of GNNs on incomplete graphs, namely T2-GNN. To avoid the interference between features and structure, we separately design feature-level and structure-level teacher models to provide targeted guidance for student model (base GNNs, such as GCN) through distillation. Then we design two personalized methods to obtain well-trained feature and structure teachers. To ensure that the knowledge of the teacher model is comprehensively and effectively distilled to the student model, we further propose a dual distillation mode to enable the student to acquire as much expert knowledge as possible.
translated by 谷歌翻译
图神经网络(GNN)在图形上学习节点表示方面表现出很大的力量。但是,他们可能会从训练数据中继承历史偏见,从而导致预测的歧视性偏见。尽管某些工作已经开发出公平的GNN,但其中大多数直接从非图形域借用了公平代表性学习技术,而没有考虑GNN中特征传播引起的敏感属性泄漏的潜在问题。但是,我们从经验上观察到,特征传播可能会改变以前无害特征与敏感特征的相关性。这可以看作是敏感信息的泄漏,可以进一步加剧预测中的歧视。因此,我们根据特征相关性设计了两个特征掩盖策略,以突出考虑特征传播和相关性变化在减轻歧视中的重要性。通过我们的分析,我们提出了公平视图图神经网络(FAIRVGNN),以通过自动识别和掩盖敏感的相关特征来生成特征的公平视图,以考虑特征传播后的相关变化。鉴于博学的公平视图,我们适应编码器的夹紧权重,以避免使用敏感相关的功能。现实世界数据集的实验表明,Fairvgnn在模型实用程序和公平性之间取得了更好的权衡。我们的代码可在https://github.com/yuwvandy/fairvgnn上公开获取。
translated by 谷歌翻译
知识蒸馏(KD)证明了其有效性,可以提高图形神经网络(GNN)的性能,其目标是将知识从更深的教师gnn蒸馏成较浅的学生GNN。但是,由于众所周知的过度参数和过度光滑的问题,实际上很难培训令人满意的教师GNN,从而导致实际应用中的知识转移无效。在本文中,我们通过对GNN的加强学习(称为FreeKD)提出了第一个自由方向知识蒸馏框架,而这不再需要提供更深入的良好优化的教师GNN。我们工作的核心思想是协作建立两个较浅的GNN,以通过以层次结构方式通过加强学习来交流知识。正如我们观察到的一个典型的GNN模型在训练过程中通常在不同节点的表现更好,更差的表现,我们设计了一种动态和自由方向的知识转移策略,该策略由两个级别的动作组成:1)节点级别的动作决定了知识的方向。两个网络的相应节点之间的传输;然后2)结构级的动作确定了要传播的节点级别生成的局部结构。从本质上讲,我们的FreeKD是一个一般且原则性的框架,可以自然与不同架构的GNN兼容。在五个基准数据集上进行的广泛实验表明,我们的FreeKD在很大的边距上优于两个基本GNN,并显示了其对各种GNN的功效。更令人惊讶的是,我们的FreeKD比传统的KD算法具有可比性甚至更好的性能,这些KD算法将知识从更深,更强大的教师GNN中提取。
translated by 谷歌翻译
图形神经网络(GNNS)已被证明是在预测建模任务中的Excel,其中底层数据是图形。然而,由于GNN广泛用于人以人为本的应用,因此出现了公平性问题。虽然边缘删除是用于促进GNNS中公平性的常用方法,但是当数据本质上缺少公平连接时,它就无法考虑。在这项工作中,我们考虑未删除的边缘添加方法,促进公平。我们提出了两个模型 - 不可知的算法来执行边缘编辑:蛮力方法和连续近似方法,公平。Fairedit通过利用公平损失的梯度信息来执行有效的边缘编辑,以找到改善公平性的边缘。我们发现Fairedit优于许多数据集和GNN方法的标准培训,同时表现了许多最先进的方法,展示了公平的能力,以改善许多领域和模型的公平性。
translated by 谷歌翻译
本文研究了用于无监督场景的图形神经网络(GNN)的节点表示。具体地,我们推导了理论分析,并在不适当定义的监督信号时,在不同的图形数据集中提供关于GNN的非稳定性能的实证演示。 GNN的性能取决于节点特征平滑度和图形结构的局部性。为了平滑通过图形拓扑和节点功能测量的节点接近度的差异,我们提出了帆 - 一个小说\下划线{s} elf- \下划线{a} u段图对比度\下划线{i} ve \ nignline {l}收入框架,使用两个互补的自蒸馏正则化模块,\ emph {Ie},内部和图间知识蒸馏。我们展示了帆在各种图形应用中的竞争性能。即使使用单个GNN层,Sail也在各种基准数据集中持续竞争或更好的性能,与最先进的基线相比。
translated by 谷歌翻译
Learning fair graph representations for downstream applications is becoming increasingly important, but existing work has mostly focused on improving fairness at the global level by either modifying the graph structure or objective function without taking into account the local neighborhood of a node. In this work, we formally introduce the notion of neighborhood fairness and develop a computational framework for learning such locally fair embeddings. We argue that the notion of neighborhood fairness is more appropriate since GNN-based models operate at the local neighborhood level of a node. Our neighborhood fairness framework has two main components that are flexible for learning fair graph representations from arbitrary data: the first aims to construct fair neighborhoods for any arbitrary node in a graph and the second enables adaption of these fair neighborhoods to better capture certain application or data-dependent constraints, such as allowing neighborhoods to be more biased towards certain attributes or neighbors in the graph.Furthermore, while link prediction has been extensively studied, we are the first to investigate the graph representation learning task of fair link classification. We demonstrate the effectiveness of the proposed neighborhood fairness framework for a variety of graph machine learning tasks including fair link prediction, link classification, and learning fair graph embeddings. Notably, our approach achieves not only better fairness but also increases the accuracy in the majority of cases across a wide variety of graphs, problem settings, and metrics.
translated by 谷歌翻译
With the growth of high-dimensional sparse data in web-scale recommender systems, the computational cost to learn high-order feature interaction in CTR prediction task largely increases, which limits the use of high-order interaction models in real industrial applications. Some recent knowledge distillation based methods transfer knowledge from complex teacher models to shallow student models for accelerating the online model inference. However, they suffer from the degradation of model accuracy in knowledge distillation process. It is challenging to balance the efficiency and effectiveness of the shallow student models. To address this problem, we propose a Directed Acyclic Graph Factorization Machine (KD-DAGFM) to learn the high-order feature interactions from existing complex interaction models for CTR prediction via Knowledge Distillation. The proposed lightweight student model DAGFM can learn arbitrary explicit feature interactions from teacher networks, which achieves approximately lossless performance and is proved by a dynamic programming algorithm. Besides, an improved general model KD-DAGFM+ is shown to be effective in distilling both explicit and implicit feature interactions from any complex teacher model. Extensive experiments are conducted on four real-world datasets, including a large-scale industrial dataset from WeChat platform with billions of feature dimensions. KD-DAGFM achieves the best performance with less than 21.5% FLOPs of the state-of-the-art method on both online and offline experiments, showing the superiority of DAGFM to deal with the industrial scale data in CTR prediction task. Our implementation code is available at: https://github.com/RUCAIBox/DAGFM.
translated by 谷歌翻译
图形预训练策略一直在图形挖掘社区吸引人们的注意力,因为它们在没有任何标签信息的情况下在参数化图形神经网络(GNN)方面的灵活性。关键思想在于通过预测从输入图中提取的掩蔽图信号来编码有价值的信息。为了平衡各种图形信号的重要性(例如节点,边缘,子图),现有方法主要是通过引入超参数来重新进行图形信号的重要性来进行手工设计的。然而,人类对亚最佳高参数的干预通常会注入额外的偏见,并在下游应用中降低了概括性能。本文从新的角度解决了这些局限性,即为预培训GNN提供课程。我们提出了一个名为Mentorgnn的端到端模型,该模型旨在监督具有不同结构和不同特征空间的图表的GNN的预训练过程。为了理解不同粒度的异质图信号,我们提出了一种课程学习范式,该课程自动重新贴出图形信号,以确保对目标域进行良好的概括。此外,我们通过在预先训练的GNN的概括误差上得出自然且可解释的上限,从而对关系数据(即图形)的域自适应问题(即图形)发出了新的启示。有关大量真实图的广泛实验验证并验证了Mentorgnn的性能。
translated by 谷歌翻译
知识蒸馏已成为获得紧凑又有效模型的重要方法。为实现这一目标,培训小型学生模型以利用大型训练有素的教师模型的知识。然而,由于教师和学生之间的能力差距,学生的表现很难达到老师的水平。关于这个问题,现有方法建议通过代理方式减少教师知识的难度。我们认为这些基于代理的方法忽视了教师的知识损失,这可能导致学生遇到容量瓶颈。在本文中,我们从新的角度来缓解能力差距问题,以避免知识损失的目的。我们建议通过对抗性协作学习建立一个更有力的学生,而不是牺牲教师的知识。为此,我们进一步提出了一种逆势协作知识蒸馏(ACKD)方法,有效提高了知识蒸馏的性能。具体来说,我们用多个辅助学习者构建学生模型。同时,我们设计了对抗的对抗性协作模块(ACM),引入注意机制和对抗的学习,以提高学生的能力。四个分类任务的广泛实验显示了拟议的Ackd的优越性。
translated by 谷歌翻译
尽管机器学习模式的发展迅速和巨大成功,但广泛的研究暴露了继承潜在歧视和培训数据的社会偏见的缺点。这种现象阻碍了他们在高利益应用上采用。因此,已经采取了许多努力开发公平机器学习模型。其中大多数要求在培训期间提供敏感属性以学习公平的模型。然而,在许多现实世界应用中,由于隐私或法律问题,获得敏感的属性通常是不可行的,这挑战了现有的公平策略。虽然每个数据样本的敏感属性未知,但我们观察到训练数据中通常存在一些与敏感属性高度相关的非敏感功能,这可以用于缓解偏差。因此,在本文中,我们研究了一种探索与学习公平和准确分类器的敏感属性高度相关的特征的新问题。理论上我们通过最小化这些相关特征与模型预测之间的相关性,我们可以学习一个公平的分类器。基于这种动机,我们提出了一种新颖的框架,该框架同时使用这些相关的特征来准确预测和执行公平性。此外,该模型可以动态调整每个相关功能的正则化权重,以平衡其对模型分类和公平性的贡献。现实世界数据集的实验结果证明了拟议模型用于学习公平模型的效力,具有高分类准确性。
translated by 谷歌翻译
知识蒸馏(KD)将知识从高容量的教师网络转移到加强较小的学生。现有方法着重于发掘知识的提示,并将整个知识转移给学生。但是,由于知识在不同的学习阶段显示出对学生的价值观,因此出现了知识冗余。在本文中,我们提出了知识冷凝蒸馏(KCD)。具体而言,每个样本上的知识价值是动态估计的,基于期望最大化(EM)框架的迭代性凝结,从老师那里划定了一个紧凑的知识,以指导学生学习。我们的方法很容易建立在现成的KD方法之上,没有额外的培训参数和可忽略不计的计算开销。因此,它为KD提出了一种新的观点,在该观点中,积极地识别教师知识的学生可以学会更有效,有效地学习。对标准基准测试的实验表明,提出的KCD可以很好地提高学生模型的性能,甚至更高的蒸馏效率。代码可在https://github.com/dzy3/kcd上找到。
translated by 谷歌翻译
图形神经网络(GNN)在高级AI系统中被广泛采用,因为它们在图形数据上的表示能力。即使GNN的解释对于增加对系统的信任至关重要,但由于GNN执行的复杂性,它也是一项挑战。最近,已经提出了许多工作来解决GNN解释中的一些问题。但是,当图形的大小巨大时,它们缺乏概括能力或遭受计算负担。为了应对这些挑战,我们提出了一个多级GNN解释框架,基于观察到GNN是图形数据中多个组件的多模式学习过程。原始问题的复杂性是通过分解为表示为层次结构的多个子部分来放松的。顶级解释旨在指定每个组件对模型执行和预测的贡献,而细粒度的级别则集中于基于知识蒸馏的特征归因和图形结构归因分析。学生模型接受了独立模式的培训,并负责捕获不同的教师行为,后来用于特定的组成部分。此外,我们还旨在实现个性化的解释,因为该框架可以根据用户偏好产生不同的结果。最后,广泛的实验证明了我们提出的方法的有效性和保真度。
translated by 谷歌翻译
Algorithmic fairness is becoming increasingly important in data mining and machine learning. Among others, a foundational notation is group fairness. The vast majority of the existing works on group fairness, with a few exceptions, primarily focus on debiasing with respect to a single sensitive attribute, despite the fact that the co-existence of multiple sensitive attributes (e.g., gender, race, marital status, etc.) in the real-world is commonplace. As such, methods that can ensure a fair learning outcome with respect to all sensitive attributes of concern simultaneously need to be developed. In this paper, we study the problem of information-theoretic intersectional fairness (InfoFair), where statistical parity, a representative group fairness measure, is guaranteed among demographic groups formed by multiple sensitive attributes of interest. We formulate it as a mutual information minimization problem and propose a generic end-to-end algorithmic framework to solve it. The key idea is to leverage a variational representation of mutual information, which considers the variational distribution between learning outcomes and sensitive attributes, as well as the density ratio between the variational and the original distributions. Our proposed framework is generalizable to many different settings, including other statistical notions of fairness, and could handle any type of learning task equipped with a gradient-based optimizer. Empirical evaluations in the fair classification task on three real-world datasets demonstrate that our proposed framework can effectively debias the classification results with minimal impact to the classification accuracy.
translated by 谷歌翻译
尽管图神经网络(GNNS)已经证明了它们在处理非欧国人结构数据方面的功效,但由于多跳数据依赖性施加的可伸缩性约束,因此很难将它们部署在实际应用中。现有方法试图通过使用训练有素的GNN的标签训练多层感知器(MLP)来解决此可伸缩性问题。即使可以显着改善MLP的性能,但两个问题仍能阻止MLP的表现优于GNN并在实践中使用:图形结构信息的无知和对节点功能噪声的敏感性。在本文中,我们建议在图(NOSMOG)上学习噪声稳定结构感知的MLP,以克服挑战。具体而言,我们首先将节点内容与位置功能进行补充,以帮助MLP捕获图形结构信息。然后,我们设计了一种新颖的表示相似性蒸馏策略,以将结构节点相似性注入MLP。最后,我们介绍了对抗性功能的扩展,以确保稳定的学习能力噪声,并进一步提高性能。广泛的实验表明,在七个数据集中,NOSMOG在转导和归纳环境中均优于GNN和最先进的方法,同时保持竞争性推理效率。
translated by 谷歌翻译
众所周知,图形神经网络(GNN)的成功高度依赖于丰富的人类通知数据,这在实践中努力获得,并且并非总是可用的。当只有少数标记的节点可用时,如何开发高效的GNN仍在研究。尽管已证明自我训练对于半监督学习具有强大的功能,但其在图形结构数据上的应用可能会失败,因为(1)不利用较大的接收场来捕获远程节点相互作用,这加剧了传播功能的难度 - 标记节点到未标记节点的标签模式; (2)有限的标记数据使得在不同节点类别中学习良好的分离决策边界而不明确捕获基本的语义结构,这是一项挑战。为了解决捕获信息丰富的结构和语义知识的挑战,我们提出了一个新的图数据增强框架,AGST(增强图自训练),该框架由两个新的(即结构和语义)增强模块构建。 GST骨干。在这项工作中,我们研究了这个新颖的框架是否可以学习具有极有限标记节点的有效图预测模型。在有限标记节点数据的不同情况下,我们对半监督节点分类进行全面评估。实验结果证明了新的数据增强框架对节点分类的独特贡献,几乎没有标记的数据。
translated by 谷歌翻译