Heterogeneous Graph Tree Networks

Nan Wu Institute for Financial Services Analytics
University of Delaware
Newark, USA
nanw@udel.edu Chaofan Wang San Jose, USA
chaofanwang123@gmail.com

Abstract

Heterogeneous graph neural networks (HGNNs) have attracted increasing research interest in recent three years. Most existing HGNNs fall into two classes. One class is meta-path-based HGNNs which either require domain knowledge to handcraft meta-paths or consume huge amount of time and memory to automatically construct meta-paths. The other class does not rely on meta-path construction. It takes homogeneous convolutional graph neural networks (Conv-GNNs) as backbones and extend them to heterogeneous graphs by introducing node-type- and edge-type-dependent parameters. Regardless of the meta-path dependency, most existing HGNNs employ shallow Conv-GNNs such as GCN and GAT to aggregate neighborhood information, and may have limited capability to capture information from high-order neighborhood. In this work, we propose two heterogeneous graph tree network models: Heterogeneous Graph Tree Convolutional Network (HetGTCN) and Heterogeneous Graph Tree Attention Network (HetGTAN), which do not rely on meta-paths to encode heterogeneity in both node features and graph structure. Extensive experiments on three real-world heterogeneous graph data demonstrate that the proposed HetGTCN and HetGTAN are efficient and consistently outperform all state-of-the-art HGNN baselines on semi-supervised node classification tasks, and can go deep without compromising performance.

heterogeneous graphs, heterogeneous graph neural networks, graph tree networks, graph convolutional networks, graph attention networks

I Introduction

Although Graph Neural Network (GNN) has demonstrated its success in many graph learning applications such as link predictions [30, 31], fraud detection [23, 3], personalized recommendations [28, 4, 5], and pattern recognition [20, 19], most existing GNNs are only applicable to homogeneous graphs with single type of nodes and edges. As a matter of fact, a large portion of real-world graphs contain various types of nodes and/or edges, which are known as heterogeneous graphs. Attributes associated with different node types usually lie in different feature spaces. For example, a citation graph may contain three types of nodes: authors, papers and venues; and four types of edges: author-paper, paper-venue and their reverse connections. The complex structure and rich semantic information make it nontrivial to extend applications of a homogeneous GNN to heterogeneous graphs.

Heterogeneous graph neural networks (HGNNs) have attracted increasing research interest in recent three years, yet are still under explored. Most existing HGNNs fall into two classes. One class is meta-path-based HGNNs which either require domain knowledge to handcraft meta-paths [26, 7] or consume huge amount of time and memory to automatically construct meta-paths [29]. The other class does not rely on meta-path construction [18, 9, 15]. This class of models takes homogeneous convolutional graph neural networks (Conv-GNNs) as backbones and extend them to heterogeneous graphs by introducing node-type- and edge-type-dependent parameters. Regardless of the meta-path dependency, most existing HGNNs employ shallow Conv-GNNs such as GCN [10] and GAT [21] to aggregate neighborhood information, and therefore may have limited capability to capture information from high-order neighborhood.

[27] recently proposes two deep convolutional graph tree networks which adopt a different message passing scheme from GCN and GAT, and demonstrates the deep capability of the proposed models both theoretically and experimentally. In light of the superior performance of the two graph tree network models proposed by [27], we propose two heterogeneous graph tree network models: Heterogeneous Graph Tree Convolutional Network (HetGTCN) and Heterogeneous Graph Tree Attention Network (HetGTAN). Both models include three modules: (1) a node-type-specific transformation to project node features into the same vector space; (2) an edge-type-specific graph tree convolutional network (GTCN) [27] or graph tree attention network (GTAN) [27] layer to aggregate one-hop neighbors connected via the same type of edges; (3) a target-specific aggregator to combine representations of the target node from different edge types. Our proposed models do not depend on meta-paths to encode heterogeneity in both node features and graph structure, and can go deep to incorporate information from high-order neighborhood without compromising performance.

When evaluating our models, we find that all popular HGNNs underestimate the performance of the two most popular baseline GNNs - GCN and GAT by simply ignoring the heterogeneity of nodes and edges [29, 9, 15] or testing on meta-path based homogeneous subgraphs and reporting the best performance [26, 7]. To better benchmark HGNNs, we propose HetGCN and HetGAT to extend the vanilla GCN and GAT to heterogeneous graphs. With the same architecture of HetGTCN and HetGTAN, HetGCN and HetGAT also include three modules: (1) a node-type-specific transformation to project node features into the same vector space; (2) an edge-type-specific GCN or GAT layer to aggregate one-hop neighbors connected via the same type of edges; (3) a target-specific aggregator to combine representations of the target node from different edge types.

We conduct comprehensive experiments on three real-world heterogeneous graph datasets on semi-supervised node classification tasks, and make fair comparisons of the proposed HetGTCN and HetGTAN with nine baseline HGNN models including HetGCN and HetGAT. We find that HetGAT already outperforms or matches the other baseline HGNNs. Our proposed HetGTCN and HetGTAN consistently and significantly outperform all baseline models. We also demonstrate the efficiency of our proposed HetGTCN and HetGTAN by comparing the runtime and memory consumption with the other baselines. Moreover, our proposed HetGTCN and HetGTAN models can go deep without compromising performance, while the performance of most baseline HGNNs degrades significantly with an increasing number of model layers.

The major contributions of this work are summarized as follows:

We propose two efficient and effective heterogeneous graph neural network models: HetGTCN and HetGTAN which can encode heterogeneity in both node features and graph structure without designing meta-paths. Three different aggregator functions are explored in this work: semantic attention aggregator, simple mean aggregator, and weighted sum aggregator with learnable weights.
We conduct extensive experiments and demonstrate that the proposed HetGTCN and HetGTAN outperform all baseline HGNNs, and can go deep without compromising performance.
We propose two baseline HGNNs - HetGCN and HetGAT which extend the two most popular homogeneous GNNs - GCN and GAT to heterogeneous graphs. We experimentally demonstrate that HetGAT outperforms or matches the other baseline HGNNs.
We provide unified pre-processing on three real-world heterogeneous graph datasets to promote fair comparisons between the proposed models and baselines.

Ii Related Work

Convolutional Graph Neural Networks (Conv-GNNs). The class of spatial-based Conv-GNNs has attracted considerable attention over the past few years following the publication of GCN [10]. These models adopt a message passing scheme to aggregate neighborhood information, among which the vanilla GCN [10] and GAT [21] are arguably the two most popular baseline GNNs. GCN uses a normalized mean aggregator to aggregate neighborhood information, while GAT uses an additive attention [1] aggregator to aggregate neighborhood information. One common issue with GCN and GAT is the over-smoothing problem [11, 14, 16, 2, 24] that the performance of GCN and GAT degrades significantly with an increasing number of model layers. Therefore, both GCN and GAT are often referred to as shallow models.

Heterogeneous Graph Neural Networks (HGNNs). Witnessing the success of homogeneous GNNs in different domains on graph-structured data, many efforts have been devoted to developing GNNs for heterogeneous graphs in recent three years. Most existing HGNNs take a homogeneous Conv-GNN such as GCN [10] and GAT [21] as their backbone, and encode the graph heterogeneity by either meta-paths or node-type- and edge-type-dependent parameters. They are essentially a combination of meta-path-specific or edge-type-specific Conv-GNNs, and can be described by a general heterogeneous message passing neural network (MPNN) formulated in Equation 1 which is an extension of [8].

\begin{matrix} m_{u}^{l + 1} = K_{a} \sum k = 1 θ_{k, a}^{l} \sum v \in N_{u}^{k} M_{k}^{l} (h_{u}^{l}, h_{v}^{l}, h_{k (u, v)}^{l}) h_{u}^{l + 1} = U_{l} (h_{u}^{l}, m_{u}^{l + 1}) \end{matrix}

(1)

where $a$ is the node type of the target node $u$ , and $h_{u}$ is the hidden feature of node $u$ . $k$ is the relation type, which can be referred to as either meta-path type or edge type. $K_{a}$ is the number of relation types connected to type $a$ nodes, $N_{u}^{k}$ is the set of relation-type-specific neighbors of node $u$ , and $h_{k (u, v)}$ is the hidden feature of meta-path instance $k (u, v)$ or edge $(u, v)$ . $M_{k} (\cdot)$ is a relation-type-specific message passing function, and $U (\cdot)$ is a node feature update function [8]. $θ_{k, a}$ denotes the importance of different relation types for target nodes of type $a$ , which can be either fixed or learnable. $l$ denotes the message passing layer number.

Representative meta-path-based HGNNs include HAN [26], MAGNN [7], and GTN [29]. Among which both HAN and MAGNN can fit into the general MPNN framework described in Equation 1. HAN [26] is one of the earliest attempts on HGNNs with $M_{k}^{l} (\cdot) = {GAT}_{k}^{l} (h_{u}^{l}, h_{v}^{l})$ , $U_{l} (\cdot) = m_{u}^{l + 1}$ , and $θ_{k, a}^{l}$ is a learnable semantic attention weight. It takes GAT as its backbone to aggregate meta-path-based neighbors, and a semantic attention aggregator to combine representations of different meta-paths. HAN ignores all intermediate nodes along a meta-path instance. MAGNN [7] improves over HAN by introducing a meta-path instance encoder to include all nodes along a meta-path instance, where $M_{k}^{l} (\cdot) = {GAT}_{k}^{l} (h_{u}^{l}, h_{k (u, v)}^{l})$ . Both HAN and MAGNN rely on domain knowledge to handcraft meta-paths for different downstream tasks, and therefore may suffer from information loss. GTN [29] learns a new graph structure defined by a weighted sum of all meta-path-based adjacency matrices, and then applies GCN to the new graph structure. Although GTN is able to extract valuable meta-paths automatically, it requires huge amount of time and memory to learn the new graph which is constructed by multiplication of soft-selected adjacency matrices of each edge type. One common issue faced by meta-path-based HGNNs is that they are not scalable when high-order meta-paths are desired to capture information from high-order neighborhood because the number of high-order meta-path instances may increase dramatically. Take the DBLP dataset [6] as an example, the number of edges with type ”Author-Paper” (AP) and ”Paper-Conference” (PC) is 19,645 and 14,328 respectively, while the number of meta-path ”APCPA” is 5,000,495.

Representative meta-path-free HGNNs include RGCN [18], HGT [9], and SimpleHGN [15], which all can fit into the general MPNN framework described in Equation 1. RGCN [18] is proposed to learn from multi-relational knowledge graphs with a single node type. It takes GCN as its backbone, and can be regarded as a weighted sum of edge-type-specific GCNs, where $M_{k}^{l} (\cdot) = {GCN}_{k}^{l} (h_{u}^{l}, h_{v}^{l})$ , and $U_{l} (\cdot) = m_{u}^{l + 1}$ . HGT [9] is a transformer-based HGNN, which uses transformer attention as $M_{k}$ , and $θ_{k, a} = 1$ . Skip connection is applied with $U_{l} (\cdot) = σ (m_{u}^{l + 1}) W_{a} + h_{u}^{l}$ . It extends [12] to heterogeneous graphs by introducing node-type-dependent transformation matrices for query, key, and value projection, respectively, and edge-type-dependent weight matrices for transformer attention and message transformation, respectively. SimpleHGN [15] extends GAT to heterogeneous graphs by introducing a learnable edge type embedding and a corresponding edge-type-dependent transformation matrix in the node pair attention. It uses GAT as $M_{k}$ and $θ_{k, a} = 1$ . It also applies the skip connection with $U_{l} (\cdot) = m_{u}^{l + 1} + h_{u}^{l}$ .

DMGI [17] is a representative unsupervised HGNN based on Deep Graph Infomax (DGI) [22]. It is trained to maximize the mutual information between the node embedding and the graph embedding regarding each relation type, and minimize a consensus regularizer, where the relation-type-specific node embeddings are generated by a relation-type-specific GCN layer. The relation can either refer to an edge or a meta-path instance.

Although the aforementioned HGNNs have been successfully applied to many applications on heterogeneous graphs, none of them has discussed about their deep capability. We demonstrate in Section V-D that all these HGNNs have limited capability to capture information from high-order neighborhood and show performance decay with an increasing number of model layers.

Iii Preliminary

In this section, we introduce the definition of heterogeneous graphs and notations used throughout this paper. We then review the two homogeneous graph tree network models proposed in [27].

Iii-a Heterogeneous Graph

A heterogeneous graph is defined as $G = (V, E, T^{v}, T^{e})$ with a node type mapping function $ϕ$ : $V \to T^{v}$ and an edge type mapping function $ψ$ : $E \to T^{e}$ , where $V$ is the set of nodes, $E$ is the set of edges, $T^{v}$ is the set of node types, and $T^{e}$ is the set of edge types. $| T^{v} |$ and $| T^{e} |$ denote the number of node types and edge types respectively, where $| T^{v} | + | T^{e} | > 2$ . A heterogeneous graph can be fully described by a set of adjacency matrices ${A_{k}}_{k = 1}^{| T^{e} |}$ and the corresponding set of degree matrices ${D_{k}}_{k = 1}^{| T^{e} |}$ , where $A_{k}$ is the adjacency matrix of edge type $k$ . Let $V_{a}$ denote the set of nodes of type $a$ . Each node $u \in V_{a}$ is associated with an input feature vector $x_{u} \in R^{1 \times d_{a}}$ . The input feature map for all nodes of type $a$ is denoted as $X_{a} \in R^{| V_{a} | \times d_{a}}$ . Let $h_{u}^{l} \in R^{1 \times f}$ denote the hidden feature of node $u$ at the $l^{th}$ layer, and $N_{u}^{k}$ denote the set of one-hop neighbors of node $u$ connected via type $k$ edges.

Iii-B Graph Tree Networks

Trees can intuitively represent the complex structure of a graph, where each target node and its neighborhood form a tree with the target node being the root node and its neighbors being the subnodes. The depth of the graph tree is determined by the size of desired receptive field.

[27] assumes that each node preserves its initial information prior to receiving new information from its child nodes in the graph tree, and proposes Graph Tree Networks (GTNets) with a message passing scheme following this assumption. In GTNet, the representation of a target node is updated by aggregating its neighbors’ updated hidden features from the previous layer and its own initial feature. In contrast, GCN and GAT update a target node’s representation by aggregating its neighbors’ and its own updated hidden features from the previous layer. Following the architecture of GTNet, [27] proposes two homogeneous graph tree network models - Graph Tree Convolutional Network (GTCN) and Graph Tree Attention Network (GTAN).

The message passing rule in one GTCN layer is described as [27]

h_{u}^{l} = \sum_{v \in N_{u}} {^A}_{u v} h_{v}^{l + 1} + {^A}_{u u} z_{u}

(2)

where $^A = {~ D}^{- \frac{1}{2}} ~ A {~ D}^{- \frac{}{1} 2}$ is the symmetrically normalized adjacency matrix. For a directed graph, the adjacency matrix $A$ is asymmetric, $~ A$ can be normalized by its inverse degree matrix ${~ D}^{- 1}$ , i.e. $^A = {~ D}^{- 1} ~ A$ . $~ A = (A + I)$ is the adjacency matrix with added self-loops. $~ D$ is the degree matrix of $~ A$ , where ${~ D}_{u u} = \sum_{v} {~ A}_{u v}$ . $l$ denotes the order (i.e., hop) of neighborhood to the target node, which is in an reverse order of the model layer number. $z_{u}$ is the initial hidden feature of node $u$ such that $h_{u}^{L} = z_{u} = MLP (x_{u})$ , where $L$ is the model depth.

The message passing rule in one GTAN layer is described as [27]

h_{u}^{l} = ELU (\sum_{v \in N_{u}} α_{u, v}^{l + 1} h_{v}^{l + 1} + α_{u, u}^{l + 1} z_{u})

(3)

where $α_{u, v}$ is the attention weight for the node pair $(u, v)$ calculated by the additive attention [1] as

α_{u, v}^{l} = {\begin{matrix} softmax (LeakyReLU ([z_{u} ∥ h_{v}^{l}] a^{l})) & if u \neq v softmax (LeakyReLU ([z_{u} ∥ z_{u}] a^{l})) & if u = v \end{matrix}

(4)

where $a^{l}$ is a learnable attention vector shared by all edges.

Built on GTCN and GTAN, we propose two heterogeneous graph tree network models - Heterogeneous Graph Tree Convolutional Network (HetGTCN) and Heterogeneous Graph Tree Attention Network (HetGTAN) in Section IV.

Iv Heterogeneous Graph Tree Networks

Fig. 1: Tree representation of a target node $u$ and its two-hop neighborhood in a sample heterogeneous graph.

Fig. 2: The overall architecture of HetGTCN and HetGTAN. Each model is consisted of three modules: (1) a node-type-specific transformation to project node features into the same vector space; (2) an edge-type-specific GTCN or GTAN layer to aggregate neighbors with the same type of relations; (3) a target-specific aggregator to combine representations from different edge types. For simplicity and clarity, we show the pipeline for a sub-graph with node $u$ and its one-hop neighbors. Node types, edge types and their dependency are denoted with different colors.

Figure 1 shows an example of a heterogeneous graph and its tree representation of node $u$ and its two-hop neighborhood. There are three types of nodes, and four types of edges in the sample graph. We use different colors to distinguish the node types and edge types. For a heterogeneous graph, each path from a subnode to the root node in its tree representation represents a meta-path instance. Messages propagate upward along each meta-path instance to pass information from subnodes to the target node. A parent node may have multiple types of child nodes connected via different types of edge. Nodes of different types may be associated with features lying in different feature spaces. An effective heterogeneous graph tree model should be able to encode such heterogeneity in both node contents and graph structure.

In this section, we propose two heterogeneous graph tree network models: Heterogeneous Graph Tree Convolutional Network (HetGTCN) and Heterogeneous Graph Tree Attention Network (HetGTAN). Both models are consisted of three modules: (1) a node-type-specific transformation to project node features into the same vector space; (2) an edge-type-specific GTCN [27] or GTAN [27] layer to aggregate one-hop neighbors connected via the same type of edges; (3) a target-specific aggregator to combine representations of the target node from different edge types. Three different aggregator functions are explored in Section V-E which are semantic attention, simple mean, and weighted sum with learnable weights. Figure 2 shows the overall architecture of the proposed HetGTCN and HetGTAN with a semantic attention aggregator. We use different colors to denote types of nodes and edges, and the type dependency of each component. This architecture can turn any homogeneous GNN into a heterogeneous GNN by replacing the second module with the particular edge-type-specific homogeneous GNN.

We introduce each module of HetGTCN and HetGTAN step by step in the following sections.

Iv-a Heterogeneous Graph Tree Convolutional Network (HetGTCN)

We assume the order of neighborhood under consideration is $L$ , i.e., the model depth is $L$ .

Node Feature Transformation. In heterogeneous graphs, attributes associated with different node types usually lie in different feature spaces. To aggregate information from different types of nodes, we need to project node features into the same vector space. We first apply a node-type-specific MLP to transform all node features into the same vector space. For a node $u \in V_{a}$ , i.e. $ϕ (u) = a$ , we have

z_{u} = σ (x_{u} W_{0, a} + b_{0, a})

(5)

and

h_{u}^{L} = z_{u}

(6)

where $x_{u} \in R^{1 \times d_{a}}$ is the input feature vector of node $u$ , $z_{u} \in R^{1 \times f}$ is the projected input feature vector of node $u$ , which is also the initial hidden feature $h_{u}^{L}$ for the first message passing layer. $W_{0, a} \in R^{d_{a} \times f}$ is the learnable weight matrix for node type $a$ , and $b_{0, a} \in R^{1 \times f}$ is the learnable bias vector for node type $a$ .

Edge-type-specific Neighbor Aggregation. After all node features are transformed into the same vector space, we apply an edge-type-specific GTCN layer to aggregate information from neighbors connected to the target node via the same type of edges.

h_{u, k}^{l} = \sum_{v \in N_{u}^{k}} {^A}_{k, u v} h_{v}^{l + 1} + {^A}_{k, u u} z_{u}

(7)

where ${^A}_{k} = {~ D}_{k}^{- 1} {~ A}_{k}$ , which is the normalized adjacency matrix of edge type $k$ .

Assume there are $K_{a}$ different types of edges connected to the node type $a$ , the edge-type-specific neighbor aggregation generates a set of $K_{a}$ edge-type-specific representations for a target node $u \in V_{a}$ , denoted as ${h_{u, 1}^{l}, h_{u, 2}^{l}, \dots, h_{u, K_{a}}^{l}}$ .

Target-Specific Aggregation. Various aggregators can be employed to combine the edge-type-specific representations for a target node. Here, we leverage the semantic attention mechanism [13] to learn the importance of each edge type to a target node, and aggregate the edge-type-specific representations to obtain the final representation of the target node. Another two aggregators are investigated in Section V-E.

For $u \in V_{a}$ , we have:

h_{u}^{l} = K_{a} \sum k = 1 β_{k}^{l} \cdot h_{u, k}^{l}

(8)

where $β_{k}^{l}$ denotes the importance of edge type $k$ , and is calculated by the following attention mechanism.

\begin{matrix} w_{k}^{l} = \frac{1}{| V_{a} |} \sum_{u \in V_{a}} {(q_{a}^{l})}^{T} \cdot tanh (W_{a}^{l} {(h_{u, k}^{l})}^{T} + b_{a}^{l}) β_{k}^{l} = softmax (w_{k}^{l}) \end{matrix}

(9)

where $l = L - 1, L - 2, \dots, 0$ , denoting the order of neighborhood to the target node. $q_{a}^{l}$ is a learnable node-type-specific semantic attention vector. $W_{a}^{l}$ and $b_{a}^{l}$ are node-type-specific learnable transformation matrix and bias vector, respectively.

For a node classification task, we apply a node-type-specific MLP layer to the final hidden layer as in Equation 10.

y_{u}^{out} = σ (h_{u}^{0} W_{out, a} + b_{out, a})

(10)

Iv-B Heterogeneous Graph Tree Attention Network (HetGTAN)

The architecture of HetGTAN is the same as HetGTCN except that an edge-type-specific GTAN layer is applied to aggregate information of neighbors in the second module. Following the description in Section IV-A, the proposed HetGTAN is formulated as below:

\begin{matrix} h_{u}^{L} = z_{u} = σ (x_{u} W_{0, a} + b_{0, a}) h_{u, k}^{l} = ELU (\sum_{v \in N_{u}^{k}} α_{u, v, k}^{l + 1} h_{v}^{l + 1} + α_{u, u, k}^{l + 1} z_{u}) h_{u}^{l} = K_{a} \sum k = 1 β_{k}^{l} \cdot h_{u, k}^{l} \end{matrix}

(11)

where $l = L - 1, L - 2, \dots, 0$ , denoting the order of neighborhood to the target node. $α_{u, v, k}^{l}$ is the node-level attention weight for node pair $(u, v)$ through type $k$ connection, and $β_{k}$ is the attention weight for edge type $k$ , which are calculated as below:

\begin{matrix} α_{u, v, k}^{l} = ⎧ ⎨ ⎩ \begin{matrix} % softmax (LeakyReLU ([z_{u} ∥ h_{v}^{l}] a_{k}^{l})) & if u \neq v softmax (LeakyReLU ([z_{u} ∥ z_{u}] a_{k}^{l})) & if u = v \end{matrix} w_{k}^{l} = \frac{1}{| V_{a} |} \sum_{u \in V_{a}} {(q_{a}^{l})}^{T} \cdot tanh (W_{a}^{l} {(h_{u, k}^{l})}^{T} + b_{a}^{l}) β_{k}^{l} = softmax (w_{k}^{l}) \end{matrix}

(12)

where $a_{k}^{l}$ is an edge-type-specific learnable attention vector.

V Experiments

In this section, we first evaluate the performance and efficiency of the proposed HetGTCN and HetGTAN on semi-supervised node classification tasks with three popular benchmark heterogeneous graphs in Section V-C. We then demonstrate the deep capability of HetGTCN and HetGTAN in Section V-D. Last, we compare the model performance with different aggregators in Section V-E.

V-a Datasets

We use two citation graphs ACM and DBLP, and a movie graph IMDB. The statistics of the three benchmark datasets are summarized in Table I. We use balanced label splits for both training and validation samples. The detailed descriptions are in Appendix A.

Dataset	#Node	#Edge	#Meta-Path	#Node Feature	Target Node	#Class	#Training	#Validation	#Test
ACM	Paper (P): 4,019	P-A: 13,407	PAP: 57,853	P: 1,902	Paper	3	600	300	3,119
	Author (A): 7,167	P-S: 4,019	PSP: 4,338,213	A: 1,902
	Subject (S): 60			S: 1,902
IMDB	Movie (M): 4,278	M-D: 4,278	MDM: 17,446	M: 3,066	Movie	3	300	300	3,678
	Director (D): 2,081	M-A: 12,828	MAM: 85,358	D: 3,066
	Actor (A): 5,257			A: 3,066
DBLP	Author (A): 4,057	A-P: 19,645	APA: 11,113	A: 334	Author	4	800	400	2,857
	Paper (P): 14,328	P-T: 85,810	APCPA: 5,000,495	P: 334
	Term (T): 7,723	P-C: 14,328	APTPA: 7,043,571	T: 50
	Conference (C): 20			C: 334

TABLE I: Statistics of benchmark datasets.

V-B Baselines

We compare the proposed HetGTCN and HetGTAN to nine HGNN baseline models including RGCN [18], HAN [26], MAGNN [26], GTN [29], HGT [9], SimpleHGN [15], DMGI [17], HetGCN and HetGAT. We also include the vanilla GCN [10] and GAT [21] as baselines by ignoring the node and edge types. We have reviewed all baselines except HetGCN and HetGAT in Section II, and will not repeat the descriptions in this section.

In this section, we briefly introduce the two proposed baselines - HetGCN and HetGAT.

We apply the same architecture described in Section IV to turn the vanilla GCN [10] and GAT [21] into HGNNs, and propose HetGCN and HetGAT accordingly in this section. Both models are consisted of three modules: (1) a node-type-specific transformation to project node features into the same vector space; (2) an edge-type-specific GCN or GAT layer to aggregate one-hop neighbors connected via the same type of edges; (3) a target-specific aggregator to combine representations of the target node from different edge types. We adopt the semantic attention aggregator in the third module for both HetGCN and HetGAT in this section. The detailed mathematical formulations of HetGCN and HetGAT are described in Appendix B.

V-C Results on Node Classification

The detailed experimental setups are described in Appendix C. The experimental results of the proposed HetGTCN and HetGTAN and all baselines on the node classification task with all three benchmark datasets are summarized in Table II. The evaluation metrics are Macro-F1 and Micro-F1. The results are obtained after 30 runs of each model. Mean and standard deviations are calculated after excluding the top and bottom $10 %$ data points.

Dataset	ACM		IMDB		DBLP
Metrics (%)	Macro-F1	Micro-F1	Macro-F1	Micro-F1	Macro-F1	Micro-F1
RGCN	91.6 $\pm$ 0.5	91.7 $\pm$ 0.4	58.0 $\pm$ 0.8	58.1 $\pm$ 0.6	92.8 $\pm$ 0.6	93.4 $\pm$ 0.5
HAN	91.2 $\pm$ 0.2	91.0 $\pm$ 0.2	57.4 $\pm$ 0.4	57.2 $\pm$ 0.6	92.4 $\pm$ 0.2	93.2 $\pm$ 0.2
MAGNN	90.7 $\pm$ 0.5	90.8 $\pm$ 0.5	58.7 $\pm$ 0.5	58.7 $\pm$ 0.7	93.0 $\pm$ 0.3	93.6 $\pm$ 0.2
GTN	90.7 $\pm$ 0.3	90.1 $\pm$ 1.1	55.7 $\pm$ 0.8	55.4 $\pm$ 0.9	93.1 $\pm$ 0.4	94.1 $\pm$ 0.3
HGT	87.1 $\pm$ 1.0	87.3 $\pm$ 1.4	56.3 $\pm$ 0.6	55.8 $\pm$ 0.9	91.6 $\pm$ 0.8	92.2 $\pm$ 0.6
SimpleHGN	91.4 $\pm$ 0.4	91.5 $\pm$ 0.3	56.3 $\pm$ 0.5	56.1 $\pm$ 0.6	93.6 $\pm$ 0.2	94.3 $\pm$ 0.2
DMGI	90.0 $\pm$ 0.6	90.5 $\pm$ 0.2	56.9 $\pm$ 1.0	56.3 $\pm$ 1.1	91.9 $\pm$ 0.3	92.9 $\pm$ 0.2
GCN	90.7 $\pm$ 0.7	90.4 $\pm$ 1.0	57.1 $\pm$ 0.7	57.2 $\pm$ 0.9	83.5 $\pm$ 0.2	84.9 $\pm$ 0.3
GAT	91.4 $\pm$ 0.6	91.2 $\pm$ 0.7	57.0 $\pm$ 1.1	56.3 $\pm$ 1.1	89.8 $\pm$ 0.8	90.9 $\pm$ 0.8
HetGCN	91.1 $\pm$ 0.5	90.7 $\pm$ 0.9	57.8 $\pm$ 0.8	57.4 $\pm$ 0.8	92.1 $\pm$ 0.7	93.2 $\pm$ 0.5
HetGAT	91.7 $\pm$ 0.4	91.7 $\pm$ 0.6	58.4 $\pm$ 0.9	57.8 $\pm$ 1.0	92.8 $\pm$ 0.5	93.6 $\pm$ 0.4
HetGTCN	92.3 $\pm$ 0.2	92.1 $\pm$ 0.6	60.5 $\pm$ 0.5	60.0 $\pm$ 0.5	94.2 $\pm$ 0.2	94.8 $\pm$ 0.2
HetGTAN	92.3 $\pm$ 0.3	92.2 $\pm$ 0.4	60.8 $\pm$ 0.7	61.0 $\pm$ 0.5	94.4 $\pm$ 0.2	95.2 $\pm$ 0.2

TABLE II: F1 score on node classification task, averaged by 30 runs of each model.

It is shown that the proposed HetGTCN and HetGTAN consistently outperform all baselines on all three datasets, with 1-3 $%$ improvements over the second best model. HetGTAN generally yields better performance than HetGTCN, demonstrating that attention between each pair of nodes may better capture the heterogeneity in nodes and edges. This is also observed from the comparison of HetGCN and HetGAT. It is seen from Table II that HetGAT outperforms HAN on all three datasets. It is not surprising because HAN manually selects meta-paths, which may lose information from other unselected meta-paths. For instance, in ACM dataset, a 2-hop HetGAT includes all 2-hop meta-paths: PAP, PSP, SPA, SPS, APA, APS, while HAN only selects two specific meta-paths: PAP and PSP. In addition, HAN ignores the intermediate nodes along the meta-paths, while HetGAT includes all intermediate nodes in the aggregation. In fact, HetGAT outperforms or matches most of the baselines, providing a new candidate baseline model for the future study.

We demonstrate the efficiency of the proposed HetGTCN and HetGTAN by comparing the time and memory consumption with all baselines. HAN, MAGNN and DMGI use 2-hop meta-paths for ACM and DBLP. For fair comparison, we use two graph layers for all other models including RGCN, GTN, HGT, SimpleHGN, HetGCN, HetGAT, HetGTCN, and HetGTAN, which is comparable to a 2-hop meta-path-based HGNN. The results are summarized in Table III.

It is seen from Table III that RGCN, HetGCN, HetGAT, HetGTCN, and HetGTAN consume comparable runtime, and are the most efficient models. All models consume similar memories, except the meta-path-based HGNNs including HAN, MAGNN, and GTN which consume significantly more memories on ACM and DBLP. The runtime and memory consumption of the meta-path-based HGNNs depend on the number of meta-path instances. Because ACM and DBLP have much more meta-path instances than IMDB, all three meta-path-based baselines consume significantly more runtime and memories than our proposed models on ACM and DBLP. The runtime of HAN is $\sim$ 2.7 $\times$ that of HetGTCN on ACM, and $\sim$ 5 $\times$ that of HetGTCN on DBLP. The runtime of GTN is $\sim$ 4.5 $\times$ that of HetGTCN on ACM, and $\sim$ 8.4 $\times$ that of HetGTCN on DBLP. MAGNN consumes the most runtime even we don’t take the unscalable pre-processing time into consideration. The runtime of MAGNN is $\sim$ 44 $\times$ that of HetGTCN on ACM, $\sim$ 11 $\times$ that of HetGTCN on IMDB, and $\sim$ 717 $\times$ that of HetGTCN on DBLP. HGT and SimpleHGN consume more runtime than HetGAT which may be due to the additional edge-type-specific learnable weight matrix and edge embeddings.

Model	ACM		IMDB		DBLP
	time (ms/epoch)	RAM (GB)	time (ms/epoch)	RAM (GB)	time (ms/epoch)	RAM (GB)
MAGNN	677.2	10.6	164.9	2.3	15,628.0^*	1.6^*
DMGI	143.5	1.5	111.5	1.4	285.3	1.8
SimpleHGN	72.9	1.2	72.2	1.2	113.0	1.1
GTN	70.3	3.1	23.6	1.3	183.3	8.5
HGT	44.6	1.2	43.7	1.2	96.7	1.2
HAN	42.0	2.1	11.0	1.2	109.5	3.6
RGCN	25.6	1.1	26.2	1.2	33.2	1.2
HetGAT	24.8	1.2	21.8	1.3	31.1	1.2
HetGTAN	18.5	1.1	17.0	1.2	27.4	1.2
HetGCN	15.8	1.2	15.5	1.2	22.0	1.2
HetGTCN	15.5	1.1	15.4	1.2	21.8	1.2

MAGNN runs on minibatch of DBLP with batch size of 8 samples.

TABLE III: Time and memory consumption in ms/epoch and GB.

V-D Deep Capability

It is sometimes desirable to increase the receptive field of an HGNN model so that it can capture information from high-order neighborhood. However, meta-path-based models suffer from scalability issue when going deep, due to the dramatically increasing number of meta-path instances. Regardless of the meta-path dependency, many HGNNs depend on GCN or GAT to aggregate information from edge-type-specific neighbors, therefore may suffer from the over-smoothing problem as GCN and GAT. The proposed HetGTCN and HetGTAN depend on GTCN and GTAN to aggregate information from edge-type-specific neighbors. The deep capability of GTCN and GTAN has been demonstrated in [27].

In this section, we compare the performance of the proposed HetGTCN and HetGTAN with six baseline HGNNs including RGCN, HAN, HGT, SimpleHGN, HetGCN, and HetGAT with different number of message passing layers (2, 5, 10, and 20). We do not include MAGNN, GTN, and DMGI in this comparison due to the runtime and memory issue with these models. The results are displayed in Figure 3.

Results. It is seen from Figure 3 that the performance of the proposed HetGTCN and HetGTAN does not compromise when going deep while the performance of all other baselines degrades significantly with an increasing model depth. Among all baselines, the performance of HGT degrades much slower than other baselines, which may be due to the skip connection.

V-E Ablation Study

We adopt the semantic attention aggregator in HetGTCN and HetGTAN to combine edge-type-specific representations in previous sections. In this section, we conduct an ablation study to explore different aggregators in HetGTCN and HetGTAN. HetGTCN_mean and HetGTAN_mean adopt a simple mean aggregator in the third module to combine edge-type-specific representations, where $h_{u}^{l} = \frac{1}{K_{a}} \sum_{k = 1}^{K_{a}} h_{u, k}^{l}$ . HetGTCN_LW and HetGTAN_LW adopt a weighted sum aggregator in the third module, where $h_{u}^{l} = \sum_{k = 1}^{K_{a}} θ_{k}^{l} h_{u, k}^{l}$ , and the weight $θ_{k}^{l}$ for each edge type is directly learned. For HetGTAN, we have an additional variant HetGTAN_ns which removes the semantic attention layer, and the message passing rule in the second module becomes

h_{u}^{l} = ELU (\sum_{k = 1}^{K_{a}} (\sum_{v \in N_{u}^{k}} α_{u, v, k}^{l + 1} h_{v}^{l + 1} + α_{u, u, k}^{l + 1} z_{u}))

(13)

Other settings of these variants stay the same as the original models. The average Macro-F1 score on the node classification task with all three datasets are summarized in Table IV. The simple mean and semantic attention aggregator yield comparable performance for HetGTCN on ACM and IMDB, while HetGTCN with the semantic attention aggregator works best on DBLP. The simple mean, weighted sum, and semantic attention aggregator all yield comparable performance for HetGTAN on IMDB and DBLP, while HetGTAN with the semantic attention aggregator works best on ACM. Overall, the semantic attention aggregator works best for both HetGTCN and HetGTAN on all three datasets.

Dataset	ACM	IMDB	DBLP
Metrics (%)	Macro-F1	Macro-F1	Macro-F1
HetGTCN_mean	92.3 $\pm$ 0.2	60.5 $\pm$ 0.7	91.2 $\pm$ 0.7
HetGTCN_LW	91.7 $\pm$ 1.3	60.1 $\pm$ 0.7	93.8 $\pm$ 0.7
HetGTCN	92.3 $\pm$ 0.2	60.5 $\pm$ 0.5	94.2 $\pm$ 0.2
HetGTAN_ns	91.8 $\pm$ 0.3	60.4 $\pm$ 0.5	94.1 $\pm$ 0.3
HetGTAN_mean	91.9 $\pm$ 0.3	60.9 $\pm$ 0.7	94.4 $\pm$ 0.2
HetGTAN_LW	91.9 $\pm$ 0.5	60.9 $\pm$ 0.5	94.4 $\pm$ 0.2
HetGTAN	92.3 $\pm$ 0.3	60.8 $\pm$ 0.7	94.4 $\pm$ 0.2

TABLE IV: Ablation study for HetGTCN and HetGTAN, using 5 layers for all models.

Vi Conclusion

In this paper, we propose two heterogeneous graph neural network models - HetGTCN and HetGTAN which are based on Graph Tree Networks. Both models are meta-path-free, and are able to go deep without compromising performance. We conduct comprehensive experiments on three real-world heterogeneous graph data with unified pre-processing and demonstrate the effectiveness and efficiency of our proposed models. We also propose two baseline HGNN models - HetGCN and HetGAT to extend the vanilla GCN and GAT models to heterogeneous graphs. Future work could include exploring potentials of the proposed HetGTCN and HetGTAN on various interesting heterogeneous graph datasets for different downstream tasks.

References

[1] D. Bahdanau, K. Cho, and Y. Bengio (2014) Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. Cited by: §II, §III-B.
[2] D. Chen, Y. Lin, W. Li, P. Li, J. Zhou, and X. Sun (2020) Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, pp. 3438–3445. Cited by: §II.
[3] Y. Dou, Z. Liu, L. Sun, Y. Deng, H. Peng, and P. S. Yu (2020) Enhancing graph neural network-based fraud detectors against camouflaged fraudsters. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM ’20, New York, NY, USA, pp. 315–324. External Links: ISBN 9781450368599, Link, Document Cited by: §I.
[4] S. Fan, J. Zhu, X. Han, C. Shi, L. Hu, B. Ma, and Y. Li (2019) Metapath-guided heterogeneous graph neural network for intent recommendation. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, New York, NY, USA, pp. 2478–2486. External Links: ISBN 9781450362016, Link, Document Cited by: §I.
[5] W. Fan, Y. Ma, Q. Li, Y. He, E. Zhao, J. Tang, and D. Yin (2019) Graph neural networks for social recommendation. In The World Wide Web Conference, WWW ’19, New York, NY, USA, pp. 417–426. External Links: ISBN 9781450366748, Link, Document Cited by: §I.
[6] M. Fey and J. E. Lenssen (2019) Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, Cited by: Appendix A, Appendix A, Appendix C, Appendix C, §II.
[7] X. Fu, J. Zhang, Z. Meng, and I. King (2020) MAGNN: metapath aggregated graph neural network for heterogeneous graph embedding. In Proceedings of The Web Conference 2020, pp. 2331–2341. External Links: ISBN 9781450370233, Link Cited by: Appendix A, Appendix A, Appendix A, Appendix C, §I, §I, §II.
[8] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl (2017) Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263–1272. Cited by: §II.
[9] Z. Hu, Y. Dong, K. Wang, and Y. Sun (2020) Heterogeneous graph transformer. In Proceedings of The Web Conference 2020, pp. 2704–2710. External Links: ISBN 9781450370233, Link Cited by: Appendix C, §I, §I, §II, §V-B.
[10] T. N. Kipf and M. Welling (2017) Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, External Links: Link Cited by: Appendix C, §I, §II, §II, §V-B, §V-B.
[11] Q. Li, Z. Han, and X. Wu (2018) Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI conference on artificial intelligence, Cited by: §II.
[12] Y. Li, X. Liang, Z. Hu, Y. Chen, and E. P. Xing (2019) Graph transformer. External Links: Link Cited by: §II.
[13] Z. Lin, M. Feng, C. N. d. Santos, M. Yu, B. Xiang, B. Zhou, and Y. Bengio (2017) A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130. Cited by: §IV-A.
[14] M. Liu, H. Gao, and S. Ji (2020) Towards deeper graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 338–348. Cited by: §II.
[15] Q. Lv, M. Ding, Q. Liu, Y. Chen, W. Feng, S. He, C. Zhou, J. Jiang, Y. Dong, and J. Tang (2021) Are we really making much progress? revisiting, benchmarking and refining heterogeneous graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1150–1160. External Links: ISBN 9781450383325, Link Cited by: Appendix C, §I, §I, §II, §V-B.
[16] K. Oono and T. Suzuki (2019) Graph neural networks exponentially lose expressive power for node classification. arXiv preprint arXiv:1905.10947. Cited by: §II.
[17] C. Park, D. Kim, J. Han, and H. Yu (2020-Apr.) Unsupervised attributed multiplex network embedding. Proceedings of the AAAI Conference on Artificial Intelligence 34 (04), pp. 5371–5378. External Links: Link, Document Cited by: Appendix C, §II, §V-B.
[18] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. v. d. Berg, I. Titov, and M. Welling (2018) Modeling relational data with graph convolutional networks. In European semantic web conference, pp. 593–607. Cited by: Appendix C, §I, §II, §V-B.
[19] Y. Shen, H. Li, S. Yi, D. Chen, and X. Wang (2018) Person re-identification with deep similarity-guided graph neural network. In Proceedings of the European conference on computer vision (ECCV), pp. 486–504. Cited by: §I.
[20] W. Shi and R. Rajkumar (2020) Point-gnn: graph neural network for 3d object detection in a point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1711–1719. Cited by: §I.
[21] P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio (2018) Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, External Links: Link Cited by: Appendix C, §I, §II, §II, §V-B, §V-B.
[22] P. Veličković, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengio, and R. D. Hjelm (2019) Deep graph infomax. In International Conference on Learning Representations, External Links: Link Cited by: §II.
[23] D. Wang, J. Lin, P. Cui, Q. Jia, Z. Wang, Y. Fang, Q. Yu, J. Zhou, S. Yang, and Y. Qi (2019) A semi-supervised graph attentive network for financial fraud detection. In 2019 IEEE International Conference on Data Mining (ICDM), Vol. , pp. 598–607. External Links: Document Cited by: §I.
[24] G. Wang, R. Ying, J. Huang, and J. Leskovec (2019) Improving graph attention networks with large margin-based constraints. arXiv preprint arXiv:1910.11945. Cited by: §II.
[25] M. Wang, D. Zheng, Z. Ye, Q. Gan, M. Li, X. Song, J. Zhou, C. Ma, L. Yu, Y. Gai, T. Xiao, T. He, G. Karypis, J. Li, and Z. Zhang (2019) Deep graph library: a graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv:1909.01315. Cited by: Appendix C.
[26] X. Wang, H. Ji, C. Shi, B. Wang, Y. Ye, P. Cui, and P. S. Yu (2019) Heterogeneous graph attention network. In The World Wide Web Conference, WWW ’19, New York, NY, USA, pp. 2022–2032. External Links: ISBN 9781450366748, Link, Document Cited by: Appendix C, Appendix C, Appendix C, §I, §I, §II, §V-B.
[27] N. Wu and C. Wang (2022) GTNet: a tree-based deep graph learning architecture. arXiv preprint arXiv:2204.12802. Cited by: §I, §III-B, §III-B, §III-B, §III, §IV, §V-D.
[28] R. Yin, K. Li, G. Zhang, and J. Lu (2019) A deeper graph neural network for recommender systems. Knowledge-Based Systems 185, pp. 105020. External Links: ISSN 0950-7051, Document, Link Cited by: §I.
[29] S. Yun, M. Jeong, R. Kim, J. Kang, and H. J. Kim (2019) Graph transformer networks. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32, pp. . Cited by: Appendix A, Appendix C, §I, §I, §II, §V-B.
[30] M. Zhang and Y. Chen (2018) Link predition based on graph neural networks. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31, pp. . Cited by: §I.
[31] Z. Zhu, Z. Zhang, L. Xhonneux, and J. Tang (2021) Neural bellman-ford networks: a general graph neural network framework for link prediction. Advances in Neural Information Processing Systems 34. Cited by: §I.

Appendix A Datasets

ACM¹¹1https://dl.acm.org/ is a citation graph. It covers papers published in KDD, SIGMOD, VLDB, SIGCOMM and MOBICOMM. The papers are divided into three classes: Data Mining, Database, and Wireless Communication. The task is to classify the type of unlabeled papers. There are three types of nodes: Paper, Author and Subject, and four types of directed edges: Paper-Author (PA), Paper-Subject (PS) and their reverse connections. The feature of a paper node is obtained from the bag-of-words representations of its keywords. The feature of an author is obtained by averaging the features of papers that the author is connected with. The feature of a subject is calculated by averaging the features of papers that the subject is connected with. We use the same data processing as in GTN [29] and MAGNN [7] when a node’s feature is not available. We choose 600 balanced samples as the training set, another 300 balanced samples as the validation set, and all remaining 3,119 samples as the test set.

IMDB²²2https://www.imdb.com/ has three types of nodes: Movie, Director and Actor, and four types of directed edges: Movie-Director (MD), Movie-Actor (MA) and their reverse connections. The task is to predict the types of movies in one of the three classes: Action, Drama and Comedy. We use a subset of IMDB provided by the Pytorch Geometric library [6], which is the same subset as used by MAGNN [7]. The feature of a movie is obtained from the bag-of-words representations of its plot keywords. The features of a director and an actor are obtained by averaging the features of movies that are connected with the director and actor, respectively. We choose 300 balanced samples as the training set, another 300 balanced samples as the validation set, and all remaining 3,678 samples as the test set.

DBLP³³3https://dblp.uni-trier.de/ is a computer science bibliography website. It contains four types of nodes: Author, Paper, Term, and Conference, and six types of directed edges: Author-Paper (AP), Paper-Term (PT), Paper-Conference (PC) and their reverse connections. The task is to classify the authors into one of the four research areas: database, data mining, artificial intelligence, and information retrieval. We use a subset of DBLP provided by the Pytorch Geometric library [6], which is the same subset as used by MAGNN [7]. The feature of an author is obtained from the bag-of-words representations of the author’s papers. The feature of a paper is obtained by averaging the features of authors that the paper is connected with. The feature of a conference is obtained by averaging of the features of papers that the conference is connected with. The features of term nodes are provided by the DBLP dataset. We choose 800 balanced samples as the training set, another 400 balanced samples as the validation set, and all remaining 2,857 samples as the test set.

Appendix B Proposed Baselines - HetGCN and HetGAT

HetGCN. The mathematical formulation of HetGCN is described as below:

h0u=zu=σ(xuW0,a+b0,a)hnu,k=∑v∈Nku∪{u}^Ak,uvhn−1vWn−1hnu=ReLU(Ka∑k=1βnk⋅hnu,k)

(14)

where $n = 1, 2, \dots, L$ , denoting the message passing layer number. $β_{k}^{n}$ is the semantic attention weight for the edge type $k$ , and is calculated as

\begin{matrix} w_{k}^{n} = \frac{1}{| V_{a} |} \sum_{u \in V_{a}} {(q_{a}^{n})}^{T} \cdot tanh (W_{a}^{n} {(h_{u, k}^{n})}^{T} + b_{a}^{n}) β_{k}^{n} = softmax (w_{k}^{n}) \end{matrix}

(15)

HetGAT. The mathematical formulation of HetGAT is described as below:

\begin{matrix} h_{u}^{0} = z_{u} = σ (x_{u} W_{0, a} + b_{0, a}) h_{u, k}^{n} = ELU (\sum_{v \in N_{u}^{k} \cup {u}} α_{u, v, k}^{n - 1} h_{v}^{n - 1} W^{n - 1}) h_{u}^{n} = ReLU (K_{a} \sum k = 1 β_{k}^{n} \cdot h_{u, k}^{n}) \end{matrix}

(16)

where $n = 1, 2, \dots, L$ , denoting the message passing layer number. $α_{u, v, k}^{n}$ is the node-level attention weight for node pair $(u, v)$ through type $k$ connection, and $β_{k}^{n}$ is the semantic attention weight for edge type $k$ , which are calculated as

\begin{matrix} α_{u, v, k}^{n} = softmax (% LeakyReLU ([h_{u}^{n} W^{n} ∥ h_{v}^{n} W^{n}] a_{k}^{n})) w_{k}^{n} = \frac{1}{| V_{a} |} \sum_{u \in V_{a}} {(q_{a}^{n})}^{T} \cdot tanh (W_{a}^{n} {(h_{u, k}^{n})}^{T} + b_{a}^{n}) β_{k}^{n} = softmax (w_{k}^{n}) \end{matrix}

(17)

Appendix C Experimental Setup

We use 64 hidden units for all models, except that we use 128 hidden units for the semantic attention layer in HAN, HetGCN, HetGAT, HetGTCN and HetGTAN. DMGI is trained using a maximum epoch of 10,000 and an early stopping patience of 20. MAGNN is trained using a maximum epoch of 100 and an early stopping patience of 10. All other models are trained using a maximum epoch of 500 and an early stopping patience of 100.

GCN [10]. We use a two-layer model. The dropout rate is set to 0.5 for all datasets, and is applied to the initial projection layer and the output of each intermediate GCN layer. Adam optimizer is used with a learning rate of 0.005 and weight decay of 5e-5 for all three datasets.

GAT [21]. We use a two-layer model with one attention head. There are two dropouts: one is the dropout after each intermediate layer and another is the attention dropout. The dropout rates are (0.8, 0.2), (0.8, 0.2) and (0, 0) for ACM, IMDB and DBLP, respectively. Adam optimizer is used with a learning rate of 0.005 and weight decay of 5e-5 for all three datasets.

RGCN [18]. We use a two-layer model. The dropout rates are set to 0.5, 0.5 and 0 for ACM, IMDB and DBLP, respectively. The number of bases is set to 5. Adam optimizer is used with a learning rate of 0.005 and weight decay of 1e-5 for all three datasets. Since RGCN is designed for multi-relational knowledge graphs with a single node type, we first apply a node-type-specific transformation to project all node features into the same vector space, and then implement RGCN with the code provided by the Pytorch Geometric library [6].

HAN [26]. We use the same settings as the original paper [26]. The pre-selected meta-paths of ACM are PAP and PSP. The pre-selected meta-paths of IMDB are MDM and MAM. The pre-selected meta-paths of DBLP are APA, APCPA and APTPA. The number of attention heads is 8, and the dropout rate is 0.6. Adam optimizer is used with a learning rate of 0.005 and weight decay of 0.001 for all three datasets. We implement HAN with the code provided by DGL [25].

MAGNN [7]. All settings are the same as the original paper except that we use 64 hidden units. The pre-selected meta-paths are the same as used by HAN [26] for all three datasets. The number of attention heads is 8 with each head having 8 hidden units. The dropout rate is 0.5. Adam optimizer is used with a learning rate of 0.005 and weight decay of 0.001 for all three datasets. [7] runs on full batch of ACM and IMDB, and minibatch⁴⁴4Running on full batch of DBLP results in the Out-of-Memory issue. of DBLP with a batch size of 8 and the number of neighbor samples as 100. The preprocessing time of MAGNN is at least O( $N^{3}$ ) such that we are unable to extract the meta-path-based neighbors for DBLP due to the Out-of-Memory and Out-of-Time issue. Therefore, we use the preprocessed meta-path information provided by the authors, and implement MAGNN with the authors’ official code⁵⁵5https://github.com/cynricfu/MAGNN.

GTN [29]. For ACM and IMDB, we use 2 channels and 32 hidden units for each channel. For DBLP, we use 2 channels and 16 hidden units for each channel as using 32 hidden units for each channel results in the Out-of-Memory issue with our 12 GB vRAM RTX 3060 GPU. We use two GTN layers for ACM and DBLP, and three GTN layers for IMDB. Adam optimizer is used with a learning rate of 0.005 and weight decay of 0.001 for all three datasets. We implement GTN with the authors’ official code⁶⁶6https://github.com/seongjunyun/Graph_Transformer_Networks.

HGT [9]. The hyperparameters are fine-tuned for all three datasets to achieve the best performance. The number of attention heads is set to 4. We use a two-layer model for ACM and IMDB, and a three-layer model for DBLP. The dropout rate is set to 0.2. Adam optimizer is used with a learning rate of 0.005 and weight decay of 5e-5 for all three datasets. We implement HGT with the code provided by the Pytorch Geometric library [6].

SimpleHGN [15]. The feature and attention dropout rates are both set to 0.5 for all three datasets. $β$ is set to 0.05. Adam optimizer is used with a learning rate of 0.01 and weight decay of 0 for all three datasets. We use a two-layer model with eight attention heads for all three datasets which yield matching or better results than with the settings in the original paper [15]. We implement SimpleHGN with the authors’ official code⁷⁷7https://github.com/thudm/hgb.

DMGI [17]. We use the same settings as the original paper [17]. The authors use the same pre-selected meta-paths as HAN [26] does. The dropout rate is 0.5. Adam optimizer is used with a learning rate of 0.0005 and weight decay of 0.0001 for all three datasets. The self-connection weight is set to 3. The consensus regularization coefficient $α$ is set to 0.001 and the semi-supervised loss coefficient $γ$ is set to 0.1. We implement DMGI with the authors’ official code⁸⁸8https://github.com/pcy1302/DMGI.

HetGCN. We use a two-layer model. The dropout rate is set to 0.5, 0.5 and 0 for ACM, IMDB and DBLP, respectively. Adam optimizer is used with a learning rate of 0.005 and weight decay of 1e-5 for all three datasets.

HetGAT. We use a two-layer model with one attention head. There are two dropouts: one is the dropout for the feature projection layer, and the other is the dropout after each intermediate layer. The dropout rates are set to (0.8, 0.2), (0.8, 0.2) and (0, 0) for ACM, IMDB and DBLP, respectively. Adam optimizer is used with a learning rate of 0.005 and weight decay of 5e-5 for all three datasets.

HetGTCN. We use a five-layer model. There are also two dropouts: one is the dropout for the feature projection layer, and the other is the dropout after each intermediate layer. The dropout rates are set to (0.8, 0.6) for all three datasets. Adam optimizer is used with a learning rate of 0.005 and weight decay of 1e-5 for all three datasets.

HetGTAN. We use a five-layer model. Other settings are the same as HetGAT.

The code to replicate our experiments is available at https://github.com/hetgnn/hetGTNet.