Learning Representations for Hyper-Relational Knowledge Graphs

Knowledge graph embeddings (KGE) use embeddings to represent the latent features of entities and relations in a KG. Many KGE techniques also employ a score function that produce a score for a given triple $(h,r,t)$ measuring the plausibility of the triple being true ji2021survey. Many different frameworks have been proposed in the literature. bordes2013translating proposed using a translational scoring function to model triples. yang2014embedding use a bilinear scoring function to score triples utilizing a diagonal matrix to model relations. RotatE sun2019rotate scores a triple as a translation in a complex space and ConvE dettmers2018convolutional does so using a convolutional neural network. Other works have focused on modeling knowledge graphs using graph neural networks. RGCN schlichtkrull2018modeling extends GCN kipf2016semi by using a relation specific weight matrix when aggregating an entity’s neighbors. However, RGCN suffers from over-parameterization when there are many relations. To alleviate this concern, CompGCN vashishth2019composition proposes to use direction specific weight matrices when aggregating instead of relational weight matrices.

Several methods have been proposed that model hyper-relational facts as N-ary facts. wen2016_transh propose m-Transh, a method that builds on TransH wang2014knowledge by transforming each hyper-relational fact using a star-to-clique conversion. RAE zhang2018_rae builds upon m-TransH wen2016_transh by further considering the relatedness between two entities. NaLP guan_nalp uses a convolutional-based framework to compute a relatedness vector for a triple and its qualifier pairs that can be used for prediction.

More recently, there has been some work that model hyper-relational KGs from a purely hyper-relational viewpoint. rosso_hinge propose doing so using a convolutional framework. For a given hyper-relational fact, the base triple is convolved by itself and with each specific qualifier pair. The resulting feature vectors are then combined and used for prediction. StarE galkin2020message extends CompGCN vashishth2019composition by encoding the qualifiers for a specific triple and combining it with the base relation of the triple. Using StarE galkin2020message as their foundation, yu_hytransformer replace the GNN aggregation with layer normalization layers ba2016layer to improve performance. Additionally they mask the qualifier entities when training as a form of self-supervised learning (SSL). Lastly, wang_gran propose GRAN, a transformer based architecture that employs edge-specific attention biases and masks all entities and relations in the sequence. A missing element in previous work is the lack of a clear flow of information from the base triples to the qualifiers. To address this, we propose a graph encoder to aggregate information from the perspective of the qualifiers, thereby creating better qualifier encodings.

Let $G=\{V,R,E\}$ be a knowledge graph with nodes (i.e. entities) $V$, edges $E$, and relations $R$. For $e\in E$, it represents a directed edge where two entities $u\in V$ and $v\in V$ are connected by a relation $r\in R$. We also denote the edge as a triple $(v,r,u)$. Further, we use $N_v$ to denote the neighboring entities and relations of a node $v$.

CompGCN vashishth2019composition, a popular method for modeling knowledge graphs, utilizes a direction specific weight matrix $W_{\lambda \left(r\right)}$ and a function $\varphi$ that combines the neighboring entity $h_u$ and relation $h_r$ of a given edge

where $f$ is a non-linear function (e.g. ReLU) and $\lambda \left(r\right)$ represents the direction of relation that can be one of: a standard, inverse, or self-loop relation. Several function are proposed for modeling the interaction of the embedding in $\varphi$ including subtraction, multiplication, and the cross correlation vashishth2019composition. The relation embedding is updated through a transformation by a weight matrix $W_{rel}$,

A hyper-relational knowledge graph can be seen as an extension of a standard knowledge graph where there is a set of qualifier pairs $\left\{\right(q{v}_i,q{r}_i\left)\right\}$, where $qv\in V$ and $qr\in R$, associated with each triple $(v,r,u)$. For simplicity we use $q$ to represent the set of neighboring qualifier pairs $(q{v}_i,q{r}_i)\in Q_{(v,r,u)}$ associated with a triple $(v,r,u)$. We can therefore represent a hyper-relational fact as $(v,r,u,q)$. We refer to hyper-relational facts as statements. Furthermore, we can represent the neighborhood for a qualifier entity $qv$ as $(v,r,u,qr)\in N_{qv}$.

A representative example for modeling hyper-relational KGs is StarE galkin2020message. It proposes a formulation based on CompGCN to incorporates an embedding $h_q$ that is an encoded representation of the qualifier pairs for the base triple $(v,r,u)$,

The embedding $h_q$ is combined with the relation embedding $h_r$ through a function $\gamma$ that performs a weighted sum. yu_hytransformer show that replacing the graph aggregation with layer normalization can achieve comparable if not better performance than other frameworks. Furthermore they show that masking the qualifier entities during training can raise the test performance on knowledge graph completion.

Knowledge graph completion (i.e. link prediction) masks one of the two entities belonging to the triple and attempts to predict the correct entity. For example, given $(v,r,u)$ we would try to predict the correct entity for both $(v,r,?)$ and $(?,r,u)$. This is defined similarly for hyper-relational KGs where we are also provided with the qualifier information for the triple. Therefore, given $(v,r,u,q)$ we would try to predict the correct entity for both $(v,r,?,q)$ and $(?,r,u,q)$.

In this subsection, we define the encoder used in QUAD. The encoder is composed of two neighborhood aggregations: one that aggregates information for the base entities and one that does so for the qualifier entities. We refer to these two aggregators as the base aggregator and qualifier aggregator, respectively. The initial entity $E$ and relation $R$ embeddings are first passed to the base aggregator and then the qualifier aggregator. The encoded entity embedding $\hat{E}$ and relation embedding $\hat{R}$ can be expressed as follows:

where $\text{Base-Agg}(\cdot )$ and $\text{Qual-Agg}(\cdot )$ are the aggregation functions in base aggregator and qualifier aggregator, respectively. In the following, we introduce the details of both aggregators.

Using the encoded representations of the entities and relations, i.e., $\hat{E}$ and $\hat{R}$, each hyper-relational fact is passed to a decoder to make the final prediction. To decode each fact we utilize a transformer vaswani2017attention that employs an architecture similar to CoKE wang2019coke extended to include qualifier information. For a given input sample $S$, we mask the entity token we are trying to predict. Eq. (13) is an example where we mask the object entity:

After being passed through the transformer, we extract the masked embedding and pass it through a fully-connected layer and score it against all possible entities. This is then passed through a sigmoid function with the highest scoring entity being chosen as our prediction.

We also consider a version of QUAD that combines the entity representations encoded by the two aggregation schemes in parallel instead of sequentially. Under this setting, both aggregate functions take the initial entity embedding matrix $E$ as input. The encoded entity representations outputted by the two encoders are then combined via a weight matrix $W_p$ and passed to the decoder. The relation embeddings are still passed sequentially as we found that this performed best. We believe that this version of our framework may be better at balancing the contribution of the base and qualifier aggregation for some datasets. We formulate the parallel encoding scheme as follows where $E_{base}$ and $R_{base}$ are the output of $\text{Base-Agg}(\cdot )$ and $E_{qual}$ the output of $\text{Qual-Agg}(\cdot )$:

In this subsection we provide a unified view on several popular hyper-relational KG methods by showing that they are subsets of our framework.

In this subsection, we conduct an ablation study to determine the importance for each component in QUAD. We do so by considering three versions of QUAD: (1) One that doesn’t mask the qualifier entities, (2) one without the qualifier aggregation component and (3) one without (1) and (2). Evaluating the results on the three ablated frameworks will help us ascertain both the individual and cumulative effect those two components have on the performance. Of most importance is the impact of the novel qualifier aggregation introduced in this paper. We report the results of this study on the WD50K (100) dataset under the original (non-parallel) setting. The results can be found in Table 3.

Evaluating the results in Table 3 we see that the removal of either or both of the two components leads to a degradation in performance. Ablating the qualifier aggregation, the main contribution of our framework, causes a 1% reduction in MRR and a 1.6% drop in H@1. Furthermore, removing just the qualifier masking results in a 3.7% reduction in MRR and ablating both components leads to the largest drop with a 6.4% drop in MRR. These results validate the importance of the qualifier aggregation as its removal leads to a degradation in performance.

In this subsection, we study the effect that the loss balancing hyperparameter $\beta$ has on the performance. To do so we train and evaluate multiple versions of our framework with values of $\beta$ in the set $\{0,0.25,0.5,0.75,1\}$. We note that a value of $\beta =0$ is equivalent to no qualifier masking while a value of $\beta =1$ is equivalent to the loss used in Hy-Transformer yu_hytransformer. We perform this study on the three WD50K splits and JF17K to determine the impact the loss balancing hyperparameter has on a variety of datasets. We utilize the parallel setting when training on JF17K and our original formulation on the WD50K splits. For simplicity we report only the MRR performance in Table 4.

The results in Table 4 shows multiple trends. For WD50K (100) there is a clear relationship with the value of $\beta$ and the performance. The higher the value of $\beta$ the better the performance. This is likely due to the fact that every triple in the dataset has at least one qualifier pair. For both WD50K (33) and (66) the MRR improves as we increase $\beta$ from 0 to 0.50 but we see little to no improvement when increasing the value from 0.50 to 1. Lastly, for JF17K the MRR is maximized when $\beta =0.25$. Furthermore, for increasing values of $\beta$ beyond 0.25 we see a noticeable drop in performance resulting in a MRR 2.2% lower at $\beta =1$ compared to $\beta =0.25$. This shows the importance of balancing the magnitude of the auxiliary loss, as depending on the dataset balancing them equally ($\beta =1$) can potentially lead to a large degradation in performance.