Exploiting Hybrid Semantics of Relation Paths for Multi-hop Question Answering Over Knowledge Graphs

In this section, we mathematically present our TERP method in detail. Our main idea is to coordinate the question view and the relation path view to identify target entities more accurately. Figure 1 shows an overview of our approach. Firstly, we obtain the representations of the entities and relations in KG via a KG embedding module and the representations of questions via a Question Encoder. Then, we use a Path Encoder to encode the relation path by integrating explicit textual semantics and implicit KG embedding features of relation paths. An attention mechanism is employed to choose the appropriate relation paths semantically consistent with the question among numerous noisy candidate paths. A Rotate-and-Scale module projects the representations of the question and the chosen relation paths into the complex space of KG embeddings. Finally, an Entity Predictor scores all candidate entities in a link prediction manner.

We first obtain the representations of entities and relations via a KG Embedding module. To mine the implicit KG structural semantics of relation paths, we adopt RotatE Sun et al. (2019) to model the composition of relations.

RotatE represents entities as complex vectors and relations as rotations in complex vector space. Given $h,t\in E$ and $r\in R$, RotatE generates $e_h,e_r,e_t\in C^d$ and defines a scoring function:

where $|{e_r}_i|=1$, and $\circ$ denotes the Hadmard (or element-wise) product. RotatE can model the composition patterns. A relation ${e_r}_3=exp\left({i\theta }_3\right)$ is a combination of other two relations ${e_r}_1=exp\left({i\theta }_1\right)$ and ${e_r}_2=exp\left(i{\theta }_2\right)$ if and only if:

Following previous work Saxena et al. (2020); Shi et al. (2021); Atzeni et al. (2021), the question encoding model aims to embed a natural language question $Q$ to a fixed dimension vector $q\in R^d$ with a pre-trained language model.

where $\mathrm{avg}$ denotes the average pooling strategy.

In addition to the question, we further model the explicit and implicit semantics in relation path between a topic entity and a candidate entity with a path encoding module. Our intuition behind this is that the textual semantics in the relation path, as well as the corresponding composed pre-trained relation embeddings produced by RotatE, are supplementary information to the question. Considering that the same relation path may have different semantics in different query contexts, we additionally add the question text before the textual description of the relation path. Formally, given a natural language question $Q$, a topic entity $h$ and a candidate entity $c$, we can obtain the shortest paths between $h$ and $c$. For a single path ${r_p}^1,{r_p}^2,\cdots {r_p}^k$ , we generate a textual relation path $P_t$, in which every text description of relation in the path is surrounded by special tokens $<r>,</r>$. We concatenate the textual question $Q$ with the textual relation path $P_t$ and feed them into the encoder to extract the textual features of the explicit relation paths. Meanwhile, we can obtain the implicit semantics from the embeddings of the relations ${{e_r}_p}^1,{{e_r}_p}^2,\cdots {{e_r}_p}^k$, which carry structural features learn by RotatE. The hybrid representation $\overline{p}$ of the relation path is produced via fusing both explicit textual representation $p_t$ and implicit KG embedding representation $p_l$ with an $FFN$:

One underlying challenge is that there could be multiple shortest paths between a topic entity and an answer entity. In TERP, we use a scaled dot-product attention mechanism to select the appropriate relation paths that are semantically consistent with the question.

where ${\overline{p}}_t$ and $\overline{p}$ denote two version of representations of all the candidate paths for a given $(h,c)$ pair. $W_1,W_2$ are learnable matrices.

However, in our preliminary exploration, directly using RotatE as our KG embedding module do not yield a robust performance. Further investigation revealed that the underlying reason is that the modulus of entities varies in RotatE. If there is an edge between two entities, they will have similar modulus because the modulus of relation representations are fixed to be 1 in RotatE. However, in a multi-hop KBQA scenario, the multi-hop relation path could amplify the difference between the topic and answer entities, hence it can be challenging to match the answer only by rotation transformation.

To this end, we propose a rotate-and-scale framework to model the two views of implied relational fact of a <topic entity, target entity> pair as a rotation transformation and a scaling transformation in the complex space. For a natural language question $Q$, two independent feedforward networks ($\mathrm{FFN}$) are used to generate the rotation transformation ${\theta }_q\in R^d$ and the scaling transformation $m_q\in R^d$:

Then, we combine the two transformations into the final representation $r_q$ for the question, which contains a real part $\mathrm{Re}$ and an imaginary part $\mathrm{Im}$ in the complex space.

In this way, our rotate-and-scale framework can serve as the bridge between entity representations of RotatE and representations of textual questions. We handle relation paths in a same way.

where ${\theta }_p\in R^d$ and $m_p\in R^d$ are rotation and scaling in the complex space respectively. As illustrated in Section 5.2, the rotate-and-scale mechanism improves the performance of KGQA with a large margin.

With the representations produced above, an entity predictor is used to score all candidate entities. Given a question $Q$, the candidate paths $P$, a topic entity $h\in E$ and the candidate entity $c\in E$, the score function is calculated as:

where $s_q$ and $s_p$ denote the scores from the question view and the relation path view, respectively. The final score is $s=(1-\lambda )s_q(h,c)+\lambda s_p(h,c)$, where $\lambda$ is a hyper-parameter. In training, the score $s$ is calculated among $N$ candidate entities sampled from the KG, where $N$ is a hyper-parameter.

The overall training objective combines the Cross-entropy (CE) loss $L_{ques}$ and $L_{path}$ for the $s_q$ and $s_p$, respectively.

where $targets$ denotes the ground truth label.

To address the challenge that huge numbers of paths may exist, we propose a two-stage inference strategy to reduce the computational cost. At stage 1, given a question $Q$, a topic entity $h$ and all the entities in the question-specified subgraph $C\subseteq E$, we first compute $s_q(h,c)$ for each $c\in C$. Then we select top-k candidate entities among them according to $s_q(h,c)$. At stage 2, we compute $s_p(h,c)$ only for the entities recalled in stage 1 and calculate the final score $s$ from them. For questions with more than one topic entity, we simply average the corresponding $s_q$ and $s_p$ calculated by different topic entities for each candidate $c$.

This two-stage answer acquisition strategy can empirically deliver a 15-40$\times$ inference speed-up on different datasets without sacrificing performance.

We evaluate our model on three widely-used KGQA datasets, MetaQA Zhang et al. (2018), WebQuestionsSP Yih et al. (2016) datasets, and Complex WebQuestions Talmor and Berant (2018).

We use the open source implementation of LibKGE Broscheit et al. (2020) to train the KG embeddings. Following Saxena et al. (2020), the pre-trained KG embeddings are frozen for WebQSP and CompWebQ in training, while tuneable for MetaQA. We use a pre-trained RoBERTa Liu et al. (2019) as the text encoder. The size of a mini-batch is set to 10. The learning rate is 3e-5 and we adopt the Adam optimizer with ${\beta }_2$ = 0.998. The number of candidate entities for WebQSP and CompWebQ is 20000. For MetaQA, we use all entities in KG as candidate entities. Other hyper-parameters are the same as the default RoBERTa-base configuration. The weight $\lambda$ for the entity predictor is 0.6. The number of candidate entities retrieved in stage 1 during inference is empirically set to be 15 for WebQSP and MetaQA, and 30 for CompWebQ.

PullNet Sun et al. (2019) iteratively retrieves a subgraph from KG to create a question-specific sub-graph and rank the entities by a variant of graph CNN Kipf and Welling (2017); EmbedKGQA Saxena et al. (2020) leverages KG embeddings to perform multi-hop KGQA. It adopts ComplEx Trouillon et al. (2016) KG embeddings to score the entities; EMQL Sun et al. (2020) leverages query embedding method and uses these embeddings to obtain the answers; TransferNet Shi et al. (2021) leverages free texts retrieved from the textual corpus and pre-defined constrained predicates to perform multi-hop reasoning; BERT-KGQA Yan et al. (2021) leverages textual information carried by the nodes and edges to perform KGQA. We choose the original version without additional annotated data for a fair comparison; SQALER Atzeni et al. (2021) addresses KGQA by first performing multi-hop reasoning on the KG and then refining the result with logical reasoning.

Table 1 shows the performance of the baseline models and our methods on three datasets under different settings. We achieve the best performance on four of six tasks. Here we mainly compare our TERP with two lines of works: embedding-based methods (e.g., EmbedKGQA) and path searching methods (e.g., SQALER and TransferNet).

Comparison with embedding-based methods. Except for the similar performance on the MetaQA 1-hop task, TERP significantly outperforms EmbedKGQA on the other tasks. The results verify the effectiveness of incorporating relation path information into the link prediction framework.

Comparison with path searching methods. Generally, TERP performs better on WebQSP and ComWebQ, while SQALER and TransferNet are more competitive on MetaQA. The possible reason is that the link prediction framework relies on high-quality KG embeddings, consequently being more effective for knowledge graphs of a larger scale. Note that the scale of WebQSP’s knowledge graph is far more extensive than that of MetaQA’s (1.8 M v.s. 43 K of entity number, and 6101 v.s. 9 of relation type number). In addition to better trained KG embeddings, the 6101 relation types of WebQSP mean 6101 relation representations, introducing much richer semantics of relations, compared with the 9 relation representations of MetaQA. The difference between these knowledge graphs roughly explains the comparison results. Furthermore, the results on more challenging tasks (WebQSP and CompWebQ) verify the effectiveness of integrating explicit textual information and implicit KG structural information in KGQA.

Note that several baselines (EMQL, BERT-KGQA and PullNet) can not accurately fall into the above two categories. Compared with them, TERP also achieves competitive results, e.g., the superior hits@1 scores in 4 of the 6 test sets.

To reveal how the rotate-and-scale mechanism helps answer reasoning, we replace it with general ComplEx-based matching and RotatE-based (without scaling) matching, achieving two model variants named w/ ComplEx and w/ RotatE respectively. Table 2 contains two groups of results corresponding to whether or not hybrid features of relation paths is introduced. The observations on the two groups are generally similar, and here we mainly analyze the results in the first group.

First, w/ RotatE demonstrates a notable performance degradation compared with  w/ ComplEx, suggesting that simply replacing ComplEx with RotatE in the link prediction-based KGQA framework, can not satisfy our initial desire to exert relation composition capabilities of RotatE. Second, by incorporating the scaling module into RotatE, w/ RotatE&Scale surpasses w/ ComplEx with a significant margin. This observation verifies that modulus scaling is necessary to capture relation semantics under the hypothesis of using complex vector rotating to match complex multi-hop questions. Third, the superiority of w/ RotatE&Scale over w/ ComplEx is more visible on 2-hop questions than that on 1-hop ones, proving that w/ RotatE&Scale more accurately distinguish relation path semantics.

Another characteristic of TERP is using hybrid features of relation paths. In Table 2, the models of the second group are ones with relation path features. By comparing the results of the first group and second group in Table 2, we find 1) incorporating relation path information can consistently improve answering questions of different hops under both complete and incomplete KGs, and 2) the improvements on 2-hop questions surpass that on 1-hop ones by a large margin, verifying the potential of relation path information for multi-hop reasoning.

We then perform an ablation study on the hybrid features. Table 3 shows two groups of results corresponding to using only textual representations and only structural representations of relation paths, respectively. ComplEx does not support relation composition, so we only experiment on RotatE and RotatE&Scale. We have three observations here.

First, both textual and structural features improve model performance, indicating that the two kinds of relation path information benefit answer selection. Second, textual information brings more significant enhancements than structural information. The reasons are two-fold. On the one hand, structural information mainly involves multiplication of relation embedding, which overlaps more with implicit semantics in the link prediction process. On the other hand, textual information provides more complementary knowledge for link prediction, from another modality in a sense. Finally, combining them delivers further improvement, verifying the efficacy of the question-aware fusing process to capture the hybrid semantics.

Considering that exploiting relation paths also introduces many spurious ones, the collaboration of their hybrid features and questions is critical to balance the positive and negative effects. Therefore, we first analyze the impact of the hyper-parameter $\lambda$, which denotes the weighting strategy between predicting scores of questions and relation paths.

In Figure 2, the blue, green, and red polylines show the Hits@1 scores of all 1-hop, 2-hop, and 1&2-hops questions on WebQSP, respectively. Looking into these three polylines’ trends, we find that our model is best-performed when $\lambda$ is 0.6, indicating the textual information can not either be ignored or overly dependent. In other words, we need to distinguish necessary features under tolerable noises introduced by a set of off-the-shelf relation paths. Another interesting observation is that the upward trend before the peak of the green line (2-hop questions) is more evident than that of the blue line (1-hop questions), though their downward trends after the peak are similar. The reason is that relation path information is more critical for multi-hop reasoning, and our method well characterizes them, hence delivering robust improvements.

To further investigate how relation paths and questions collaborate, we calculate the cosine similarity between relation path text and question text representations for WebQSP. Since there may be multiple candidate relation paths, the relation path with maximum similarity is selected. We then equally divide data samples in the test set into five groups, based on the cosine similarity scores. The performance of two compared models (w/ path and w/o path) for each group is shown in Figure 3, from which we observe two interesting trends.

First, model performance degrades as cosine similarity decreases. For example, the hits@1 for “Very High” and “Very Low” differ enormously (e.g., 98.4 v.s. 39.0 with our full TERP). Intuitively, a question is relatively easy to answer if it is similar to a potential relation path. Otherwise, it is more challenging to find the answer. In other words, the question may not provide enough clues, making question understanding more difficult. Second, the relation path information provides more significant improvement for more difficult questions. Incorporating relation path information may even hinder model performance for the groups of “Very High” and “High” (e.g., -0.2 and -0.1 hits@1). This is because many relation paths will bring noises but no extra valuable clues. On the contrary, the hits@1 improvements on “Medium”, “Low”, and “Very Low” are +1.0, +2.2 and +7.3, respectively. These results clearly demonstrate that relation paths provide complementary information for hard questions, and our method effectively extracts and synthesizes essential features of relation paths. That is where the superiority of our method mainly comes.