Supporting Medical Relation Extraction via Causality-Pruned Semantic Dependency Forest

Previous work performs the MRE task by constructing the 1-best dependency tree of sentences Peng et al. (2017); Song et al. (2018). However, the accuracy of the 1-best dependency tree generated by the out-of-domain parser is relatively low, resulting in a fall in MRE performance. Therefore, Song et al. (2019) proposes to use dependency forests to solve this problem, which uses EDGEWISE and KBESTEISNER algorithm to pick edges to construct dependency forests. Jin et al. (2020) encodes all effective dependency trees generated by a parser into dependency forests. Guo et al. (2021) utilizes multi-head attention and Kirchhoff’s Matrix-Tree Theorem (MMT) Koo et al. (2007) to automatically generate latent dependency forests without the usage of any parser. In general,  Song et al. (2019) and Jin et al. (2020) focus more on the syntactic information in the 1-best dependency tree generated by the out-of-domain parser, while Guo et al. (2021) directly discards the syntactic information and focuses only on the semantic information.

Causal explanation is designed to explain the importance of each module in a machine learning model on the prediction, which receives increasing attention recently Datta et al. (2016); Schwab and Karlen (2019); Lin et al. (2021). There are several viable forms of causality, including Granger causality Granger (1969), causal Bayesian networks Pearl (1985), and structural causal models Pearl (2009). Chattopadhyay et al. (2019) proposes an attribution method based on the first principles of causality. Schwab and Karlen (2019) models the explanation task of image deep learning models as a causal learning task and proposes a causal explanation model based on Granger causality. Lin et al. (2021) proposes a framework for explaining graph neural networks using the first principles of Granger causality.

Our task is to extract relation between entities in a sentence, focusing on both binary relation extraction and ternary relation extraction. Formally, the input to our task is a sentence $S=\{w_1,w_2,\dots ,w_n\}$ with $n$ words and $w_i$ denotes the $i$-th word in the sentence. $S$ is annotated with entity mentions $E_1$ and $E_2$¹¹1$E_1$, $E_2$ and $E_3$ for ternary relation extraction.. The output is the relation between entities from a predefined relation set $R=\{r_1,r_2,\dots ,r_m\}$, where $m$ denotes the number of relations.

Graph Neural Network is a set of models that can effectively encode the information of graph structure, the classical models including Graph Attention Networks (GATs) Velickovic et al. (2017), Graph Convolutional Networks (GCNs) Kipf and Welling (2016), etc. Densely-Connected Graph Convolutional Networks (DCGCNs) Guo et al. (2019b) is a variant of GCNs, which introduces dense connections to GCNs. Thus being able to build multi-layer GCNs models with a large depth and learn richer information than the shallower GCNs models. More specifically, DCGCNs differs from GCNs in that the embedding of node $v$ in the $l$-th layer receives information from all the preceding layers, which can be formulated as follows:

where $\times$ denotes matrix multiplication, $h_v^{\left(l\right)}$ is the embedding of node $v$ in the $l$-th layer, $\rho$ is an activation function, $N\left(v\right)$ denotes the neighbours of node $v$, $W^{\left(l\right)}$ and $b^{\left(l\right)}$ are the weight matrix and bias vector of the $l$-th layer respectively, and $g_u^{\left(l\right)}$ indicates the information about node $u$ from all the preceding layers. Mathematically, $g_u^{\left(l\right)}$ can be calculated by concatenating the initial embedding $x_u$ and the node embedding $h_u^{\left(1\right)};\dots ;h_u^{(l-1)}$ produced in layer $1,\dots ,l-1$, respectively.

To construct the 1-best dependency tree, we use Standard CoreNLP Toolkits (SCT) Manning et al. (2014) to obtain the dependency tree $T$ for each input sentence $S$ and represent $T$ by a adjacency matrix $T=(t_{i,j}{)}_{n\times n}$²²2The adjacency matrix $T$ adds the self-loop of each word to the dependency tree $T$ with the “self” dependency type and regards the dependencies between words as unoriented., where $t_{i,j}$ is the dependency type (e.g., dobj) between $w_i$ and $w_j$, e.g., $t_{i,j}=0$ if the connection between $w_i$ and $w_j$ do not exist. Then, we encode $t_{i,j}$ to the corresponding embedding $c_{i,j}^t$ with a learnable matrix, and use $C=(c_{i,j}^t{)}_{n\times n}$ to denote the syntactic matrix.

In the medical domain, the quality of the 1-best dependency tree generated by the out-of-domain parsers is relatively deficient. Thus, we propose the Causality-pruned Semantic dependency Forest Generator (CSFG) to generate dependency forests enriched with syntactic and semantic information and derive task-relevant information from them.

Given $N$ pruned dependency forests, DCGCNs are used to encode information from the forest structure. For the $p$-th pruned dependency forest, which is represented by the adjacency matrix ${\hat{F}}^p$. We use DCGCNs with $L$ layers to aggregate information about neighbors in ${\hat{F}}^p$, and the representation of node $i$ at the $l$-th layer can be calculated as:

where ${\hat{F}}_{ij}^p$ denotes the weight between node $i$ and node $j$ in ${\hat{F}}^p$. $g_{p_j}^{\left(l\right)}$ denotes the information about node $j$ in the $p$-th pruned dependency forest from all the preceding layers and can be obtained by the same way as Eq. 2.

Then, we concatenate the representations obtained from the $N$ dependency forests and fuse them using a linear layer. This process can be formulated as follows:

where $H^i$ is the node representations obtained by DCGCNs for the $i$-th dependency forest, and $H_{\text{b}}$ denotes the node representations of each block. $M$ identical blocks are combined in the same way as above to obtain the final node representations for sentence $S$, denoted as $H$.

To predict the relations among entities, the max pooling mechanism is used. We obtain the global sentence representation $h_S$ by applying the max pooling function to all the words in sentence $S$:

where $h_i$ is the feature vector of word $w_i$, and then obtain the representation of each entity by applying the max pooling function to the words that belongs to an entity mention (i.e., $E_q$). Therefore, the entity representation of $E_q$ can be obatined by:

The sentence representation and entity representations are concatenated and fed into a feed-forward neural network (FFNN), and then we transform it into an $m$-dimensional vector $h_R$ using a linear layer to make a prediction:

where $Q$ is 2 in the binary relational extraction task and is 3 in the ternary relational extraction task, $m$ denotes the number of relations $R$.

We evaluate our model on three datasets with two types of tasks: cross-sentence n-ary relation extraction and sentence-level relation extraction following Guo et al. (2021).

For the cross-sentence n-ary relation extraction task, we use the dataset extracted by Peng et al. (2017) based on PubMed. Most of the instances in this dataset contain multiple sentences, and the entities in the instances are cross-sentence. In detail, this dataset contains 6987 instances of ternary relations and 6087 instances of binary relations, each of them is divided into five folders according to Song et al. (2018). The relation between entities in each instance belongs to one of the relation sets, including “resistance or non-response”, “sensitivity”, “response”, “resistance”, and “None”. Following Guo et al. (2021), we define two sub-tasks on this dataset: multi-class and binary-class relation extraction. For multi-class relation extraction, we keep the original dataset unchanged, and for binary-class relation extraction, we define the first four relations as “Yes” and the “None” as “No”.

For the sentence-level relation extraction task, we use two datasets for Medical Relation Extraction, namely, BioCreative Vi CPR (CPR) Krallinger et al. (2017) and Phenotype-Gene relation (PGR) Sousa et al. (2019). CPR focuses on the relations between chemical components and human proteins, which contains six relation types (“CPR:3”, “CPR:3”, “CPR:4”, “CPR:5”, “CPR:6”, “CPR:9”, “None”). PGR focuses on whether human phenotypes are related to human genes, which contains two relation types (“TRUE” for related and “FALSE” for unrelated). The number of instances for train/dev/test sets of CPR and PGR datasets is shown in Table 1.

During the causal explanation generation process, we use a pre-trained CP-GCN model without the task-specific causal pruning module as $f_{\text{MRE}}(\cdot )$ and choose 1/5 of the training set for the Peng et al. (2017) dataset while the full training set for other datasets to generate the full dependency forests. Then, we set $K=20$ to construct causal explanation datasets.

For evaluation, we follow previous studies to use the test accuracy averaged over five cross validation folds for the cross-sentence n-ary task and F1 scores for the sentence-level task. Refer to the supplementary files for the details.

See Appendix A.1 for the hyper-parameter experiment on $N$, $\alpha$, and $\beta$.

For the cross-sentence n-ary relation extraction task, We compare CP-GCN against two kinds of models and report the average test accuracies on the Peng et al. (2017) dataset in table 2.

Tree: models use the 1-best dependency tree. DAG LSTM, GRN, and GCN(Full) use the full dependency tree directly, while GCN(Pruned) generates a pruned dependency tree with some rules Zhang et al. (2018). Besides, DAG LSTM uses graph-structure LSTM to encode the dependency tree, while GRN and GCN use graph recurrent networks and graph convolutional networks, respectively.

Forest: models construct dependency forests. ACGCN treats a fully connected graph obtained by multi-head attention as a forest. LF-GCN automatically generates latent forests using multi-head attention and MMT. AC-GCN generates dependency forests with multi-head attention and encodes them with a 2D convolutional network.

As shown in Table 2, our proposed CP-GCN model achieves state-of-the-art performance in most settings. Specifically, the model using the pruned dependency tree performs better than those using the full dependency tree, suggesting that noisy information does exist in the 1-best dependency tree. In addition, the forest structure shows an advantage on this task, while CP-GCN surpasses the current state-of-the-art forest-structured model (AC-GCN) by 0.7 and 0.3 points on the binary-class ternary relation extraction task. The multi-class n-ary relation extraction task in Peng et al. (2017) dataset is more challenging due to the unbalanced distribution of each relation, and CP-GCN can consistently achieve comparable performance.

For the sentence-level relation extraction task, we implement our approach on the CPR and PGR datasets and compare it against state-of-the-art models. We classify these models into three groups according to their syntax type.

None: models do not use tree or forest structures. Att-GRU adds a self-attention layer to GRU, and Bran uses a bi-affine self-attention model to capture interactions in sentences. BioBERT is a biomedical pre-trained language representation model.

Tree: models use the 1-best dependency tree. GCN, Tree-DDCNN, and Tree-GRN encode the full tree with GCN, DDCNN, and GRN, respectively. BO-LSTM prunes the tree, retaining only the shortest dependency path.

Forest: models construct dependency forests. Edgewise-GRN chooses edges with weights greater than the pre-defined threshold to form the dependency forest. KBest-GRN constructs the forest by aggregating K-best trees. ForestFT-DDCNN generates forests with a learnable dependency parser.

The results of CPR and PGR datasets are shown in Table 3 and Table 4. CP-GCN achieves state-of-the-art performance on both datasets. F1 score increases by 1.5 and 0.5 on the CPR and PGR datasets, respectively. Compared to models with forest structure, CP-GCN performs significantly better than both models with a bias towards syntactic information (Edgewise-GRN, KBest-GRN, and ForestFT-DDCNN) and models using almost only semantic information (AGGCN, LF-GAN, and AC-GCN), which demonstrates the effectiveness of our proposed CSFG method.

Ablation study. To validate the effectiveness of the ingredients of CP-GCN, i.e., the semantic embedding module and the task-specific causal pruning module, we conduct the ablation study on CPR. We train the complete CP-GCN, an ablation model without the semantic embedding module, and another ablation model without the task-specific causal pruning module, respectively. Our experimental results are reported in Table 5. We observe that the performance of the model dropped (compared with complete CP-GCN) regardless of which module is removed, suggesting that both modules can help construct dependency forests that are more conducive to predicting relation. Comparing these two modules, the removal of the task-specific causal pruning module has a greater impact on performance, which suggests that the proposed causal pruning method can effectively distinguish vital information from noise.

Performance against sentence length. Figure 4 compares the F1 scores of our CP-GCN model and the LF-GCN model Guo et al. (2021) under different sentence lengths. Following Guo et al. (2021), We divide the test set of CPR into three groups ((0,25], (25,50], >50) based on sentence length. In general, CP-GCN outperforms LF-GCN against various sentence lengths. Otherwise, our model achieves a significant improvement on the more challenging long sentences, which demonstrates the ability of our model to capture long-range dependencies. Moreover, the dependency forests of the long sentences are more sophisticated, thus indicating that the task-specific causal explainer is able to extract task-relevant information from the sophisticated graph structure.

Case study. To further validate the efficiency of CP-GCN, we conduct a case study on an example sentence “Aspirin induced autophagy, a feature of mTOR inhibition”, which can be correctly predicted by our model to be a “down regulator” relation between Aspirin($E_1$) and mTOR($E_2$). Figure 5(a) shows its 1-best dependency tree, and Figure 5(b) visualizes the top 10 edges with the highest causal weights in the pruned dependency forest generated by the proposed CSFG and the thicker lines referring to higher causal weights. In this example, the connection between “induced” and “feature” enhances in the pruned dependency forest, and we reckon the latent reason is that CP-GCN can capture richer semantic information. We observe that there exists a strong connection between “autophagy” and “feature” in the pruned dependency forest, which improves the prediction of the relation between Aspirin and mTOR, supporting the effectiveness of the task-specific causal explainer.