Focus-Driven Contrastive Learning for Medical Question Summarization

The medical question summarization task is defined by Ben Abacha and Demner-Fushman (2019). They construct a benchmark dataset Meqsum, and apply a pointer-generator model to generate question summary. At the question summarization campaign of MEDIQA-21 organized by Ben Abacha et al. (2021), almost all approaches rely on the fine-tuning of pre-trained transformer models. Transfer learning, knowledge-base, and ensemble methods are widely utilized by participanting teams to achieve better performance He et al. (2021); Yadav et al. (2021); Mrini et al. (2021b); Sänger et al. (2021). In this paper, we also base our method on the strong pre-trained BART model.

Recently, Yadav et al. (2021) propose a RL framework with two question-aware semantic rewards: question-type identification reward (QTR) and question-focus recognition reward (QFR). QTR is to identify whether the question types are consistent with the gold question, and QFR is designed to capture question focus. But in their work, the question types and question focuses in the dataset should be manually labeled, which is both time-consuming and labor-intensive for large-scale datasets such as HealthcareMagic and iCliniq. Moreover, the RL training process is unstable. Mrini et al. (2021) claim an equivalence between medical question summary and recognizing question entailment(RQE), and employ multi-task learning to train the model to not only perform next-word-prediction but also carry question entailment recognition. These two studies demonstrate that the pre-trained models achieve better performance after capturing the underlying sentence semantics of generated questions. Different from these works, we exploit contrastive learning to obtain focus-aware question representations.

Different from the traditional methods which learn representations in pixel-level for computer vision tasks, contrastive learning encodes high-level features to distinguish different objects and has achieved great success Henaff (2020); Chen et al. (2020); Misra and van der Maaten (2020); He et al. (2020), and it has also been applied in several NLP tasks such as machine translation Pan et al. (2021), pre-training Chi et al. (2021) and question answering Yang et al. (2021). In the field of summarization, Liu and Liu (2021) present a contrastive framework to bridge the gap between the learning objective and evaluation metrics, Cao and Wang (2021) design several negative sample construction strategies to solve the factual inconsistency problem. In contrast, we use the MoCo structure to handle with the large volume of negative samples, and propose a new negative sample construction method.

Chen et al. (2020) prove that large size of negative samples can improve the performance of contrastive learning, but it also brings heavy burden on computation cost. To address this issue, He et al. (2020) propose MoCo, which maintains a queue as the memory bank to store negative samples. MoCo adopts two encoders with the same structure: key encoder and query encoder, where the key encoder is momentum updated from the query encoder.

We employ the pre-trained BART Lewis et al. (2020) as our basic model to generate question summaries. For contrastive learning, we adopt the MoCo architecure He et al. (2020), which contains a key encoder $E_k$ with the same structure as the BART encoder $E_q$, and a queue to store simple negative samples with large volume. The simple negative samples in the queue are progressively replaced by current mini-batch of representations extracted from the key encoder. All samples in the queue will be used as negative samples in the next batch. In addition, QFCL employs a hard negatives generator to generate hard negative samples.

In our model, the BART encoder $E_q$ and the decoder are updated via back propagation by combining three types of loss functions, as described in the subsequent sections. The parameters of $E_k$ are frozen and updated slowly towards that of $E_q$:

where $m$ is a momentum coefficient.

At the inference, only the BART encoder and decoder are retained, other parts such as the key encoder, the queue, and the hard negatives generator are all discarded.

In the medical question summarization task, the input question CHQ should be semantically close to its reference summary FAQ but different from other question summaries. Therefore, we regard the CHQ $c_i$ in the $i$-th pair as the anchor, FAQ $f_i$ in the same pair as the positive sample and randomly select $f_j$ from other different pairs to serve as simple negative samples.

Let $R_s$ denote the average decoded output of an arbitrary sentence $s$, the objective function of the simple contrastive learning is defined as:

where $R_{ci}$ indicates the sentence representation of the $i$-th CHQ extracted from $E_q$, and $R_{fi}$ and $R_{fj}$ are extracted from the key encoder $E_k$ for the $i$-th and $j$-th FAQ respectively. The operation $sim$ is to calculate the cosine similarity, $\tau$ is a temperature hyper-parameter. $M_s$ is the memory bank which contains one positive sample and $K$ simple negative samples in the queue with respect to an anchor.

The above simple negative samples are randomly selected. As claimed by Kalantidis et al. (2020), hard negative samples that are more similar to positive samples can facilitate the model to get better performance. Inspired by this, we build a bridge between hard sample generation and question focus prediction.

An imbalance existing in the above method is that contrastive learning is only utilized at the encoder. We fine-tuned BART on iCliniq dataset, and found that the decoder lacks the ability to distinguish the representations between the generated summary and the positive samples/unrelated negative samples, as $s_{g\_faq}^{+}$, $s_{g\_sim}^{-}$, $s_{g\_hard}^{-}$ shown in Figure 4. Therefore, we try to improve the similarity between the generated summary and its reference FAQ, and at the same time enlarge the dis-similarity between the generated summary and other unrelated questions.

Specially, we regard the generated summary as an extra anchor, and denote the representation of the generated summary as $g_i$. Since the output summary should be semantically consistent with the corresponding FAQ, we consider the representation of the FAQ $f_i$ in the same pair as the positive sample, and select the simple negative samples randomly from the queue and generate hard negative samples using the hard negatives generator. The object functions of contrast loss $L_{ctrGS}$ and $L_{ctrGH}$ at the decoder end are defined in a similar style as Equation 2 and 3, except that the anchor $c_i$ is replaced by another anchor $g_i$.

For predicting next tokens in the generated summary, we use the cross entropy loss $L_{ce}$:

In our model, the overall loss function consists of five parts: the cross entropy loss $L_{ce}$ and four different loss functions of contrastive learning: $L_{ctrCS}$, $L_{ctrCH}$ for the anchor at the encoder end, $L_{ctrGS}$, $L_{ctrGH}$ for the anchor at the decoder end. We define the contrastive learning loss with respect to these two anchors as:

where $\alpha$, $\beta$ are hyper-parameters to control the balance between simple negatives and hard ones. The weights of contrastive learning loss at the encoder and decoder are considered as equal, and the overall loss is defined as:

We conduct experiments on three English benchmark medical question summarization datasets, including Meqsum, HealthcareMagic and iCliniq. Meqsum is a high-quality dataset from NIH ¹¹1www.nlm.nih.gov/medlineplus, constructed by Ben Abacha and Demner-Fushman (2019). Mrini et al. (2021a) extracted HealthCareMagic and iCliniq datasets from MedDialog Zeng et al. (2020) , which are collected automatically from the online healthcare service platforms ²²2www.healthcaremagic.com ³³3www.icliniq.com. MeqSum’s and HealthcareMagic’s summaries are written by medical experts in formal style, while iCliniq’s are patient-written. We list some statistics of these datasets in table 2. Following previous works, we adopt ROUGE Lin (2004)⁴⁴4https://pypi.org/project/py-rouge as the evaluation metric.

We utilize BART-large Lewis et al. (2020) in huggingface⁵⁵5huggingface.co/facebook/bart-large as our pre-trained model. The learning rate of BART baseline is set to 3e-5 as the same with Mrini et al. (2021). For contrastive learning in QFCL, the learning rate is optimized to 1e-5. Betas of Adam optimizer is set to 0.9 and 0.999. Batch size is set to 16. The number of hard negative samples $n_h$ is set to 64. For Moco, the queue size $K$ is set to 4096, temperature $\tau$ is 0.07, and the momentum coefficient $m$ is 0.999. In Equation 5, $\alpha$ and $\beta$ are set to 1 and 0.5 respectively through grid search on MeqSum development set. Experiments were all performed on a single NVIDIA RTX 3090 GPU. The average runtimes of each epoch for MeqSum, iCliniq and HealthcareMagic are 4.2h, 0.6h and 0.1h respectively.

We report our experimental results in Table 3. Our model achieves new state-of-the-art results on all three datasets. Compared with the previous best results, we obtain an improvement of 0.99 ROUGE-L score on MeqSum, 8.44 on iCliniq, and 0.51 on HealthcareMagic, respectively.

MTL+Data augmentation Mrini et al. (2021) obtains the previous state-of-the-art results on iCliniq and HealthcareMagic, which utilizes the question entailment data to augment summarization data. In contrast, our method doesn’t need other classification models or external data. The work of ProphetNet+QTR+QFR Yadav et al. (2021) gets the previous best result on MeqSum, which presents a reinforcement learning-based framework with question-aware rewards. Comparing with this competitive model, our method obtains consistent better performance on all metrics, with 2.28 improvement on R1, 4.66 improvement on R2 and 0.89 improvement on RL. We did not compare the results of Yadav et al. (2021) on the other two datasets, since their method requires manually labeled question focuses and question types.

We perform ablation study to evaluate the impacts of different components employed in QFCL, and report the results in Table 3. In particular, for Meqsum dataset, due to the small size which may cause the training unstable, we conducted five separate experiments and computed the average ROUGE score of these five checkpoints as the final result. Compared with the base BART model, we obtain an absolute improvement of 5.33 points on average. T-test is implemented on such five ROUGE scores and the p-value is less than 1e-2, validating that this improvement is significant. On Cliniq the absolute improvement is 12.85 points and on HealthcareMagic 3.81 points. In comparison to BART, the relative improvements of our model are $12.2\%$, $28.7\%$ and $9.6\%$ on Meqsum, Cliniq and HealthcareMagic respectively.

The results demonstrate that each component of our model is helpful. On MeqSum, there is an increase of 3.15 points for BART+S compared to the baseline, indicating that the contrastive learning on simple negative samples largely improves model performance. It shows an continuous increase of 0.77 points for BART+S+H, and the highest ROUGE-L score is obtained when three parts are all implemented in our model. It suggests that each component in QFCL contributes positively, and metrics like ROUGE evaluating the similarity between whole sentences benefit from our contrastive learning strategy.

To quantitatively assess the results, we compare our method with the baseline BART through human judgement. We randomly selected 50 samples from each of three datasets, and hired 3 graduate students to categorize each generated summary into one of the following categories: ’Incorrect’, ’Acceptable’, and ’Perfect’. We compute the average number of each category, and report the result in Table 4. The average Spearman correlation coefficient between three annotators is 0.68, which guarantees a high quality of our annotation data. The evaluation results show that our model generates a higher proportion of perfect samples and a lower proportion of incorrect ones, by enhancing the model’s ability of capturing sentence semantics and question focuses.

To clearly show the output question summary, we list two samples to compare our model with BART in Table 5. In Case 1, BART captures the question focus "Ampicillin" but misses "drink alcohol", and in Case 2 it misses the question focus "breast milk". In contrast, our model successfully extract multiple question focuses from the lengthy CHQ, and generate summaries which more conform to the meaning of original questions.

Since the auxiliary structures are discarded at the inference stage, we make further analysis to check that whether the retained model has the ability to distinguish different sentence-level semantics when facing unknown data. We train QFCL and BART on the training set for 20 epochs and save each checkpoint, and evaluate these checkpoints on the development set.

Four types of sentence representations are extracted from these checkpoints: CHQ’s representation $R_c$, FAQ’s representation $R_f$, hard negatives’ representation $R_h$, and the generated summary’s representation at decoder end $R_g$. Then we calculate the cosine similarity between them, and draw the relationship between these similarity scores and the epoch numbers, as shown in Figure 4.

Regarding the anchor CHQ in the curve of iCliniq, $s_{c\_faq}^{+}$, $s_{c\_sim}^{-}$ and $s_{c\_hard}^{-}$ are very close to each other at epoch 0, suggesting that the initial encoder lacks the ability to capture different semantics. With the increase of training steps, $s_{c\_faq}^{+}$ changes smoothly, while $s_{c\_sim}^{-}$ decreases sharply to near zero and $s_{c\_hard}^{-}$ decreases gradually and converges at a middle level between $s_{c\_faq}^{+}$ and $s_{c\_sim}^{-}$. This suggests that, powered by contrastive learning, our model has learned to distinguish sentences of different meanings at the encoder end.

With the generated summary as another anchor, we find out that $s_{g\_faq}^{+}$, $s_{g\_sim}^{-}$, $s_{g\_hard}^{-}$ are all near to 0 initially, which depict that the decoder is also weak in representing sentence-level semantics. After training, $s_{g\_faq}^{+}$ increases significantly, $s_{g\_hard}^{-}$ converges between $s_{g\_faq}^{+}$ and $s_{g\_sim}^{-}$, and $s_{g\_sim}^{-}$ keeps very low all the time. It suggests that the decoder has strengthened its power to distinguish different semantics as the same to the encoder end.

Another chart is drawn to show this relationship for BART baseline in Figure 4. The similarities between the anchor and the positive samples, negative samples are very close, and never improve significantly with the progress of training. This situation suggesting that the BART baseline has a relatively weaker performance to distinguish the sentences of different meanings at both encoder and decode, since it only focuses on the prediction of next tokens.

We also draw this correlation curve on MeqSum and HealthcareMagic. The curve of HealthcareMagic is similar to iCliniq. On MeqSum, our model can still distinguish sentences with different semantics better than the baseline, but the signal is not as significant as iCliniq or HealthcareMagic due to the limited size of training set.

To study whether our model pays more attention to the question focus, we evaluate the accuracy of question focuses in generated summaries. We use the sequence labeling model trained by Yadav et al. (2021) to predict question focuses on the MeqSum dataset, and regard the 812 predicted question focuses in test set as the gold-standard. For QFCL and BART, we train five checkpoints and generate summaries on these checkpoints, and compute the accuracy of question focuses on test set. As shown in Table 6, the average accuracy is 37.09% for BART and 45.44% for QFCL. Our model exceeds the baseline by 8.35 points for question focus generation. P-value of t-test on these two sets of results is 1.04e-3, indicating that this improvement is statistically significant.