Enhancing Semantic Understanding with Self-supervised Methods
for Abstractive Dialogue Summarization

In recent years, self-supervised learning has pushed the performance of a wide range of natural language processing (NLP) tasks to new state-of-the-art and becoming a dominant paradigm in NLP. Numerous pre-training methods based on self-supervised learning for text summarization have been introduced. A common approach lets a model predict the original input tokens from randomly masked tokens (or sentences) in a document to resemble a target downstream task, i.e., abstractive summarization [lewis-etal-2020-bart, pmlr-v119-zhang20ae, song2019mass]. Zhang et al. [zhang-etal-2019-hibert] adopt masked sentence prediction in pre-training stage and employed it as a sentence encoder for extractive summarization on large-scale news corpus. The most similar approach to ours is Wang et al. [wang-etal-2019-self], where they introduced three self-supervised pre-training tasks for extractive summarization with CNN/DM datasets [see-etal-2017-get]. Compared to previous work, our approach focuses more on how to incorporate the heterogeneous attributes of a dialogue to self-supervised methods in order to overcome the challenge that fine-tuning is often unstable on small datasets and causes performance degradation [rothe-etal-2020-leveraging].

Liu & Lapata [liu-lapata-2019-text] has shown that BERT can beneficially be applied to both extractive and abstractive document summarization. For abstractive task, their model consists of a BERT pre-trained on extractive summarization task as the encoder and randomly initialized the 6-layer Transformer blocks for the decoder. On the contrary, we aim to leverage the full power of the proposed self-supervised methods by employing BERT as both an encoder and a decoder, which is pre-trained on the task destined to enhance semantic understanding of a dialogue. In recent empirical studies by Google Research [rothe-etal-2020-leveraging, goodman2019multistage], it is possible to achieve state-of-the art results on text summarization without any auxiliary task with the encoder-decoder network utilizing pre-trained BERT, RoBERTa, GPT-2 (so-called bert2bert, bert2gpt2). Our implementation of bert2bert architecture, however, could not have reached higher performance than baselines because the dataset of our choice is much smaller (about 10 times smaller) than previous work. This means that bert2bert architecture has a great potential for a nice warm-starting model but may not be sufficient for low-resource datasets to achieve a better performance.

We have chosen a dialogue dataset of messenger-like natural conversations that have very distinct features from formally-written styled documents like a news article. Most salient pieces of conversations are scattered across the utterances by multiple interlocutors, and it makes difficult to decide what the key point of a dialogue is. Moreover, There are no large enough annotated datasets for abstractive dialogue summarization to train deep neural generative models. To address these problems, Ganesh and Dingliwal [ganesh2020restructuring] propose a two-phase pipeline method that uses discourse labels and an existing document summarizer in the zero-shot learning perspectives. Feng et al. [feng2020incorporating] propose the first to incorporate commonsense knowledge into abstractive dialogue summarization with a graph neural net that includes both utterance and knowledge nodes. Another approach using the graph structure for dialogue summarization is in Zhao et al. [zhao-etal-2020-improving]. They have tackled the problem of previous sequence-to-sequence models [Bahdanau2015NeuralMT] about not paying an attention to handle the sentence-level long-distance dependency and capture the cross-sentence relations by proposing a method that can construct the whole dialogue as a graph for abastractive dialogue summarization. These two approaches constitute the most recent work on the SAMSum corpus. In this work, we choose them as the baselines to compare our model’s score with.

We evaluate our approach on the SAMSum corpus [gliwa-etal-2019-samsum] that is constructed by linguists fluent in English, which contains over 16k chat dialogues consists of over 182K utterances (see Table 1 for statistics on dataset). Each utterance has the specified format that a colon at the beginning of an utterance separates the interlocutor’s name and their speaking content. And also, we append 311 facial emojis into our vocab that appear most often in the train dataset in order to avoid including too many [unk] in the input sequence. The effect of appended tokens is on Table 2.

All our experiment start from publicly available ‘bert-base-uncased’ version. For self-supervised tasks, we preprocess the dialogue datasets for our experiments following steps: a) add [SEP] token at every end of utterance which improves structural information of dialogue [zhao-etal-2020-improving] and is used as the utterance representation in our self-supervised methods, b) replace each name with a single token included in our vocabulary to simplify the problem so that model can learn the semantic relations between the subject and action ("Who did What"), rather than distribution of name tokens. Then, we add a linear layer with dropout (0.1) on top of BERT, where [SEP] and [MASK] tokens are fed into for classification task. Throughout abstractive summarization task, we use the shared encoder-decoder architecture (so called bertShare  in [rothe-etal-2020-leveraging]) where both model’s weights are shared and initialized with BERT trained by one of proposed self-supervised methods. Note that weight sharing between encoder and decoder is necessary rather than optional in that it reduces model parameters and improves the final performance as well in our experimental setup.

The hyper-parameters of self-supervised learning and summarization are almost identical except training epochs. We use the AdamW optimizer [adamw] for both self-supervised and summarization task, with batch size 128, input sequence length 512 and learning rate from 2e-5 to 5e-5, warmup steps are 500. For self-supervised learning, we train the model until the train loss converged (upper bounded by 5K steps). After that, we fine-tune the model for summarization task until the validation loss converges.

ROUGE [lin-2004-rouge] is one of standard measures to evaluate machine generated text over many natural language processing fields. However, the metric based on only n-gram overlapping may not be the best choice for abstractive dialogue summarization [gliwa-etal-2019-samsum]. Such measure is lacking aspects of fluency, intelligibility, and repetition [Savelieva2020AbstractiveSO]. In order to address this issue, we also report consine-similarity between model’s prediction (i.e. generated summary) and ground truth using the sentence encoder [reimers-2019-sentence-bert], ‘stsb-roberta-large (available at https://www.sbert.net/)’ fine-tuned on Semantic Text Similarity dataset. More precisely, we use only the longest 100 samples for calculating cosine-similarity in order to uncover the differences between good and poor quality summaries more clearly.

Table 2 shows the result of the proposed self-supervised learning methods on SAMSum dataset. The upper part of Table 2 is the scores reported in each paper or [gliwa-etal-2019-samsum]. We use the official PyRouge package (https://pypi.org/project/py-rouge) to compute the ROUGE score. In contrast to bertShare ’s promising results [rothe-etal-2020-leveraging] on a large corpus of news like CNN/DM, it shows the relatively lower performance on our dataset compared to recent strong baselines. However, all combinations of bertShare and our methods improve performance dramatically by up to 6.08% (Switching Utterance) in averaged ROUGE score.

On the other hand, Masking Interlocutor methods shows the poor result compared to other methods. This is because the generative training objective that predicts masked tokens from thousands of candidates usually requires much longer training steps with a larger corpus than other binary decision task (i.e. switched or not), and we will investigate proper training setup in future work.

In addition, we confirm that the inclusion of facial emojis in the vocabulary is advantageous for abstractive summarization task, even though they do not appear in the reference summary at all. This is also evidence that if the encoder is enhanced and is employed as decoder too, it leads to better performance on sequence generation task.