Cross-lingual Transfer Learning for Fake News Detector in a Low-Resource Language

Until now, research on FND has exploited three major factors: knowledge, context, and style potthast2018stylometric. Approaches that exploit knowledge ciampaglia2015computational detect information inconsistencies between web documents and a constructed knowledge base of news articles; verification of claims by reference to knowledge can distinguish credibility clearly, but the knowledge bases must be updated in real time, and this task is expensive. Approaches that exploit context ma2016detecting, ma2017detect, ma2018rumor analyze social network responses about a news article, especially the pattern in which news is spread through social media; these approaches show the highest accuracy in the FND task, but the social response takes time, so the detection of FN can be delayed. Approaches that exploit style popat2018declare, wang2017liar, sitaula2020credibility attempt to encode the writing style or appearance of words in FN as a verification feature; this approach has many advantages when applied to the latest news, and obtains quick response; therefore, in this paper, our approach focuses on style exploitation.

When the source of an article is credible, the article is most likely credible. From the knowledge, some FND systems popat2018declare, wang2017liar, sitaula2020credibility exploit the source information to detect FN. They commonly exploited it by encoding information about the authors or publishers; i.e., affiliation of the author, credibility history of the publisher, or the number of authors in the article. They taught us that source information is useful, but these approaches require human labor, including extracting sources from the articles wang2017liar or encoding the information by applying human-generated labels popat2018declare, sitaula2020credibility. We suggested a fully automatic approach including extracting and encoding source information.

Several FND datasets are available in low-resource languages cruz-etal-2020-localization, gereme2021combating, hossain2020banfakenews, amjad2020bend. The datasets are commonly credibility-labeled by considering crawled news articles from web pages or Facebook posts. However, the labeling principles used to label the datasets are not precise or transparent. For example, some methods assume that news articles from non-credible sources are all fake hossain2020banfakenews, amjad2020bend, whereas those from credible sources are all real cruz-etal-2020-localization, but this absolute dichotomy is not reliably true. Another method gereme2021combating uses hired journalists to label the news, but the reasons used for labeling were not provided. In this study, our dataset includes news articles that have been verified by experts from public fact-check services like Politifact²²2https://www.politifact.com/ and SNUFactCheck³³3https://factcheck.snu.ac.kr/, which provide the reasoning.

Language transfer is currently one of the most-intensively studies research areas in NLP huang2019cross, zhou2019dual, zhou2018massively, kocmi2018trivial. However, existing studies mostly consider tasks that are very distinct from the FND task; examples include named-entity recognition, machine translation, and dialog systems. Among the whole tasks, the task that is most similar to FND is sentiment analysis. ADAN chen2018adversarial, a cross-lingual sentiment analyzer, uses adversarial training to transfer the knowledge of sentiment analysis from English to Chinese. ADAN does not require labeled data in the target language as training data, so we adopted the basic idea in our system. Our system is a cross-lingual FN detector that transfers the knowledge of FND from English to Korean. Especially, our system encodes source information of articles to exploit it as a cross-lingual feature and the encoding methods are specialized for FND. To the best of our knowledge, this is the first work that applies language transfer to FND without using target-language training data.

We generated a labeled Korean dataset (Table 1) by crawling the SNUFactCheck web service, which provides FN verification and reasoning. SNUFactCheck provides six types of verification information: Speaker, Claim, Label, Reasoning, Sources, and Claim Source. A Speaker is a person or organization who makes claims. A Claim is the main idea that the speaker asserts. A Label is the verification of the claim, as provided by a journalist, who is an expert in the service. Reasoning is a supporting article that verifies the label provided by the expert. Sources are reference articles, either news or other materials, which may include reasoning. A Claim Source is a news article that includes the claim. Our labeled Korean dataset is constructed by crawling claim sources and the label that belongs to each article. To decrease the number of credibility classes from five (true, mostly true, half-true, mostly false, and false) to two (real, fake), we regarded the first three as ’real’, and the last two as ’fake’. Our constructed labeled Korean dataset includes 528 news articles from March 2017 to May 2020. We undersampled FN data to balance our dataset to 50:50.

To generate the labeled English dataset (Table 1), we crawled the PolitiFact site, which provides U.S. FN verification and reasoning. PolitiFact provides five types of verification information, similar to the types provided in SNUFactCheck, except for the claim source. PolitiFact does not include a claim source, so we should choose the most claim-related news article from sources by calculating the similarity between the claim and news articles. To calculate the similarity between claims and news articles, we computed the term frequency-inverse document frequency (TF-IDF) vector similarity between them. For the English data, we also reduced six class labels (true, mostly true, half-true, mostly-false, false, and pants-on-fire) to two class labels by grouping the first three classes as ’real’, and the last three as ’fake’. The labeled English dataset contains 12,547 news articles published from October 2009 to May 2020. We maintained a real-to-fake ratio of 50:50 for our training data.

Language transfer using adversarial learning requires a large set of data in a target language. To meet this requirement, we crawled 24,000 Korean news articles from a Korean news platform, Naver news⁴⁴4https://news.naver.com/, that were published in the same period as the labeled English dataset. To minimize cultural differences between unlabeled Korean data and labeled English data, the unlabeled Korean data included international news only.

Our system basically follows the architecture of ADAN chen2018adversarial, which is a bilingual sentiment analyzer. The key differences between ADAN and our system are that our system includes a source encoder, a bilingual word embedding, and the output of news encoder.

The encoder includes two major components: article encoder $A$ and source encoder $S$. $A$ generates an article-embedding vector $a$ from an article representation matrix $v$ which is transformed from an input article $x$ by using bilingual word embedding (BWE). We generated our own BWE by aligning pre-trained 100-dimensional Korean⁶⁶6https://ratsgo.github.io/embedding/ and English GloVe embeddings pennington2014glove by applying Procrustes grave2019unsupervised alignment. The alignment used a bilingual word dictionary given in lample2018word. Our $A$ uses a convolutional neural network (CNN), which is commonly used to encode documents kim-2014-convolutional, conneau2017very.

$S$ encodes the source information that is extracted from news articles. In this study, we exploited $S$ as a cross-lingual feature to assist the knowledge transfer by encoding the source information to its credibility. In general, source information includes press companies, authors, and speakers who make claims in news articles popat2018declare, wang2017liar, sitaula2020credibility. We defined our source information as speakers who make claims in articles, because it is the most content-related information among these options. To fulfill the objective, the system generates source data by extracting speaker candidates from an input article, then classifies the candidates according to their credibility.

An output of the news encoder from $x$ is a news-embedding vector $e$, which is generated by concatenating $a$ and $s$. The FN detector $N$ and the language discriminator $D$ make final decisions by considering $e$. $D$ discriminates the language of the article, and $N$ detects whether the article is fake or not. Both modules are feed-forward networks, but only $N$ has a softmax layer at the end because the output value of $D$ is assumed as Wasserstein distance between the distributions of $e$ in both languages.

To exploit source information, the source extractor $E$ generates source data $z$ from $x$. $z$ is composed of speakers who have made a claim in news articles, and is obtained by extracting speaker candidates from $x$. Speaker candidates are extracted from $x$ by recognizing PERSON-typed named entities, by using the SpaCy⁷⁷7https://spacy.io/ python library for English and named-entity extractor for Korean⁸⁸8https://github.com/eagle705/pytorch-bert-crf-ner. The extractor generates source data by arranging the candidates in appearance order.

Our extracting strategy uses analysis of speakers in news articles. First, we identified the speakers: from the page that provides a whole list of speakers in PolitiFact⁹⁹9https://www.politifact.com/personalities/, we crawled and analyzed all of their labels. They show the identity of each speaker, including political parties and jobs. We found that speakers are mostly people, especially politicians (Figure 2). In detail, the ‘None’ class is large, so we analyzed 100 random samples from it; 52 were people and 48 were organizations.

Second, to identify where the speakers appear in the news articles, we used a percentage to represent the location of the index of speakers relative to the beginning and end of the article. Most speakers were identified near the beginning of news articles (Figure 3).

These analyses yielded two major insights. First, we can collect speaker candidates by extracting the names of people and organizations from news articles. Second, the sequence of speaker candidates is informative because a candidate that is mentioned early is usually the speaker. From the additional analysis (Section 5.2), we determined that identifying PERSON-typed named entities in appearance order is the best way to generate the source data.

To encode $z$ as a cross-lingual feature for FND, a word embedding that represents speakers’ credibility is required. In addition, the word embedding must be bilingual to exploit source data as a cross-lingual feature. To fulfill those requirements, we generated a new bilingual word embedding, bilingual source embedding (BSE).

To represent the credibility of speakers when they make claims, we generated BSE from news articles published in the same period as the labeled dataset. For Korean, we generated embedding space by analyzing 33,000 crawled domestic Korean news articles that were published in the same period as the labeled Korean dataset. For English, we used pre-existing labeled data. Our idea to generate BSE applies the intuition that speakers who are not credible will produce fake news more frequently than real news. After generating two source embeddings in each language, we convert them to a bi-lingual embedding by applying Procrustes grave2019unsupervised alignment.

To verify whether the source embedding represents credibility, we plotted the top twenty most frequently appearing speakers of our labeled dataset in both languages in a two-dimensional space (Figure 4), then colored each speaker according to the credibility score that was calculated from our labeled dataset. Calculation of credibility score $\widehat{s_i}$ for speaker $i$ is

where $c_i$ represents the number of articles in which speaker $i$ appeared with score $b_{ij}$ for each article $j$. BSE space (Figure 4b) shows clusters divided according to credibility, whereas BWE space (Figure 4a) assembles speakers that have different credibility scores. For example, Figure 4a reveals clusters of speakers that have different credibility scores around (-2,-1), (1,-2) coordinates, but these clusters are not apparent in Figure 4b.

As a quantitative measure, we calculated credibility score difference $dif{f}_{cred}$ between nearby speakers. For $m$ speakers with $n$ nearest neighbors, $dif{f}_{cred}$ is calculated as

where $N_{ij}$ represents the $j$th nearest neighbor of speaker $i$. Our calculation used $m$ = 40 and $n$ = 5. BWE achieved $dif{f}_{cred}$ = 0.4171, and BSE achieved $dif{f}_{cred}$ = 0.2789. As we expected, BSE got lower speaker distance than BWE.

After the BSE generates an embedding matrix $u$, $S$ encodes it to a vector $s$ that records all source information of news article $x$. $S$ encodes $u$ on bi-directional LSTM (bi-LSTM) to encode sequential information together. The average of bi-LSTM nodes becomes $s$, which was the best encoding option from Section 5.2.

Our system also follows training procedure of a Wasserstein generative adversarial network (WGAN) martin2017wasserstein, as ADAN did. To exploit source information for FND, the objective functions are modified. Especially, the language discriminator uses

the FN detector uses

the article encoder uses

and the source encoder uses

where ${\theta }_d$ are parameters of language discriminator $D$, ${\theta }_n$ are parameters of FN detector $N$, ${\theta }_a$ are parameters of article encoder $A$, and ${\theta }_s$ are parameters of source encoder $S$. From dataset $X$, $x$ represents sampled data, $y$ includes labels of $x$, $a$ represents encoded vector from article encoder, and $s$ is the encoded vector from source encoder. As in ADAN, we assumed Eq. (1) as Wasserstein distance between distributions of source and target language data. As we assumed, the neural net tries to increase $J_d$ for English articles and to decrease it for Korean articles. $J_n$ minimizes cross-entropy loss $L_n$ between the FND result and the label $y$. Finally, $A$ and $S$ minimize both objectives, $J_d$ and $J_n$, to discourage $D$ and encourage $N$ simultaneously. In this work, we set hyperparameters $\lambda$ = 0.01, $c$ = 0.01 and $k$ = 5.

Hyperparameters of our system are either pre-existing or newly-added. Values of the newly-added parameters were selected by considering pre-defined parameters as given below. $A$ has 3-, 4- and 5-sized 400 kernels for each size and $a$ is a 900-dimensional vector. $S$ encodes $s$ as a 900-dimensional vector by applying BSE with bi-LSTM. After applying source information, $N$ and $D$ got an 1,800-dimensional vector as input. $N$ and $D$ are binary classifiers that use fully-connected networks with two hidden layers, including 1,800 hidden units, and only $N$ includes ReLUs to resolve non-linearities. For training, we applied Adam kingma2015adam optimizer with 0.0005 learning rate for each module.

We trained our system for 30 epochs on the validation set of a labeled Korean dataset and used early stopping to select the best model.

Our training process uses Nvidia GeForce TitanX, which has GDDR 5 12GB memory and a 1,000-Mhz GPU base clock. The whole training process took 20 hours.

In experiments, we compared our system with three baselines: ADAN, a system that uses machine translation (SMT), and text-CNN kim-2014-convolutional (Table 2). By comparing with ADAN, we can verify that exploitation of source information is crucial for cross-lingual FND. SMT uses machine translation without language transfer; this system includes $A$ and $N$ only, and is trained on a labeled English dataset and tested on a labeled Korean dataset that had been translated to English by Google translate¹⁰¹⁰10https://cloud.google.com/translate. Text-CNN is an effective text encoder for FND wang2018eann. No labeled training dataset is available for Korean, so we measured Text-CNN accuracy by five-fold cross-validation of a labeled Korean dataset. We also added RANDOM accuracy that randomly selects FN from input articles for comparison at the end. Our system achieved the highest accuracy, especially the highest F1 score among the methods. All given experimental results are averages of five trials with a 200-word length limitation for calculation efficiency.

Results (Table 2) demonstrated that a cross-lingual approach that exploits source information is effective for FND in a low-resource language. In detail, compared with ADAN (Table 2, row 2), our system is well-improved for FND by exploiting source information as a cross-lingual feature. SMT (Table 2, row 3) got similar accuracy to ADAN, but with very low recall; i.e., the detection result of the system is biased toward real news. We conclude the reason that it is not trained well because the quality of machine-translated articles is insufficient. In the case of Text-CNN (Table 2, row 4), the FND accuracy with a small amount of training data in the target language was similar to RANDOM (Table 2, row 5). We conclude that language transfer for FND in a low-resource language is essential.

In additional experiments, we analyzed the effect of components that exploit source information (Table 3). The analyzed components include source data, embedding, and encoder.

In Figure 2, most of the speakers are PERSON-type or ORGANIZATION-type named entities. To determine which type of named entity is essential, our experiment compared various source data by combining named entity types. We determined that PERSON is the most critical feature for transfer learning, whereas ORGANIZATION decreases the accuracy of FND (Table 3, row 1). We hypothesize that the accuracy is decreased by including ORGANIZATION type because organizations are so numerous that they do not provide reliable clues to detect FN. From the whole analysis, our final source data were generated by extracting PERSON-type named entities from a news article. We also checked whether the sequential information of source data is necessary, by shuffling speaker candidates to remove sequential information. This experiment demonstrated that to detect FN, sequential information of source data is necessary (Table 3, row 3).

To check the effect of BSE, we compared FND accuracy with source data encoded using BWE. Applying BSE got higher accuracy than BWE (Table 3, row 4). Interestingly, even BWE got relatively high accuracy; we suppose that this result occurs because some non-credible speakers who use social networking services are already separated from credible speakers in BWE space.

Finally, to find the best method of encoding the data, we compared CNN (Table 3, row 5), and other LSTM encoding methods: last node (Table 3, row 6), first node (Table 3, row 7), and attention-based luong2015effective (Table 3, row 8). Our proposed system that averages hidden nodes of bi-LSTM, was more accurate than other encoding methods, and CNN was the least accurate.