Application of Data Encryption in Chinese Named Entity Recognition

Named entity recognition aims to quickly extract entity information of specific types from complex natural language texts, which provides the foundation for information extraction to generate structured data. The earliest methods of NER are rule-based methods, lexicon-based methods and statistical machine learning-based methods, such as the Support Vector Machine Model (SVM) [13], Hidden Markov Model (HMM) [50], Conditional Random Field (CRF) [38], and so on. However, these methods suffer from feature engineering and fail to achieve the desired results for NER.

With the development of artificial intelligence, numerous neural network methods have been applied and achieved good results for named entity recognition. At that time, researchers mainly used LSTM-CRF or CNN-CRF models to encode and decode the input features [17, 21, 30, 4, 40, 46, 47]. But there are some issues with these methods. 1) The character-based methods do not use dictionary information. 2) The word-based method will cause error propagation due to word segmentation errors.

Accordingly, Zhang and Yang proposed a character-word-based hybrid model in 2018. This model can integrate word information into the character of the input sequence, which further improves the performance of the NER task. Therefore, many researchers have derived a large number of models based on the approach . For example, [27] proposed the WC-LSTM model, [11] proposed the LR-CNN model, and [29] proposed the Softword method. [7, 41, 12] used graph neural networks to improve NER performance. It is well known that NER tasks rely heavily on word embeddings. The quality of word embeddings can determine the performance of the model. The emergence of dynamic word embeddings such as BERT pushed the performance of NER to a new level. For instance, [25] and [32] proposed a FLAT model and a Porous Lattice Transformer Encoder based on Lattice LSTM, respectively.

Named entity recognition introduces data protection [37, 9, 33] to protect the private information contained in a dataset from being widely read and leaked. However, data leakage is the intentional or unintentional disclosure or loss of data to untrustworthy third parties. This paper focuses on protecting private data using data desensitization and access control strategies to prevent data leakage. Data desensitization is a crucial method to remove the sensitivity of personal privacy data and minimize the risk of data leakage. It is mainly used to process the original data utilizing data replacement, data randomization, data encryption, and hash transformation. We performed a hash transformation on the NER dataset using data desensitization. Second, access control generally means that the data provider sets up security rules or policies. Users can obtain and decrypt the encrypted data according to their permissions or attributes.

Hash functions are used to encrypt data in our model framework in Figure 1. To verify the authenticity and validity of encrypted data training and encrypt the same data using more than one encryption method, we also introduce Serial Cipher and Base64 methods besides hash functions.

Serial Cipher [39], also called Stream Cipher, is one of the symmetric cryptographic algorithms. It encrypts the plaintext by adding the key. Since serial encryption has the advantages of simple implementation and fast encryption speed, we take it as an optional method for data encryption. Meanwhile, the Caesar Cipher and Affine Cipher can be consistent with the Serial Cipher by adjusting the coefficients. We grouped the three encryption algorithms into one category.

As we all know, Base64 [18] is the most common encoding method for transmission in the network, which is used to transmit 8-bit byte codes. Although it violates the principle of non-disclosure of encryption keys, Base64 encoding can process text data due to its unreadable advantages. Therefore, the method provides a way for verifying the validity of the encryption method and performing multiple encryptions.

MD5 message-digest algorithm and Has256 algorithm [36] are hash functions widely used in computer security. They provide the ability to transform data with arbitrary length to a fixed value. At the same time, the MD5 algorithm and the Has256 algorithm are irreversible, which can effectively ensure the security of the dataset. In the experiment, to realize the feasibility of multi-encryption of the same data, we also use the Base64 algorithm on top of the Has256 algorithm for encryption.

Traditional attribute-based encryption (ABE) [10] systems describe the ciphertext by attributes and embed the policy into the user’s key. Where attributes are characteristics of things or information files, and policy is a logical expression composed of attributes and relationships. CP-ABE [1, 43] uses attributes to portray the user’s eligibility. The data provider makes the ciphertext acquisition policy to decide who can decrypt the ciphertext. In other words, the attributes are embedded in the key, and the policy is embedded in the ciphertext.

As shown in the upper right of Figure 1, firstly, the data provider initializes a public key (PK) and a master key (MK). Secondly, the public key, the master key, and the user’s attribute set (Au) generate a private key (SK). Then, the data provider constructs an access control policy (Ac-cp) based on the user’s attributes. The public key, the plaintext M ( M indicates the data encrypted with the hash algorithm in the figure), can generate the ciphertext (C) through the access control policy. Finally, the user could decrypt the data based on the public and private keys. If the user attribute is legitimate, the ciphertext can be decoded successfully; otherwise, it cannot.

As shown in the bottom right of Figure 1, we describe how the user trains the model and predicts with the model on the encrypted dataset. First, the legitimate user follows the CP-ABE rule and decodes the data to obtain three data files using the ciphertext, public key, and private key provided by the data provider. Text1 represents the ciphertext encrypted with the hash algorithm, Text2 represents the label text corresponding to the ciphertext, and Text3 is the length of each sequence in the ciphertext. According to the three texts, users can get the training data completely. If users need to test the model’s performance with their data, users could encrypt the data using the same hashing algorithm in Text1 before prediction.

With the continuous exploration of researchers, neural networks have developed rapidly. Li et al. significantly improved the performance of biomedical named entity recognition by using recurrent neural networks [24]. Meanwhile, applying deep neural networks such as convolutional neural networks, self-attention mechanisms, and transformers [42, 2] has effectively promoted the development of NER. Therefore, the user can build models based on the above deep neural networks after obtaining encrypted data. Here, we use the classical model BiLSTM-CRF as a benchmark. BiLSTM comprises a forget gate, an input gate, and an output gate. These three gate mechanisms interact with each other and update the cell state. The specific formulas are as follows.

Here, $F^T$ represents the information that the cell state will forget. $I^T$ and $O^T$ denote the input and output gates, respectively. ${\tilde{C}}^T$ denotes the current cell state and $C^T$ denotes the final cell state. Where $W$ is the hyperparameter and $H$ denotes the output of the hidden state.

We use a conditional random field [8, 20, 26] to constrain the transfer between tags after the encoding layer. As shown in Figure 2, B-L denotes the start of a location, I-L denotes the middle of a location, and E-L denotes the end of a location. S-L indicates that a single entity constitutes a location, and O is a non-entity. According to the figure, we perform the following constraints. O is impossible to transfer into I-L and E-L. B-L is impossible to transfer into B-L, S-L, and O. I-L is impossible to transfer into B-L, S-L, and O. E-L is impossible to transfer into E-L and I-L.

In this section, we use four models proposed in recent years to verify the effectiveness of encryption algorithms.

BiLSTM-CRF. BiLSTM-CRF was proposed by Lample et al. in 2016. The method is a classical model in named entity recognition. Compared with the traditional machine learning models, it shows a dramatic enhancement in performance.

WC-LSTM. WC-LSTM (2019) is a word-character-based model proposed by Liu et al. for addressing the shortcomings of Lattice LSTM. The model provides four methods that effectively integrate lexicon knowledge into characters.

Multi-digraph Model. Multi-digraph (2019) is a model proposed by modifying the gated graph neural network (GGNN), which can effectively integrate word information into characters.

SoftLexicon. SoftLexicon (2020) is a novel approach to utilizing dictionary information proposed by Ma et al. Its encoding framework is very flexible and can enormously improve the performance of entity recognition.

Table 3 shows the results on three public datasets, including CCKS2017, Resume, and MSRA. The results of four different models are given in Table3. For each model, we do five experiments. In the first experiment, we train the model and predict with the model using plaintext. In the other four experiments, we train the model and predict with the model using ciphertext, encrypted by Serical Cipher, Base64, MD5 and Has256-Base64 respectively. Has256-Base64 means that the data encrypted first by Has256 and then by Base64. For CCKS2017 and Resume, the experimental results of the four encryption methods show barely any change in performance on the four baseline models when comparing the plaintext training method with the ciphertext training methods. some results obtained with the ciphertext training methods are even better than the plaintext training method. For the MSRA dataset, when training Has256-Base64 encrypted data using Multi-digraph, Precision and F1 value are 0.56% and 0.11% higher than the plaintext training method. When training LSTM-CRF, there is almost no degradation in the performance. When training WC-LSTM and SoftLexicon, the experimental performance degradation may be caused by the under-utilization of lexical knowledge by the encryption methods and the effect of the absence of development sets.

Table 4 shows the results of the historical dataset. As shown in the table, compared to unencrypted, the performance of the four encryption algorithms barely degrades when trained with the LSTM-CRF model. Moreover, some encryption algorithms have better performance than unencrypted ones. For example, Precision is 0.87% and 0.78% higher than unencrypted by using the MD5 encryption algorithm on the History-9types and History-2types datasets. Precision is improved by 0.79% using serial encryption algorithms, and Recall is improved by 1.78% using MD5 encryption algorithms on the History-3types dataset. When the WC-LSTM model is trained using Serial Cipher encrypted data, Recall is 0.89% higher than the unencrypted dataset on the History-3types dataset. Precision, Recall, and F1 value are 2.28%, 0.55%, and 1.2% higher than the unencrypted on the History-2types dataset. When trained with the Multi-digraph model, Precision and F1 value are 3.11% and 1.27% higher than unencrypted when using the Base64 algorithm on History-9types. When trained with the SoftLexicon model, we find the effectiveness of the encryption algorithm drops significantly. Because the SoftLexicon model breaks the convention of using vocabulary knowledge, it causes poor performance.

We train a NER model using the Chinese historical dataset to improve the performance of its downstream tasks, which can accurately identify entities in unstructured text. Figure 3 shows a simple knowledge graph constructed after we extracted partial entity types from the text by the model.

The red nodes indicate the Chinese dynasties, such as the Qing Dynasty, Ming Dynasty, and Tang Dynasty. The red edges indicate the substitution between dynasties, such as the Qing Dynasty overthrew the Ming Dynasty and the Tang Dynasty overthrew the Sui Dynasty. The yellow nodes express the capital of each dynasty. As a result of capital city migration, a red node may be connected to multiple yellow nodes. A black node describes the country’s leader, and a black edge indicates a father-son or brother relationship. Green nodes are emperors’ aliases.

In Figure 3, we can know most of the relationships between the various dynasties, which fully justifies the necessity of constructing a Chinese historical dataset. It paves the way for future research.