Which anonymization technique is best for which NLP task? - It depends.
A Systematic Study on Clinical Text Processing

While clinical text processing has gained more and more attention in recent years, access to data still remains a major challenge as it typically contains sensitive, patient-related information. A straightforward solution is to apply one of the many existing de-identification and anonymization techniques, and control the access to the data, as for instance in Kittner et al. (2021) or Henry et al. (2019). But each technique has different properties and modification of the source text has effects on machine learning. What happens when you train a model on an anonymized corpus and test it on your own local data (not anoymized)? In which way does this affect the performance of your model?

To put it more generally: If each anonymization technique has different characteristics, how do they affect different natural language processing (NLP) tasks? Is there some rule of thumb we can follow when choosing an anonymization technique, e.g., to share data for a specific task? To explore those questions in detail, this work conducts a systematic analysis regarding the influence of different anonymization techniques and their effects on the performance of (state-of-the-art) machine learning (ML) models. In course of this, we train and test the models using six different datasets corresponding to five different natural language processing tasks. Main contributions of this work are a set of learnings and recommendations regarding text anonymization for NLP tasks, as well as a small, fictitious re-identification experiment to explore the (in-)effectiveness of the different techniques. The software implementations of the different anonymization techniques will be made publicly available.

In accordance with the HIPAA Safe Harbor HIPAA (2022) method, we define de-identification as the removal of protected health information (PHI) that directly relate to an individual such as name, address, birth date, etc. However, de-identification does not guarantee anonymity for data subjects. Anonymization on the other hand is defined as any irreversible procedure, in which no information can be linked to any individual Meystre et al. (2010), making the data subjects anonymous and no longer identifiable.

A range of different text anonymization approaches exist in the literature, which modify the text structure within a dataset, delete, replace, or introduce synthetic information, to make it harder to identify or infer factual information on the patient. The following approaches have been explored for this work:

• Suppression Mamede et al. (2016) is a technique that either completely removes certain words or sentences or masking them with a neutral label denoting their suppression.

• Perturbation Zuo et al. (2021) modifies data through permutation or data swapping, in case of text, similarly to data augmentation, by flipping characters, or changing the order of words.

• Substitution Mamede et al. (2016) replaces certain information with more general terms.

• Finally, Aggregation (k-anonymity) Samarati and Sweeney (1998) groups individual data subjects together, e.g. by their attribute values, to make it more difficult to identify a single individual.

Only limited work has been done to describe the (systematic) influence on text anonymization on the performance of ML models, most work targets de-identification only. Meystre et al. (2014) for instance examine evaluates information loss after de-identification by examining the text-level changes rather than the performance of the machine learning models. The work concludes that only 1.2-3% of the clinical concepts are changed by de-identification and that the overall impact on the clinical information is minimal but not negligible.

The work of Obeid et al. (2019) analyzes the impact of de-identification on a binary classification task, and concludes, that there is no significant difference in performance between training on the original texts and training on the de-identified texts. Similarly, Berg et al. (2020) examines the effect of different PHI concealment strategies on named entity recognition (NER) tasks and show that using moderate to high precision de-identification models with the right concealment strategy leads to similar performance. Furthermore, Vakili et al. (2022) explored the effects of two approaches to de-identification, namely pseudonymizing PHI in a text and removing PHI-including sentences. The work concludes, that there is no negative impact on the performance of the models on downstream NLP tasks such as NER, text classification, etc.

Also the work of Lange et al. (2020) explore the performance of concept extraction using de-identified data and conclude the performance drop is only marginal. Finally, although not clinical text, in Lampoltshammer et al. (2019) show that anonymization can cause significant negative changes in the sentiment analysis performance on Twitter data.

This work however goes beyond existing related work, as we carry out a structured analysis regarding anonymization of clinical text, testing seven different techniques, on six datasets, including five different NLP tasks.

The experiments in this work are based on the following datasets and tasks:

• 2010 i2b2/VA Uzuner et al. (2011) (named entity recognition, NER)

• 2018 n2c2 Henry et al. (2019) (named entity recognition, NER)

• 2006 Smoking Challenge Uzuner et al. (2008) (multi-class classification, MCC)

• 2008 Obesity Challenge Uzuner (2009) (multi-label classification, MLC)

• MedNLI Shivade (2019) (natural language inference, NLI)

• ClinSTS Wang et al. (2020) (semantic textual similarity, STS)

While the first four datasets include annotated discharge summaries, the last two datasets include pairs of sentences extracted from MIMIC-III Johnson et al. (2016). Due to limited space, we refer the reader to the source papers and to the appendix.

Using those datasets, different text anonymization techniques are applied to the training split. The following techniques are used, based on Suppression, Perturbation, Substitution and Aggregation, as described above:

1. De-identification (DeI) Using the tool Philter Norgeot et al. (2020), all PHI data in the text is replaced by "XXXX".

2. Mask Numbers (MNr) All occurrences of numbers in a given text, both in numerical or alphabetical form, are replaced using “XX”.

3. Shuffle Sentences (ShS) Sentences in a given text are shuffled.

4. Random Swap (RaS) A certain percentage of words are randomly chosen and swapped all over the document.

5. Synonym Replacement (SyR) A certain percentage of the non-stop words in the document are replaced with WordNet synonyms.

6. Clinical Concept Synonym Replacement (CnR) All signs/symptoms, diseases/disorders, and medications are replaced by a random UMLS synonym, using cTAKES Savova et al. (2010) for entity linking.

7. Text Aggregation (AgX) is done by merging a certain amount of shuffled documents (X) into one.

Finally, in order to examine the effect of anonymization on the performance of state-of-the-art machine learning models, we rely on existing BERT solutions which have achieved promising results on the different datasets in the recent past. Particularly we rely on BERT base (uncased) Devlin et al. (2019), Bio+Clinical BERT Alsentzer et al. (2019), as well as BERT long document classification Mulyar et al. (2019).

Based on the outcomes of the previous two experiments, we draw the following conclusions regarding clinical text anonymization:

There is no one-size-fits-all anonymization technique that can always be recommended. The optimal technique needs to be selected depending on the (security) requirements, the sensitivity of the data as well as underlying NLP task. Overall, the results indicate a correlation between performance loss and strength of anonymization technique, but each technique of course comes at a cost. While some can be quickly conducted, such as sentence shuffle or aggregation, others require additional tools and resources such as DeI or SyR, which can prohibit their use in some scenarios.

Text aggregation is the strongest of the presented techniques. It offers relatively good security against re-identification, but leads to the strongest performance loss. This technique not only aggregates the texts and their contexts together but also results in less training data, which is one of the reasons for the performance loss, as seen in the appendix. Although text aggregation is generally the technique of choice for providing the maximum security, in case of multi label classification, it suffers a strong performance drop. In this case, we recommend relying on substitution techniques, such as synonym replacement. To provide further abstraction, substituted data could be shuffled and be possibly enriched with additional sentences.

Another disadvantage of text aggregation is the fact that long text documents get even longer. Depending on the NLP task, documents need to be processed at once (e.g. document classification) and standard BERT models can deal only with up to 512 input tokens. Relevant information might get lost, and models result in lower performance.

Tasks which can be conducted on sentence level can be easier detached from a patient, and thus weaker anonymization techniques can be applied. Finally, the applied perturbation techniques appear not to be useful for the given tasks due to the weak anonymization.

This work presented a structured analysis regarding text anonymization and its influence on the machine learning model performance. Our experiment tested seven different anonymization techniques on multiple datasets, including five different clinical NLP tasks. In addition, we conducted a simple fictitious re-identification experiment to examine the robustness of each technique. Together with the results, we present some recommendations and learnings. For short, we did not find a one-size-fits-all anonymization technique that would perform best in all tasks. The particular decision depends on several factors. In addition, we provide the software to conduct the text anonymization studies. In future work, more de-identification and anonymization techniques could be added to hopefully arrive at a comprehensive overall picture.