Demystifying the COVID-19 vaccine discourse on Twitter

Figure (a)a presents the yearlong time series of the volume of tweets classified as pro-vax (blue) and anti-vax (red) and the total number of tweets (grey). The first notable feature is a $10$-fold increase in vaccine related tweets from Nov. 9, 2020, which coincides with Pfizer announcing preliminary efficacy results for their COVID-19 vaccine [Pfizer-announcement]. We use this announcement date to divide the year into two significantly different periods, viz. the ‘pre-vaccine’ and ‘post-vaccine’ periods. The second significant aspect is that the number of pro-vax tweets is over 3 times than anti-vax tweets, noting that the pro-vax tweets contain tweets from governments, pharmaceutical companies, and media agencies, besides regular users.

Figure (c)c shows the growth of unique user IDs associated with pure anti-vax (red), dual-stance (violet), and pure pro-vax (blue) tweets over the year, while Fig. (d)d shows the number of users with a given number of tweets during the study period. Both anti-vax and pro-vax classes follow power law distribution with similar exponent. This observation conforms to the previous studies [Lu2014, Bild2015], which found user aggregate behaviour over Twitter often follows power law distributions. Figure (e)e plots separately the tweets/user statistics for dual-stance users from pure anti-vax and pure pro-vax users, which also shows that dual-stance users are more active than those who posted only single stance tweets. Moreover, both classes of anti-vax and pro-vax also exhibit almost identical user growth rates – lower and higher in the pre-vaccine and post-vaccine periods, respectively. However, pro-vax tweets have a higher growth rate than anti-vax tweets.

Using topic modelling, we identify 53 distinct discussion topics, with 31 and 43 topics exclusive to pro- and anti-vax tweets, respectively. 21 topics are common across both classes. Note that the topics are not mutually exclusive, and one tweet can contain multiple topics, while $20-25\%$ of both pro- and anti-vax tweets were not classified into any topic. Figure (a)a shows the percentages of tweets for the identified topics, relative to the total number of pro- and anti-vax tweets. One striking difference is the diversity of topics covered by anti-vax tweets. Whereas ‘vaccine development’ ($51.3\%$) dominates pro-vax tweets, the anti-vax tweets cannot be summarised with a few topics. The top anti-vax theme ($29.5\%$) is ’vaccine safety’, a combination of ‘side effects’, ‘rushed vaccine’, and ‘jokes (side effects)’, followed by ‘vaccine mandate’ ($17.9\%$), and all other topics with a less than $10\%$ share. Note that we separate the humorous tweets into two groups, with jokes regarding side effects in ‘jokes (side effects)’ and the rest in ‘jokes’, mainly to quantify the safety concerns.

To characterise dual-stance users’ tweets relative to pure anti- and pure pro-vax users, we compute the normalised difference between the actual and the expected contributions of dual-stance users in each topic, as shown in Fig. (b)b. The expected contribution of dual-stance users for a pro-vax (anti-vax) topic is calculated by multiplying the number of topic tweets by the ratio of dual-stance pro-vax (anti-vax) tweets to total pro-vax (anti-vax) tweets, i.e., $0.53$ ($0.63$). An excess contribution by dual-stance users corresponds to lesser contribution, by equal amount, from the pure pro-vax/anti-vax users. The most striking observations are: 1) dual-stance users are more active than both pure pro-vax and anti-vax users in the majority of the discussion topics, and 2) jokes are mostly popular with both the pure pro- and anti-vax users (see Supplementary Information for more detail).

Next, we examine the key topics¹¹1Details about less significant topics are in Supplementary Information. and their dynamics over time by plotting daily tweets and linking their significant peaks to COVID-19 related current events.

We start with anti-vax tweets. Despite their relative infrequency, anti-vax tweets appear to drive many significant dynamics of vaccine discourse. The peaks numbered in Fig. (a)a of the top 10 anti-vax topics appear to be linked to COVID-19 related current events.

1. Aug. 12, 2020 A peak in the topic ‘vaccine mandate’ occurred one day after clinical outcomes for the Russian vaccine Sputnik were announced.

2. Sep. 11, 2020 AstraZeneca’s phase 3 trials were put on hold on Sep. 8, 2020, after one volunteer developed an unknown reaction [AZ-phase3]. Peaks are observed in the topics of ‘side effects’ and ‘jokes (side effects)’.

3. Dec. 7-20, 2020 Multiple peaks are observed in the topics of ‘side effects’, ‘Pfizer/Moderna’, ‘jokes (side effects)’, ‘rushed vaccine’, ‘vaccine mandate’, and ‘masks/lockdowns’, throughout the two weeks of Dec. 2020, coinciding with the Pfizer-BioNTech and Moderna vaccines gaining various approvals in the United Kingdom (UK) and from the USA’s Food and Drug Administration (FDA) [Pfizer-approved-UK, Pfizer-eua, Pfizer-recommend-FDA, Moderna-recommend-FDA, Moderna-eua], and the launch of UK’s public vaccination program on Dec. 8, 2020.

4. Dec. 28, 2020-Jan. 3, 2021 Several countries closed their borders to the UK in response to the Alpha variant of COVID-19 on Dec. 21, 2020 [UK-variant], while the UK government changed the gap between Pfizer doses and approved the AZ vaccine on Dec. 30, 2020 [UK-change-gap, AZ-approved]. We observe higher tweet activity for ‘vaccine mandate’, ‘Pfizer/Moderna’, and ‘jokes (side effects)’ on multiple days including Dec. 31, 2020.

5. Jan. 15-17, 2021 A viral meme, about zombie apocalypse as a result of failed vaccine in the movie ‘I am Legend’ and its relevance to the current pandemic, drove peaks in the ‘side effects’, ‘jokes (side effects)’, ‘Pfizer/Moderna’, and ‘ineffective vaccine’ topics on Jan. 15, 2021. On Jan. 16-17, 2021, the most significant news shared under the ‘jokes (side effects)’ and ‘Pfizer/Moderna’ topics was about 23 deaths in Norway post vaccination [Norway-deaths]. This incident is discussed in detail in the sequel.

6. Feb. 22-26, 2021 UK’s review of vaccine passports on Feb. 23, 2020 [UK-VacPassport], may be behind the spikes in ‘vaccine mandate’ on Feb. 24 and 26, 2021. On Feb. 21, 2021, Dr. Fauci’s stated that Americans may need to wear masks in 2022 [Fauci-masks], which might be linked to a peak in the plot of ‘masks, lockdowns, etc.’ the next day.

7. Mar. 11-17, 2021 On Mar. 11, 2021, use of the AZ vaccine was suspended across many countries due to fears of blood clots [AZ-banned], causing peaks in ‘side effects’ on Mar. 15, 2021 and ‘Pfizer/Moderna’ on Mar. 16, 2021. Coincidentally, the third COVID-19 wave was sweeping across Europe [Third-wave] and there were multiple spikes in the ‘ineffective vaccine’ and ‘vaccine mandate’ topics.

Whereas it is challenging to establish a causal link between a possible triggering current event and the topics discussed in the tweet dataset, many of the anti-vax tweets appear to be spontaneous responses to key COVID-19 vaccine events. This suggests that it may be challenging to develop counter-narratives in a timely manner and deploy them to prebunk vaccine-related misinformation [ecker2022psychological]. Moreover, memes and funny tweets seem to play a significant role in progressing anti-vax narratives, and we will return to this in detail later. Next, we discuss the major anti-vax topics in detail.

Pro- and anti-vax tweets discussed the common topics from opposite viewpoints as illustrated below.

Of the top 10 pro-vax topics shown in Fig. (b)b, ‘vaccine development’ is dominant throughout the study period. Over 50% discussed this topic, including updates from clinical trials, news about the efforts of governments to fund the research and acquire vaccines for their people, vaccine rollout details, scientific information about the vaccines - especially mRNA technology, news about public leaders and celebrities making donations and publicly receiving a vaccine, etc. The topic plot shows various peaks which can be attributed to a combination of different causes, such as press releases about clinical trials - including trial results and issues, important vaccine announcements, updates about approval procedures, and the launch of various contact tracing apps. We observe peaks in ‘vaccine development’ on Aug. 12, 2020 and during Dec. 7-20, 2020 (for the particular vaccine-related events, see The Anti-vax Tweets). One peak is attributed to the start of the Indian vaccination program on Jan. 16, 2021. The US Presidential Debates were also referenced in the context of ‘vaccine development’, especially in the weeks leading up to the elections.

The topic ‘debunking anti-vax conspiracies’ also shows multiple peaks during Dec. 14-20, 2020, which is due to a viral meme mocking anti-vax concerns of the ingredients in vaccines. This topic mostly contains humorous tweets making fun of anti-vax conspiracies, such as, vaccine containing microchip or government trying to kill people using vaccine, etc.

The topic of ‘pro-vax (general)’ is the second most popular pro-vax topic with 9% tweets and include positive statements about COVID-19 vaccine and vaccine in general, such as, people posting after getting COVID-19 vaccine, sharing updates about vaccine rollout and eligibility, encouraging other people to get the vaccine, etc. There is significant overlap between the topics of ‘vaccine development’, ‘distribution’, ‘setbacks and issues’, ‘access to vaccine’, ‘pro-vax (general)’, and ‘debunking anti-vax conspiracies’. Altogether, they define almost $70\%$ of the pro-vax tweets. The rest of the pro-vax topics are discussed in the Supplementary Information.

Memes or humorous tweets, with occasional variations, are among the most frequently retweeted in both anti- and pro-vax groups. our results are summarised in Table 1. These funny messages, although not carrying any argument or information, are perhaps more memorable and shareable. Of the top 10 most retweeted anti-vax tweets, 7 are jokes or memes with multiple variants, which constitute almost $5\%$ of the total volume of anti-vax tweets. We also examine the top 15 most retweeted pro-vax tweets (the top 10 have only 2 memes) and find 5 different jokes and memes. Altogether, they account for $1.3\%$ of the total pro-vax tweets.

Determining whether memes and humour were deliberate attempts to advance the anti-vaccine discourse, is an important question but one beyond the scope of this paper. What is perhaps remarkable is that, based on the timeline identified in Table 1, there almost appears to be a reactionary tête-à-tête between anti-vax and pro-vax memes, triggered by a sequence of news events around vaccine approvals and development (see the events 3-5 in The Anti-vax Tweets).

A key problem of interest is to understand the contribution of misinformation and falsehoods toward the anti-vax discussion. To investigate, we separate the keywords/key-phrases, used for anti-vax topic classification, into three groups: (1) genuine issues, (2) falsehood, and (3) neither. Then, we count the number of anti-vax tweets with topics appearing in the first two groups, as the ‘neither’ group could also be put together with the anti-vax tweets not classified into any topic. It is possible that one tweet can be classified into multiple groups, as shown by the plot of ‘About genuine and false issues’ in Fig. 13, as the tweet may contain keywords related to genuine issues and falsehoods.

The topics of genuine or legitimate concerns are identified by consensus among the authors, as detailed in the Supplementary Information. We consider these topics genuine concerns as they offer some grounds for reasoned debate and argumentation, as opposed to demonstrably false claims or fictitious issues where no reasoned discussion is possible. The genuine concerns are (see Supplementary Information for details)

• Mandatory vaccines and loss of freedoms

• Fast-tracked vaccine

• Historical issues with vaccines and clinical trials

• Pharmaceutical companies profiteering

• General vaccine side effects

• Blood clots after receiving the AstraZeneca vaccine

• Waning immunity and virus variants

• Administrative mismanagement

• Animal abuse during vaccine development.

Conversely, topics found under falsehoods include conspiracies, anti-vax memes, and tweets about fictitious issues. The conspiracies we identify were: ‘experimental mRNA/altering DNA’, ‘vaccine contains microchip’, ‘metal/nanoparticles in vaccine’, ‘cells from aborted babies in vaccine’, ‘fertility issues’, ‘Robert F. Kennedy’ (as a figurehead in the vaccine scepticism movement), ‘related to flu vaccine’, ‘eugenics’, ‘vax makes you gay’, and ‘QAnon’. The Supplementary Information provides a detailed breakdown on this group of 10 ‘conspiracies’. Fictitious issues include blaming COVID-19 vaccines for the deaths of 23 elderly people in Norway, claims that COVID-19 fatality rates were lower in comparison to influenza, claims that COVID-19 has naturally diminished without a vaccine, or that there was no point to have vaccines. The tweets which are not classified into either ‘genuine issues’ or ‘falsehood’ are typically simple statements such as “I do not want a vaccine”, or accusations at public leaders because of something they said or did, discussions about alternative treatments being a better choice than a vaccine, and also some tweets which are not classified because of lack of suitable keywords.

Figure 13 shows the fraction of tweets under each group for each week throughout the year. Only 10-15% of anti-vax tweets contained some genuine issue, while a large portion was based on fictitious issues and falsehoods (although ‘conspiracies’ accounted for just $6.6\%$ of the anti-vax tweets). Around 38% of the yearlong anti-vax tweets belonged to neither of the groups. We remark that our keyword-based classification cannot distinguish between well- and poorly-reasoned tweets about a genuine topic. Many tweets identified as discussing a ‘genuine concern’ in fact used excessive exaggeration or presented information about true facts/events out-of-context, such as the following tweet.

"The last rushed vaccine cost the government £60M in payments for compensation of injury. How long will it be before we hear about the side effects of a C19 vaccine when its rolled out to millions? Humanity has lost it’s way in thinking a poisonous concoction is ‘for our safety’."

Nonetheless, there are some well-reasoned tweets in this group as well, such as

"Take the jab or lose your job: Medical journal calls for a MANDATORY Covid vaccine, says ‘noncompliance should incur a penalty’."

From Fig. 13, falsehoods clearly dominate the topics discussed in anti-vax tweets, although there is a slight decline towards the end of the observation period. While our findings around dual-stance users suggests that the users on Twitter are not as polarised around COVID-19 vaccines as one may initially assume, the significant amount of falsehoods found in anti-vax tweets does confirm that addressing online misinformation around vaccines is still a critical challenge.

This study uses the publicly available dataset collected and maintained by R. Lamsal [Lamsal2021], which collected the tweet IDs for global English language tweets that featured COVID-19 related keywords. The complete list of keywords can be found in Ref. [Lamsal2021]. We started hydrating Lamsal’s dataset in Feb. 2021, which involved retrieving the tweet and associated attributes from the tweet ID in the Lamsal dataset. We observed that $20-40\%$ of tweets could not be retrieved for the period Mar.-Apr. 2020. This hydration loss gradually came down to $5-15\%$ towards Feb.-Mar. 2021. In order to extract the vaccine related tweets, we further filtered the dataset with the keywords ‘vaccine’, ‘vaccination’, ‘vax’, ‘vac’, ‘jab’, and ‘shot’.

Our study used a stance detection tool created from OpenAI’s GPT transformer model [Rao2019] to classify tweets into three categories: favour, against, and none, for the topic ‘vaccine hesitancy’. The classification accuracy, measured in terms of composite F-score for favour and against classes [Rao2019], depends critically on the labelled set used to fine-tune the GPT model. After various trials, we ended up selecting around 100 tweets per day, which gave us reasonable F-scores of 0.6 or above for 20 test sets we picked and labelled from the yearlong dataset [Lamsal2021]. Test sets are comprised of 250-1000 randomly selected tweets from day 1, 20, 40, 60, …, from the 368 days in the dataset. The composite F-scores for the 20 test sets are in the range of 0.67-0.87.

In order to make this yearlong study possible with reasonable confidence, we had around $42,000+$ tweets to label. As the conversation was changing throughout the year, it was critical to have a relevant labelled set of tweets to fine tune the GPT model, which can then be used for detecting stances for the tweets of that specific period. Considering the enormous task load, we devised an annotation strategy with two phases. A primary annotator provided one set of labels for all $42,000+$ tweets in the first phase. The second phase focused on estimating the bias error in annotation by the primary annotator from the first phase. Extra annotators provided at least two additional sets of independent labels for 3000 tweets from Mar-Apr 2020, and 3000 tweets from Sep-Oct 2020, and a majority rule was applied to get the final label. There was a 93% similarity between the final labels and primary annotations for Mar-Apr 2020 and 89% for Sep-Oct 2020. We remark that the period Sep-Oct 2020 was exceptionally complex for annotation due to an ongoing contentious debate between Democrat and Republican voters in the US, and leaders from both sides were called anti-vax. We labelled all such political tweets with the stance ‘none’, noting that it was not always clear if the main focus of the tweet was vaccines or politics.

The stance prediction results when the GPT model was fine-tuned using both sets of labels, i.e., final labels and primary annotation only, are 91% similar for the Mar-Apr 2020 period and 84.1% for the Sep-Oct 2020 period. Both set of labels result in almost similar predictions. We thus argue that the bias error is marginal in this study and can be safely ignored. Moreover, any misclassification of tweets due to personal bias can be handled, along with the prediction error of the GPT model, when each class is thoroughly analysed for specific discussion points through the subsequent topic modelling.

LDA topic modelling relies on co-occurrence of words in a document and a major issue with tweets, which comes under Short Text Topic Modelling (STTM), is that it is limited to a maximum of 280 characters [Qiang2019]. We tested several topic modelling tools, such as, LDA⁶⁶6https://nicharuc.github.io/topic_modeling/, GS-DMM [Yin2014], Bertopic [Grootndorst2022], Top2vec [Angelov2020] — we found that GS-DMM, an LDA based topic modelling tool optimised for STTM, was most suitable in finding relatively distinct and meaningful cluster of words. However, GS-DMM alone was not sufficient for the topic classification tasks, because as it turns, the core topics are highly intertwined with one another and use similar words.

We, then, decided to use GS-DMM for keyword collection instead and added another phase in our strategy where these keywords are manually searched in the tweet files and key-phrases are selected for final topic classification. The key-phrases lowered the misclassification rate considerably. We manually checked the 50 high frequency tweets under each topic to calculate the error rate. More than half of the topics have zero error, and only 7 topics have an error rate higher than $10\%$. The topic of ‘RNA altering DNA’ has the highest error rate of $25\%$, ‘side effects’ has $19\%$ error, and $16.8\%$ error was seen in the topic of ‘microchip’ but these errors appear to be mostly due to stance misclassification of satirical tweets. The topic ‘Bill Gates’ has an error rate of $14.7\%$ for the anti-vax class, and this is due to the stance detection tool misclassifying tweets that discussed legislative bill and Bill Clinton.

The two main tools used to build a retweet network are NetworkX and Gephi. In a retweet network, users are represented by nodes, and retweet relationships are represented by edges. Edges are directed. The retweeted users and the original tweeted users in the database are extracted to construct an edge table. Then NetworkX is used to convert the edge table into a network file. Gephi was used to visualise the network file.

The code used and the data generated in the simulations is available at https://github.com/https://github.com/ZainabRZaidi/COVID-19-Vaccine-Hesitancy-.