Combining keyphrase extraction and lexical diversity to characterize ideas in publication titles

Keyphrase extraction has been explored for many natural language processing tasks, most often utilizing full text when it is available Nguyen and Kan (2007), Kim and Wilbur (2001), and Turney (2000). Relatively few projects have attempted to extract keyphrases from short text or publication titles. Keyphrase Extraction and Ranking by Topic (KERT) aims to generate topical keyphrases from short texts, arguing that ”content-representative titles are cleaner and more efficient to deal with than entire documents. Li et al. (2010) also explores keyphrase detection in titles, noting that ”the title has a similar role to the key phrases” and arguing that when a keyphrase occurs in a title, it ought to be ranked higher. Bhowmik (2008) proposed using a simple neural network to extract keyphrases from titles and abstracts, and they compared their results with author provided keyphrases. Another often mined collection of short texts are available on Twitter and they pose similar challenges for keyphrase extraction as document titles. Most of this work has been concerned with summarizing tweets via keyphrase extraction (Zhao et al. (2011), Zhang et al. (2016), and Bellaachia and Al-Dhelaan (2012)).

Lexical diversity has been applied to a range of language evaluation use cases, including language acquisition, language comprehension, assessment of language disorders and language impairment, and generally to measure an individual’s reading, speaking, or writing skills. It is a measure of the richness of the vocabulary used or comprehended by an individual. General guidance from McCarthy et al recommends that researchers “consider using [multiple metrics] in their studies, rather than any single index, noting that lexical diversity can be assessed in many ways and each approach may be informative as to the construct under investigation.” McCarthy and Jarvis (2010) In recent years, we know of only two studies that have used lexical diversity to characterize the concepts and ideas in scientific publication titles. Milojević (2015) explored lexical diversity as a means to ”quantify the extents of cognitive domains of different bodies of scientific literature” using publication titles he found that ” periods of cognitive growth do not necessarily coincide with the trends in publication volume.” Bérubé et al. (2018) developed an approach to ”assess the evolution of lexical diversity in scholarly titles using a new indicator based on Zipfian frequency-rank distribution tail fits.” They found that they could measure lexical concentration of titles as well as predict the leveling off of vocabulary expansion in a given field.

Our test corpora consisted of the titles of publicly available reports and papers in our local library repository. Our organization employs a pre-release review, which means that most publications intended for external publication are submitted by the primary author and stored in a local database. Only items that are not flagged as drafts, and are not embargoed, are included in our data. We used publication titles spanning 2012 to 2020 (see table  2). Since there is sometimes a lag for submissions, the number of 2020 articles may be lower due to items not yet submitted or made public. We employed several means for identifying lexical units that reflect distinct concepts, or keyphrases, including statistical techniques as well as part of speech tagging and extraction. These are described in the methods section below.

Basic text normalization was employed including lower case, tokenization via punctuation, and a very small stopwords list was also used, consisting of ’a, an, the.’ Since many phrases may contain words that are usually considered stopwords, we elected not to use a full stopword list. By removing these three words, we vastly reduced the number of article + noun phrase constructs identified by the part of speech model. Since we were interested in keyphrases and not all words in titles, this approach allowed us to primarily accumulate more complex n-gram phrases.

As in Bérubé et al. (2018), we are interested in extracting and analyzing keyphrases, rather than just keywords or all words. We focused on fast, simple algorithms for keyphrase extraction, that require minimal training, are unsupervised, or were based on available off-the-shelf models. We extract keyphrases from titles, reducing each title to its constituent phrases. This results in a distinct corpus variant per keyphrase detection algorithm. Then we measure the lexical diversity of these variants of the original title corpus using four different lexical diversity metrics, one due to its intuitive nature, and three others which have been empirically demonstrated to produce diversity scores that are less susceptible to the variable text length problem that plague some lexical diversity metrics.

We selected three techniques for detecting and extracting keyphrases in the title corpora:

• A pre-trained phrase model trained only on the aggregation of all titles spanning 2012-2020 (the title phraser),

• A pre-trained phrase model trained on several decades of (our organization) reports (the full-text phraser),

• Noun phrase detection based on part of speech detection supported by a large English language model implemented in the SPACY NLP library.

The title and full-text phrases use the same training algorithm with different size corpora: Pointwise Mutual Information (PMI) Bouma (2009). The objective of PMI is to establish the likelihood that two words co-occur as a single lexical unit, as reflected by equation  1

SPACY Honnibal and Montani (2017)utilizes part of speech detection in order to identify noun phrases in text sequences. Noun phrases have been explored in the past for improving access to content in digital libraries Tolle and Chen (2000). Part of speech detection is best performed over the original text rather than text which has been modified by stopword elimination and lemmatization. As mentioned above we do remove three articles from the corpus, but otherwise the original titles are presented as input with punctuation introduced at the end of each title (a period) to provide SPACY with the equivalent of sentences rather than a continuous stream of text. We used the large SPACY english language model (en_core_web_lg).

We applied each title phrase extraction model to each year of the title corpus. These titles were reduced to only the phrases detected by the phrase extraction model. This resulted in three modified versions of the title corpus, one per phrase extraction model. For comparison, we also used unprocessed versions of the corpora in their original form, with no keyphrase detection applied (we refer to the unmodified title corpora as the “all words” version). Table  1 illustrates the conversion process from the original sentence to various phrased forms of the sentence.

Lexical diversity is broadly defined as a measure of the variability and richness of a given text. The simplest way to compute this is to count the number of unique words in a text and divide this value by the total word count. This metric is called the type-token ratio (TTR). For example, if a document has a total of 24 words and 15 of which are unique, then the TTR score is 0.625 (15/24). If the document contains 19 unique words, the TTR score would be 0.79. So the higher the TTR, the greater the lexical diversity. It may not be immediately apparent, but this approach is sensitive to text length. This is due to the fact that the total number of tokens will always grow more quickly than the unique token count, so the TTR score can only decrease as the length of text increases. This makes TTR an unstable metric for comparing texts, if there’s a significant difference in the length of the texts being compared. Since TTR is problematic for many lexical diversity use cases, a number of alternate metrics have been proposed which aim to neutralize the effect of variable text length. Many of these metrics employ statistical sampling, or normalization and scaling.

Two examples that use statistical sampling are hypergeometric density (HD-D) and Measure of Textual Lexical Diversity (MTLD). Treffers-Daller (2013). HD-D starts with a list of all of the unique words in a corpus. A score is calculated for each word based on the observed probability of finding it in a random sample of 42 words from a text. The final HD-D score for a corpus is the sum of the probabilities for all distinct token types.

The Measure of Textual Lexical Diversity (MTLD) uses the TTR described above in conjunction with a sampling technique. The text is split into segments and the segment sizes are increased or decreased until a TTR score of .72 is achieved for a given segment. The final MTLD score is the mean text length for of all of these segments. The HD-D sample size (42) and the target score for MTLD (.72) were arrived at empirically by the researchers who developed these respective metrics.

An example lexical diversity metric that uses normalization is the Maas index Treffers-Daller (2013). The Maas index is normalized and scaled version of the TTR, which is represented by equation  3:

Generally speaking, HD-D and MTLD scores will increase as lexical diversity increases. However, the Maas index will decrease as lexical diversity increases.

Table  2 contains information about each of the four corpora, including year, document count, mean title length, and the percent change for unique lexical unit types over time. Percentage change is in relation to the previous year, so for example in 2013, there was a 14.32% increase in the word count, a 14.09% increase in detected title phrases, etc. We analyze each version of the corpora using the four lexical diversity metrics described above (TTR, MTLD, Maas, HD-D) applying them to each of the four versions of the title corpus. This is repeated for each year of report titles. The MTLD index (figure  4) for (our organization) reports spanning 2012-2020 exhibits the most significant and persistent shift for the various corpora, starting in 2018. MTLD scores were also highly correlated across all versions of the title corpus, including variants with noun phrases, phrases from the full-text model, and phrase from the titles only model. For the other three metrics, TTR (figure  1 had four highly correlated relationships out of six. There were only one pair of title representations that were highly correlated for Maas (Keywords and Noun phrases). Similarly, only one relationship was highly correlated per HD-D scores (Full-text phraser and Noun phrases).

Compared to MTLD, the Maas index  3 exhibited dramatically different results, with a relatively stable score across all years of the corpus, except for a drop at the end which is likely due to incomplete data for 2020. This drop provides some evidence that MAAS is more susceptible to text length than MTLD, which has been reported by others.

HD-D scores (figure  2) for 2012-2020 exhibited relative stability until 2017 when the score for all words and for full-text phrases decreased, and the score for title phraser increased. Table  3 contains correlation scores between corpus types, by lexical diversity time series scores.

Mindful of the potential impact of text length on the results, we reviewed the output statistics for raw text and derivative versions of the text corpus. It appears that there was a sustained increase in publication output starting after 2017 (table  2 and in some metrics a commiserate shift in lexical diversity. Output increased by several over 1700 publications in 2017 and continued to increase in the three subsequent years. But as noted above, upwards trends in lexical diversity did not start until after 2017. One possibility is that there were a higher than usual number of institutional reports or whitepapers added in 2017.

Word counts per year (table  2) may also provide clues about lexical diversity scores. In 2016 there was a small dip in the number of unique words per title set per year, followed by an increase of almost 2000 words after 2017. In subsequent years, the unique word count remained almost constant. So it would appear that while MTLD did detect a shift in lexical diversity after 2017, the increase it reported for seemed out sized compared to the unique word count increase, but tracks more closely with the increase in the number of reports released. The HD-D index most accurately reflected the changes in unique word count by year. It also seemed not to be affected by the number of reports released per year. So it appears that for tracking year over year shifts in the lexical diversity of (our organization) report titles, the HD-D index performance seemed most likely to reflect an actual trend in the data with respect to new language and possibly indicative of new or expanding research, which manifested as a gradual upward trend until 2016 when a dip in lexical diversity occurred, followed by a more significant upward trend starting after 2017. This pattern seems to be real, confirmed by both the MTLD index scores and the HD-D index scores across multiple versions of the corpus.

The TTR and HD-D lexical diversity metrics were in close agreement over time: the noun phrases titles had the highest lexical diversity, followed by full text phrases, title phrases, with the original titles tokenized by word exhibiting the lowest scores. MTLD and Maas differed slightly: again noun phrases scored highest over time, with full text phrases, all words, and title phrases reported to have the lowest lexical diversity per these metrics.

A number of studies have suggested that MTLD is the least sensitive to factors such as variable text length. This, combined with the fact that the scores reported over time by MTLD for all corpora variants were highly correlated lead us to conclude that this metric provides the most accurate representation of lexical diversity in our title corpora.

Next we examine the discrete sets of phrased identified by each phrase extraction approach. We are particularly interested in the intersection of these sets, as well as gaining a better understanding of where they were not in agreement and whether these differences were in some way useful for improving the task of evaluating concept expansion. For example, we wanted to know if phrases identified by some of the more liberal models but not included in the smaller models were less interesting phrases, or vice-versa.

We found that there was a consistent ratio for the intersections between the corpus produced using the title phraser, full-text phraser, and the noun phraser. The noun phraser model always detected a larger number of phrases than the other two (figure 7), while the title phraser, trained on only the aggregated title corpus for 2012-2020, identified the smallest number of phrases. The size of each phrase collections also seem to be consistent in relation to one another from year to year (see Venn diagrams, figure  5 ). But it was also apparent that each model yielded distinctly meaningful phrases that were not detected by the other models. In some cases, it was a matter of finer granularity, which perhaps counter-intuitively is represented by longer phrases.

Tables  4 and  5 provide some specific examples of how the phrase models intersect in 2020, as well as how they differ. The full-text phraser was better at identifying named entities as well as important concepts. However some of the named entities were rather generic. While they are clearly valid phrases, they obviously do not represent concepts or research topics. It would be interesting to specifically apply named entity detection over the titles to see if there is an intersection between full text phrases, noun phrases, and named entities that might form the basis of a ”stop phrase” list. It may make sense to eliminate phrases high frequency phrases that were detected by the full-text phraser, but not by the title phraser, perhaps by empirically establishing a frequency threshold value for this task. This may allow for automatic generation of stop phrases which could be safely removed by the pre-processing pipeline without requiring humans to produce or revise such a list, which was noted as a problem by Milojević (2015).