Topic Modeling (LDA)

Used by TermExtractor with method="topicmodeling".

`TopicModelingTermExtractor(num_topics=20, top_words=200, threshold=None, n=1, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)`

Bases: BaseTermExtractor

Topic-model term extraction, after the Novel Topic Model of Li et al. (2013) as implemented in ATR4S (Astrakhantsev, 2018).

An LDA topic model is trained on the single words of the corpus, leaving out stopwords and words without a letter or digit. The typical words are the top_words most probable words of each topic. A candidate ( t ) with words ( w_1 \dots w_L ) and frequency ( f(t) ) in the corpus scores ( \log f(t) \cdot \sum_j \max_k \phi_{k w_j} ), where ( \phi_k ) is the word distribution of topic ( k ), and words that are not typical count 0. Scores are computed over the whole corpus, so a term has the same score in every document it occurs in.

Unlike the Novel Topic Model, the topic model has no background or document-specific topics, and is trained with scikit-learn's variational LDA rather than Gibbs sampling.

Initialize the Topic Modeling extractor.

Parameters:

Name	Type	Description	Default
`num_topics`	`int`	Number of topics for topic modeling.	`20`
`top_words`	`int`	The number of most probable words of each topic that count towards the scores of candidates.	`200`
`threshold`	`Optional[float]`	Minimum score for term inclusion.	`None`
`n`	`int`	Maximum n-gram size (e.g., 1 for unigrams, 2 for bigrams, etc.).	`1`
`stopwords`	`Union[str, Iterable[str], None]`	Drop candidates that start or end with a stopword, and leave stopwords out of the topic model: a name in `STOPWORD_LISTS`, a list of words, or None.	`'english'`
`words_layer`	`Optional[str]`	For Teanga input, the span layer holding the words, or None for `words` or, failing that, `tokens`.	`None`
`lemma_layer`	`Optional[str]`	For Teanga input, a sequence layer holding the lemma of each word, to count inflected forms such as "cell" and "cells" as one term, or None to use the words as written.	`None`
`min_frequency`	`int`	Drop candidates that occur fewer times in the corpus.	`1`
`word_pattern`	`Optional[str]`	Drop candidates with a word that does not fully match this regular expression or named pattern ("alphanumeric", "atr4s"), or None to keep every word.	`'alphanumeric'`
`pos_pattern`	`Optional[str]`	For Teanga input, count only the occurrences whose part-of-speech tags match this pattern, or None.	`None`
`pos_layer`	`Optional[str]`	The part-of-speech layer for `pos_pattern`, or None for `upos` or, failing that, `pos`.	`None`

`topic_words(words)`

Get the words of a document that the topic model is trained on.

Parameters:

Name	Type	Description	Default
`words`	`List[str]`	The words of the document.	required

Returns:

Type	Description
`list[str]`	List[str]: The words, lower-cased, without stopwords or words that have no
`list[str]`	letter or digit.

`train_topic_model(docs)`

Train an LDA topic model and find the typical words of its topics.

Parameters:

Name	Type	Description	Default
`docs`	`List[List[str]]`	The words of each document, as given by `topic_words`.	required

`compute_ntm_scores(ngrams)`

Score the candidates of a corpus.

Parameters:

Name	Type	Description	Default
`ngrams`	`List[str]`	Every n-gram occurrence in the corpus.	required

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: The score of each candidate with at least one typical word.

`extract_terms_teanga(corpus)`

Extract terms from a Teanga corpus using topic modeling.

Parameters:

Name	Type	Description	Default
`corpus`	`Corpus`	Teanga Corpus object.	required

Returns:

Name	Type	Description
`ATEResults`	`ATEResults`	Results with ranked terms and scores, and term offsets.

`extract_terms_strings(corpus)`

Extract terms from a list of strings using topic modeling.

Parameters:

Name	Type	Description	Default
`corpus`	`List[str]`	List of documents as strings.	required

Returns:

Name	Type	Description
`ATEResults`	`ATEResults`	Results with ranked terms and scores.

`teanga_words(doc)`

Get the words of a Teanga document with their offsets.

Parameters:

Name	Type	Description	Default
`doc`	`Document`	A Teanga document with a words layer (see `words_layer`).	required

Returns:

Type	Description
`list[tuple[int, int, str]]`	List[Tuple[int, int, str]]: A (start, end, text) tuple for each word, where
`list[tuple[int, int, str]]`	text is the word's lemma if `lemma_layer` is set.

`candidate_ngrams_teanga(corpus)`

Find the occurrences of the candidates in a Teanga corpus.

Parameters:

Name	Type	Description	Default
`corpus`	`Corpus`	A Teanga corpus with a words layer (see `words_layer`) and, if `pos_pattern` is set, a part-of-speech layer.	required

Returns:

Type	Description
`dict[str, list[tuple[str, tuple[int, int]]]]`	Dict[str, List[Tuple[str, Tuple[int, int]]]]: For each document ID, every
`dict[str, list[tuple[str, tuple[int, int]]]]`	occurrence of a candidate, with its start and end character offsets.

`candidate_ngrams_strings(docs)`

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
`docs`	`List[List[str]]`	The words of each document.	required

Returns:

Type	Description
`list[list[str]]`	List[List[str]]: Every occurrence of a candidate in each document.

Raises:

Type	Description
`ValueError`	If `pos_pattern` is set, since plain text has no tags.

`candidate_occurrences(docs, tags=None)`

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
`docs`	`List[List[str]]`	The words of each document.	required
`tags`	`Optional[List[List[str]]]`	The part-of-speech tag of each word of each document, needed if `pos_pattern` is set.	`None`

Returns:

Type	Description
`list[list[tuple[str, int, int]]]`	List[List[Tuple[str, int, int]]]: Every occurrence of a candidate in each
`list[list[tuple[str, int, int]]]`	document, with its start and end (exclusive) word index.

Raises:

Type	Description
`ValueError`	If `pos_pattern` is set and there are no tags.

`frequent_candidates(ngrams)`

Find the candidates that occur at least min_frequency times.

Parameters:

Name	Type	Description	Default
`ngrams`	`Iterable[str]`	Every occurrence of a candidate in the corpus.	required

Returns:

Type	Description
`set[str]`	Set[str]: The candidates that occur often enough.

`generate_ngrams_teanga(words_with_offsets)`

Generate n-grams with offsets for a Teanga corpus.

Parameters:

Name	Type	Description	Default
`words_with_offsets`	`List[Tuple[int, int, str]]`	List of (start, end, text) tuples for words.	required

Returns:

Type	Description
`list[tuple[str, tuple[int, int]]]`	List[Tuple[str, Tuple[int, int]]]: List of n-grams with their start and end offsets.

`generate_ngrams_strings(words)`

Generate n-grams for a plain list of strings.

Parameters:

Name	Type	Description	Default
`words`	`List[str]`	List of words in the document.	required

Returns:

Type	Description
`list[str]`	List[str]: List of n-grams.

`compute_tf(ngrams, total=None)`

Compute the relative frequency of each n-gram in a document.

Parameters:

Name	Type	Description	Default
`ngrams`	`List[str]`	The n-grams in the document.	required
`total`	`Optional[int]`	The number to divide the counts by, or None for the number of `ngrams`.	`None`

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: Count of each n-gram divided by the total number of n-grams.

`ngram_total(num_words)`

Count the n-grams of up to n words in a document, candidates or not.

Parameters:

Name	Type	Description	Default
`num_words`	`int`	The number of words of the document.	required

Returns:

Name	Type	Description
`int`	`int`	The number of n-grams.

`compute_idf(corpus_ngrams)`

Compute the inverse document frequency of each n-gram as log(N / (1 + df)).

Parameters:

Name	Type	Description	Default
`corpus_ngrams`	`List[List[str]]`	The n-grams of each document.	required

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: IDF score for each n-gram, where N is the number of documents
`dict[str, float]`	and df is the number of documents containing the n-gram.

`doc_terms(doc_id, scores)`

Build one document's entry in ATEResults.terms.

Parameters:

Name	Type	Description	Default
`doc_id`	`str`	The document ID.	required
`scores`	`Dict[str, float]`	The score of each candidate term.	required

Returns:

Type	Description
`dict[str, Any]`	Dict[str, Any]: `{"doc_id": doc_id, "terms": [{"term": ..., "score": ...}, ...]}`, with
`dict[str, Any]`	terms below the threshold or starting or ending with a stopword removed, and the rest ranked from highest to lowest score.

`doc_offsets(doc_id, ngrams_with_offsets, doc_terms)` `staticmethod`

Build one document's entry in ATEResults.offsets.

Parameters:

Name	Type	Description	Default
`doc_id`	`str`	The document ID.	required
`ngrams_with_offsets`	`Iterable[Tuple[str, Tuple[int, int]]]`	Every n-gram occurrence in the document with its (start, end) character offsets.	required
`doc_terms`	`Dict[str, Any]`	The document's entry in `ATEResults.terms`.	required

Returns:

Type	Description
`dict[str, Any]`	Dict[str, Any]: `{"doc_id": doc_id, "offsets": [{"term": ..., "offset": (start, end)}, ...]}`,
`dict[str, Any]`	listing each occurrence of an extracted term in document order.

Topic Modeling (LDA)

TopicModelingTermExtractor(num_topics=20, top_words=200, threshold=None, n=1, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)

topic_words(words)

train_topic_model(docs)

compute_ntm_scores(ngrams)

extract_terms_teanga(corpus)

extract_terms_strings(corpus)

teanga_words(doc)

candidate_ngrams_teanga(corpus)

candidate_ngrams_strings(docs)

candidate_occurrences(docs, tags=None)

frequent_candidates(ngrams)

generate_ngrams_teanga(words_with_offsets)

generate_ngrams_strings(words)

compute_tf(ngrams, total=None)

ngram_total(num_words)

compute_idf(corpus_ngrams)

doc_terms(doc_id, scores)

doc_offsets(doc_id, ngrams_with_offsets, doc_terms) staticmethod

`TopicModelingTermExtractor(num_topics=20, top_words=200, threshold=None, n=1, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)`

`topic_words(words)`

`train_topic_model(docs)`

`compute_ntm_scores(ngrams)`

`extract_terms_teanga(corpus)`

`extract_terms_strings(corpus)`

`teanga_words(doc)`

`candidate_ngrams_teanga(corpus)`

`candidate_ngrams_strings(docs)`

`candidate_occurrences(docs, tags=None)`

`frequent_candidates(ngrams)`

`generate_ngrams_teanga(words_with_offsets)`

`generate_ngrams_strings(words)`

`compute_tf(ngrams, total=None)`

`ngram_total(num_words)`

`compute_idf(corpus_ngrams)`

`doc_terms(doc_id, scores)`

`doc_offsets(doc_id, ngrams_with_offsets, doc_terms)` `staticmethod`