NMF

Used by TermExtractor with method="nmf".

NMFTermExtractor(threshold=None, n=1, n_topics=10, max_features=None, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)

Bases: BaseTermExtractor

NMF-based term extraction with n-gram support.

Initialize the NMF extractor.

Parameters:

Name	Type	Description	Default
threshold	Optional[float]	Minimum score threshold for term inclusion.	None
n	int	Maximum n-gram size (e.g., 1 for unigrams, 2 for bigrams, etc.).	1
n_topics	int	Number of topics for NMF-based extraction.	10
max_features	Optional[int]	Maximum number of features for TF-IDF vectorization.	None
stopwords	Union[str, Iterable[str], None]	Drop candidates that start or end with a stopword: a name in STOPWORD_LISTS, a list of words, or None.	'english'
words_layer	Optional[str]	For Teanga input, the span layer holding the words, or None for words or, failing that, tokens.	None
lemma_layer	Optional[str]	For Teanga input, a sequence layer holding the lemma of each word, to count inflected forms such as "cell" and "cells" as one term, or None to use the words as written.	None
min_frequency	int	Drop candidates that occur fewer times in the corpus.	1
word_pattern	Optional[str]	Drop candidates with a word that does not fully match this regular expression or named pattern ("alphanumeric", "atr4s"), or None to keep every word.	'alphanumeric'
pos_pattern	Optional[str]	For Teanga input, count only the occurrences whose part-of-speech tags match this pattern, or None.	None
pos_layer	Optional[str]	The part-of-speech layer for pos_pattern, or None for upos or, failing that, pos.	None

score_terms(docs)

Score the candidates of a corpus by their largest weight in any NMF topic.

Parameters:

Name	Type	Description	Default
docs	List[List[str]]	The occurrences of candidates in each document.	required

Returns:

Type	Description
dict[str, float]	Dict[str, float]: The score of each candidate.

extract_terms_strings(corpus)

Extract terms from a plain list of strings using NMF.

extract_terms_teanga(corpus)

Extract terms from a Teanga corpus using NMF.

teanga_words(doc)

Get the words of a Teanga document with their offsets.

Parameters:

Name	Type	Description	Default
doc	Document	A Teanga document with a words layer (see words_layer).	required

Returns:

Type	Description
list[tuple[int, int, str]]	List[Tuple[int, int, str]]: A (start, end, text) tuple for each word, where
list[tuple[int, int, str]]	text is the word's lemma if lemma_layer is set.

candidate_ngrams_teanga(corpus)

Find the occurrences of the candidates in a Teanga corpus.

Parameters:

Name	Type	Description	Default
corpus	Corpus	A Teanga corpus with a words layer (see words_layer) and, if pos_pattern is set, a part-of-speech layer.	required

Returns:

Type	Description
dict[str, list[tuple[str, tuple[int, int]]]]	Dict[str, List[Tuple[str, Tuple[int, int]]]]: For each document ID, every
dict[str, list[tuple[str, tuple[int, int]]]]	occurrence of a candidate, with its start and end character offsets.

candidate_ngrams_strings(docs)

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
docs	List[List[str]]	The words of each document.	required

Returns:

Type	Description
list[list[str]]	List[List[str]]: Every occurrence of a candidate in each document.

Raises:

Type	Description
ValueError	If pos_pattern is set, since plain text has no tags.

candidate_occurrences(docs, tags=None)

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
docs	List[List[str]]	The words of each document.	required
tags	Optional[List[List[str]]]	The part-of-speech tag of each word of each document, needed if pos_pattern is set.	None

Returns:

Type	Description
list[list[tuple[str, int, int]]]	List[List[Tuple[str, int, int]]]: Every occurrence of a candidate in each
list[list[tuple[str, int, int]]]	document, with its start and end (exclusive) word index.

Raises:

Type	Description
ValueError	If pos_pattern is set and there are no tags.

frequent_candidates(ngrams)

Find the candidates that occur at least min_frequency times.

Parameters:

Name	Type	Description	Default
ngrams	Iterable[str]	Every occurrence of a candidate in the corpus.	required

Returns:

Type	Description
set[str]	Set[str]: The candidates that occur often enough.

generate_ngrams_teanga(words_with_offsets)

Generate n-grams with offsets for a Teanga corpus.

Parameters:

Name	Type	Description	Default
words_with_offsets	List[Tuple[int, int, str]]	List of (start, end, text) tuples for words.	required

Returns:

Type	Description
list[tuple[str, tuple[int, int]]]	List[Tuple[str, Tuple[int, int]]]: List of n-grams with their start and end offsets.

generate_ngrams_strings(words)

Generate n-grams for a plain list of strings.

Parameters:

Name	Type	Description	Default
words	List[str]	List of words in the document.	required

Returns:

Type	Description
list[str]	List[str]: List of n-grams.

compute_tf(ngrams, total=None)

Compute the relative frequency of each n-gram in a document.

Parameters:

Name	Type	Description	Default
ngrams	List[str]	The n-grams in the document.	required
total	Optional[int]	The number to divide the counts by, or None for the number of ngrams.	None

Returns:

Type	Description
dict[str, float]	Dict[str, float]: Count of each n-gram divided by the total number of n-grams.

ngram_total(num_words)

Count the n-grams of up to n words in a document, candidates or not.

Parameters:

Name	Type	Description	Default
num_words	int	The number of words of the document.	required

Returns:

Name	Type	Description
int	int	The number of n-grams.

compute_idf(corpus_ngrams)

Compute the inverse document frequency of each n-gram as log(N / (1 + df)).

Parameters:

Name	Type	Description	Default
corpus_ngrams	List[List[str]]	The n-grams of each document.	required

Returns:

Type	Description
dict[str, float]	Dict[str, float]: IDF score for each n-gram, where N is the number of documents
dict[str, float]	and df is the number of documents containing the n-gram.

doc_terms(doc_id, scores)

Build one document's entry in ATEResults.terms.

Parameters:

Name	Type	Description	Default
doc_id	str	The document ID.	required
scores	Dict[str, float]	The score of each candidate term.	required

Returns:

Type	Description
dict[str, Any]	Dict[str, Any]: {"doc_id": doc_id, "terms": [{"term": ..., "score": ...}, ...]}, with
dict[str, Any]	terms below the threshold or starting or ending with a stopword removed, and the rest ranked from highest to lowest score.

doc_offsets(doc_id, ngrams_with_offsets, doc_terms) staticmethod

Build one document's entry in ATEResults.offsets.

Parameters:

Name	Type	Description	Default
doc_id	str	The document ID.	required
ngrams_with_offsets	Iterable[Tuple[str, Tuple[int, int]]]	Every n-gram occurrence in the document with its (start, end) character offsets.	required
doc_terms	Dict[str, Any]	The document's entry in ATEResults.terms.	required

Returns:

Type	Description
dict[str, Any]	Dict[str, Any]: {"doc_id": doc_id, "offsets": [{"term": ..., "offset": (start, end)}, ...]},
dict[str, Any]	listing each occurrence of an extracted term in document order.