BaseTermExtractor

All unsupervised extractors inherit from BaseTermExtractor. It provides the n-gram, TF and IDF helpers they share, and builds their results so that every extractor returns ATEResults in the same format. To add a new extractor, subclass it and implement extract_terms_teanga and extract_terms_strings.

BaseTermExtractor(threshold=None, n=1, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)

Bases: ABC

Common interface and helpers for the unsupervised term extractors.

Subclasses implement extract_terms_teanga and extract_terms_strings, and build their results with doc_terms and doc_offsets so that every extractor returns ATEResults in the same format.

Initialize the extractor.

The candidates are the n-grams of the corpus that pass the filters set by stopwords, word_pattern, pos_pattern and min_frequency. Only the occurrences that pass them are counted, so they change the scores as well as which terms are returned.

Parameters:

Name	Type	Description	Default
threshold	Optional[float]	Minimum score for term inclusion.	None
n	int	Maximum n-gram size (e.g., 1 for unigrams, 2 for bigrams, etc.).	1
stopwords	Union[str, Iterable[str], None]	Candidates that start or end with one of these words are dropped: a name in STOPWORD_LISTS, such as "english" (the default) or an ISO 639-1 code such as "fr", a list of words, or None to keep every candidate.	'english'
words_layer	Optional[str]	For Teanga input, the span layer holding the words, or None for words or, failing that, tokens.	None
lemma_layer	Optional[str]	For Teanga input, a sequence layer, based on the words layer, holding the lemma of each word. If given, every word is read as its lemma, so inflected forms such as "cell" and "cells" are counted as one term, which is reported by its lemmas. None (the default) reads the words as written.	None
min_frequency	int	Drop candidates that occur fewer times than this in the corpus. ATR4S uses 2. The default of 1 keeps every candidate.	1
word_pattern	Optional[str]	Drop candidates with a word that does not fully match this regular expression, or the name of a pattern in termxtract.candidates.WORD_PATTERNS: "alphanumeric" (the default) drops words without a letter or digit, and "atr4s" is ATR4S's noise filter. None keeps every word.	'alphanumeric'
pos_pattern	Optional[str]	For Teanga input, count only the occurrences whose part-of-speech tags match this pattern. Tags are written in angle brackets, as in "<ADJ|NOUN>*<NOUN>", or use the name of a pattern in termxtract.pos.POS_PATTERNS, such as "noun_phrase". None (the default) allows every tag sequence.	None
pos_layer	Optional[str]	The part-of-speech layer used by pos_pattern, based on the words layer: None (the default) for upos or, if the corpus has no upos layer, pos.	None

extract_terms_teanga(corpus) abstractmethod

Extract terms from a Teanga corpus.

Parameters:

Name	Type	Description	Default
corpus	Corpus	A Teanga corpus with a words span layer.	required

Returns:

Name	Type	Description
ATEResults	ATEResults	Results with ranked terms and scores, and the offsets of each term
	ATEResults	where the extractor supports them.

extract_terms_strings(corpus) abstractmethod

Extract terms from a list of strings.

Parameters:

Name	Type	Description	Default
corpus	List[str]	The documents as plain strings.	required

Returns:

Name	Type	Description
ATEResults	ATEResults	Results with ranked terms and scores.

teanga_words(doc)

Get the words of a Teanga document with their offsets.

Parameters:

Name	Type	Description	Default
doc	Document	A Teanga document with a words layer (see words_layer).	required

Returns:

Type	Description
list[tuple[int, int, str]]	List[Tuple[int, int, str]]: A (start, end, text) tuple for each word, where
list[tuple[int, int, str]]	text is the word's lemma if lemma_layer is set.

candidate_ngrams_teanga(corpus)

Find the occurrences of the candidates in a Teanga corpus.

Parameters:

Name	Type	Description	Default
corpus	Corpus	A Teanga corpus with a words layer (see words_layer) and, if pos_pattern is set, a part-of-speech layer.	required

Returns:

Type	Description
dict[str, list[tuple[str, tuple[int, int]]]]	Dict[str, List[Tuple[str, Tuple[int, int]]]]: For each document ID, every
dict[str, list[tuple[str, tuple[int, int]]]]	occurrence of a candidate, with its start and end character offsets.

candidate_ngrams_strings(docs)

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
docs	List[List[str]]	The words of each document.	required

Returns:

Type	Description
list[list[str]]	List[List[str]]: Every occurrence of a candidate in each document.

Raises:

Type	Description
ValueError	If pos_pattern is set, since plain text has no tags.

candidate_occurrences(docs, tags=None)

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
docs	List[List[str]]	The words of each document.	required
tags	Optional[List[List[str]]]	The part-of-speech tag of each word of each document, needed if pos_pattern is set.	None

Returns:

Type	Description
list[list[tuple[str, int, int]]]	List[List[Tuple[str, int, int]]]: Every occurrence of a candidate in each
list[list[tuple[str, int, int]]]	document, with its start and end (exclusive) word index.

Raises:

Type	Description
ValueError	If pos_pattern is set and there are no tags.

frequent_candidates(ngrams)

Find the candidates that occur at least min_frequency times.

Parameters:

Name	Type	Description	Default
ngrams	Iterable[str]	Every occurrence of a candidate in the corpus.	required

Returns:

Type	Description
set[str]	Set[str]: The candidates that occur often enough.

generate_ngrams_teanga(words_with_offsets)

Generate n-grams with offsets for a Teanga corpus.

Parameters:

Name	Type	Description	Default
words_with_offsets	List[Tuple[int, int, str]]	List of (start, end, text) tuples for words.	required

Returns:

Type	Description
list[tuple[str, tuple[int, int]]]	List[Tuple[str, Tuple[int, int]]]: List of n-grams with their start and end offsets.

generate_ngrams_strings(words)

Generate n-grams for a plain list of strings.

Parameters:

Name	Type	Description	Default
words	List[str]	List of words in the document.	required

Returns:

Type	Description
list[str]	List[str]: List of n-grams.

compute_tf(ngrams, total=None)

Compute the relative frequency of each n-gram in a document.

Parameters:

Name	Type	Description	Default
ngrams	List[str]	The n-grams in the document.	required
total	Optional[int]	The number to divide the counts by, or None for the number of ngrams.	None

Returns:

Type	Description
dict[str, float]	Dict[str, float]: Count of each n-gram divided by the total number of n-grams.

ngram_total(num_words)

Count the n-grams of up to n words in a document, candidates or not.

Parameters:

Name	Type	Description	Default
num_words	int	The number of words of the document.	required

Returns:

Name	Type	Description
int	int	The number of n-grams.

compute_idf(corpus_ngrams)

Compute the inverse document frequency of each n-gram as log(N / (1 + df)).

Parameters:

Name	Type	Description	Default
corpus_ngrams	List[List[str]]	The n-grams of each document.	required

Returns:

Type	Description
dict[str, float]	Dict[str, float]: IDF score for each n-gram, where N is the number of documents
dict[str, float]	and df is the number of documents containing the n-gram.

doc_terms(doc_id, scores)

Build one document's entry in ATEResults.terms.

Parameters:

Name	Type	Description	Default
doc_id	str	The document ID.	required
scores	Dict[str, float]	The score of each candidate term.	required

Returns:

Type	Description
dict[str, Any]	Dict[str, Any]: {"doc_id": doc_id, "terms": [{"term": ..., "score": ...}, ...]}, with
dict[str, Any]	terms below the threshold or starting or ending with a stopword removed, and the rest ranked from highest to lowest score.

doc_offsets(doc_id, ngrams_with_offsets, doc_terms) staticmethod

Build one document's entry in ATEResults.offsets.

Parameters:

Name	Type	Description	Default
doc_id	str	The document ID.	required
ngrams_with_offsets	Iterable[Tuple[str, Tuple[int, int]]]	Every n-gram occurrence in the document with its (start, end) character offsets.	required
doc_terms	Dict[str, Any]	The document's entry in ATEResults.terms.	required

Returns:

Type	Description
dict[str, Any]	Dict[str, Any]: {"doc_id": doc_id, "offsets": [{"term": ..., "offset": (start, end)}, ...]},
dict[str, Any]	listing each occurrence of an extracted term in document order.

NestedTermExtractor

C-value, Basic and ComboBasic score candidates by their frequency and by how they are nested in each other, over the whole corpus. They inherit from NestedTermExtractor, which counts the candidates, finds which contain which, and leaves the formula to score_terms.

NestedTermExtractor(threshold=None, n=1, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)

Bases: BaseTermExtractor

Base class for extractors that score candidates by their frequency and by how they are nested in other candidates: C-value, Basic and ComboBasic.

Subclasses implement score_terms. Frequencies and nesting are counted over the whole corpus, so a term has the same score in every document it occurs in. One candidate is nested in another when its words are a contiguous part of the other's words, so "heart" is nested in "heart failure" but not in "hearth". Only candidates, which pass the filters of BaseTermExtractor, are counted as containing others, and only their occurrences that pass the filters are counted.

Initialize the extractor.

The candidates are the n-grams of the corpus that pass the filters set by stopwords, word_pattern, pos_pattern and min_frequency. Only the occurrences that pass them are counted, so they change the scores as well as which terms are returned.

Parameters:

Name	Type	Description	Default
threshold	Optional[float]	Minimum score for term inclusion.	None
n	int	Maximum n-gram size (e.g., 1 for unigrams, 2 for bigrams, etc.).	1
stopwords	Union[str, Iterable[str], None]	Candidates that start or end with one of these words are dropped: a name in STOPWORD_LISTS, such as "english" (the default) or an ISO 639-1 code such as "fr", a list of words, or None to keep every candidate.	'english'
words_layer	Optional[str]	For Teanga input, the span layer holding the words, or None for words or, failing that, tokens.	None
lemma_layer	Optional[str]	For Teanga input, a sequence layer, based on the words layer, holding the lemma of each word. If given, every word is read as its lemma, so inflected forms such as "cell" and "cells" are counted as one term, which is reported by its lemmas. None (the default) reads the words as written.	None
min_frequency	int	Drop candidates that occur fewer times than this in the corpus. ATR4S uses 2. The default of 1 keeps every candidate.	1
word_pattern	Optional[str]	Drop candidates with a word that does not fully match this regular expression, or the name of a pattern in termxtract.candidates.WORD_PATTERNS: "alphanumeric" (the default) drops words without a letter or digit, and "atr4s" is ATR4S's noise filter. None keeps every word.	'alphanumeric'
pos_pattern	Optional[str]	For Teanga input, count only the occurrences whose part-of-speech tags match this pattern. Tags are written in angle brackets, as in "<ADJ|NOUN>*<NOUN>", or use the name of a pattern in termxtract.pos.POS_PATTERNS, such as "noun_phrase". None (the default) allows every tag sequence.	None
pos_layer	Optional[str]	The part-of-speech layer used by pos_pattern, based on the words layer: None (the default) for upos or, if the corpus has no upos layer, pos.	None

score_terms(frequencies, containing) abstractmethod

Score the candidates of the corpus.

Parameters:

Name	Type	Description	Default
frequencies	Counter[str]	The number of occurrences of each candidate in the corpus.	required
containing	Dict[str, Set[str]]	For each candidate, the other candidates it is nested in.	required

Returns:

Type	Description
dict[str, float]	Dict[str, float]: The score of each candidate.

nesting(candidates)

Find the candidates that each candidate is nested in.

Parameters:

Name	Type	Description	Default
candidates	Set[str]	The candidates, with words separated by spaces.	required

Returns:

Type	Description
dict[str, set[str]]	Dict[str, Set[str]]: For each candidate, the longer candidates that contain
dict[str, set[str]]	its words as a contiguous sequence.

corpus_scores(corpus_ngrams)

Count the candidates of a corpus and score them with score_terms.

Parameters:

Name	Type	Description	Default
corpus_ngrams	List[List[str]]	The occurrences of candidates in each document, from candidate_ngrams_teanga or candidate_ngrams_strings.	required

Returns:

Type	Description
dict[str, float]	Dict[str, float]: The score of each candidate.

extract_terms_teanga(corpus)

Extract terms from a Teanga corpus.

Parameters:

Name	Type	Description	Default
corpus	Corpus	A Teanga Corpus object.	required

Returns:

Name	Type	Description
ATEResults	ATEResults	Results with ranked terms and scores, and term offsets.

extract_terms_strings(corpus)

Extract terms from a plain list of strings.

Parameters:

Name	Type	Description	Default
corpus	List[str]	List of documents as strings.	required

Returns:

Name	Type	Description
ATEResults	ATEResults	Results with ranked terms and scores.

teanga_words(doc)

Get the words of a Teanga document with their offsets.

Parameters:

Name	Type	Description	Default
doc	Document	A Teanga document with a words layer (see words_layer).	required

Returns:

Type	Description
list[tuple[int, int, str]]	List[Tuple[int, int, str]]: A (start, end, text) tuple for each word, where
list[tuple[int, int, str]]	text is the word's lemma if lemma_layer is set.

candidate_ngrams_teanga(corpus)

Find the occurrences of the candidates in a Teanga corpus.

Parameters:

Name	Type	Description	Default
corpus	Corpus	A Teanga corpus with a words layer (see words_layer) and, if pos_pattern is set, a part-of-speech layer.	required

Returns:

Type	Description
dict[str, list[tuple[str, tuple[int, int]]]]	Dict[str, List[Tuple[str, Tuple[int, int]]]]: For each document ID, every
dict[str, list[tuple[str, tuple[int, int]]]]	occurrence of a candidate, with its start and end character offsets.

candidate_ngrams_strings(docs)

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
docs	List[List[str]]	The words of each document.	required

Returns:

Type	Description
list[list[str]]	List[List[str]]: Every occurrence of a candidate in each document.

Raises:

Type	Description
ValueError	If pos_pattern is set, since plain text has no tags.

candidate_occurrences(docs, tags=None)

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
docs	List[List[str]]	The words of each document.	required
tags	Optional[List[List[str]]]	The part-of-speech tag of each word of each document, needed if pos_pattern is set.	None

Returns:

Type	Description
list[list[tuple[str, int, int]]]	List[List[Tuple[str, int, int]]]: Every occurrence of a candidate in each
list[list[tuple[str, int, int]]]	document, with its start and end (exclusive) word index.

Raises:

Type	Description
ValueError	If pos_pattern is set and there are no tags.

frequent_candidates(ngrams)

Find the candidates that occur at least min_frequency times.

Parameters:

Name	Type	Description	Default
ngrams	Iterable[str]	Every occurrence of a candidate in the corpus.	required

Returns:

Type	Description
set[str]	Set[str]: The candidates that occur often enough.

generate_ngrams_teanga(words_with_offsets)

Generate n-grams with offsets for a Teanga corpus.

Parameters:

Name	Type	Description	Default
words_with_offsets	List[Tuple[int, int, str]]	List of (start, end, text) tuples for words.	required

Returns:

Type	Description
list[tuple[str, tuple[int, int]]]	List[Tuple[str, Tuple[int, int]]]: List of n-grams with their start and end offsets.

generate_ngrams_strings(words)

Generate n-grams for a plain list of strings.

Parameters:

Name	Type	Description	Default
words	List[str]	List of words in the document.	required

Returns:

Type	Description
list[str]	List[str]: List of n-grams.

compute_tf(ngrams, total=None)

Compute the relative frequency of each n-gram in a document.

Parameters:

Name	Type	Description	Default
ngrams	List[str]	The n-grams in the document.	required
total	Optional[int]	The number to divide the counts by, or None for the number of ngrams.	None

Returns:

Type	Description
dict[str, float]	Dict[str, float]: Count of each n-gram divided by the total number of n-grams.

ngram_total(num_words)

Count the n-grams of up to n words in a document, candidates or not.

Parameters:

Name	Type	Description	Default
num_words	int	The number of words of the document.	required

Returns:

Name	Type	Description
int	int	The number of n-grams.

compute_idf(corpus_ngrams)

Compute the inverse document frequency of each n-gram as log(N / (1 + df)).

Parameters:

Name	Type	Description	Default
corpus_ngrams	List[List[str]]	The n-grams of each document.	required

Returns:

Type	Description
dict[str, float]	Dict[str, float]: IDF score for each n-gram, where N is the number of documents
dict[str, float]	and df is the number of documents containing the n-gram.

doc_terms(doc_id, scores)

Build one document's entry in ATEResults.terms.

Parameters:

Name	Type	Description	Default
doc_id	str	The document ID.	required
scores	Dict[str, float]	The score of each candidate term.	required

Returns:

Type	Description
dict[str, Any]	Dict[str, Any]: {"doc_id": doc_id, "terms": [{"term": ..., "score": ...}, ...]}, with
dict[str, Any]	terms below the threshold or starting or ending with a stopword removed, and the rest ranked from highest to lowest score.

doc_offsets(doc_id, ngrams_with_offsets, doc_terms) staticmethod

Build one document's entry in ATEResults.offsets.

Parameters:

Name	Type	Description	Default
doc_id	str	The document ID.	required
ngrams_with_offsets	Iterable[Tuple[str, Tuple[int, int]]]	Every n-gram occurrence in the document with its (start, end) character offsets.	required
doc_terms	Dict[str, Any]	The document's entry in ATEResults.terms.	required

Returns:

Type	Description
dict[str, Any]	Dict[str, Any]: {"doc_id": doc_id, "offsets": [{"term": ..., "offset": (start, end)}, ...]},
dict[str, Any]	listing each occurrence of an extracted term in document order.