TF-IDF

Used by TermExtractor with method="tfidf".

`TFIDFTermExtractor(threshold=None, n=1, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)`

Bases: BaseTermExtractor

TF-IDF based term extraction with n-gram support and a threshold option.

The term frequency of a candidate is its number of occurrences in a document divided by the number of n-grams of the document, candidates or not, so the filters on candidates do not change the scores of the candidates whose occurrences all pass.

Initialize the extractor.

The candidates are the n-grams of the corpus that pass the filters set by stopwords, word_pattern, pos_pattern and min_frequency. Only the occurrences that pass them are counted, so they change the scores as well as which terms are returned.

Parameters:

Name	Type	Description	Default
`threshold`	`Optional[float]`	Minimum score for term inclusion.	`None`
`n`	`int`	Maximum n-gram size (e.g., 1 for unigrams, 2 for bigrams, etc.).	`1`
`stopwords`	`Union[str, Iterable[str], None]`	Candidates that start or end with one of these words are dropped: a name in `STOPWORD_LISTS`, such as "english" (the default) or an ISO 639-1 code such as "fr", a list of words, or None to keep every candidate.	`'english'`
`words_layer`	`Optional[str]`	For Teanga input, the span layer holding the words, or None for `words` or, failing that, `tokens`.	`None`
`lemma_layer`	`Optional[str]`	For Teanga input, a sequence layer, based on the words layer, holding the lemma of each word. If given, every word is read as its lemma, so inflected forms such as "cell" and "cells" are counted as one term, which is reported by its lemmas. None (the default) reads the words as written.	`None`
`min_frequency`	`int`	Drop candidates that occur fewer times than this in the corpus. ATR4S uses 2. The default of 1 keeps every candidate.	`1`
`word_pattern`	`Optional[str]`	Drop candidates with a word that does not fully match this regular expression, or the name of a pattern in `termxtract.candidates.WORD_PATTERNS`: "alphanumeric" (the default) drops words without a letter or digit, and "atr4s" is ATR4S's noise filter. None keeps every word.	`'alphanumeric'`
`pos_pattern`	`Optional[str]`	For Teanga input, count only the occurrences whose part-of-speech tags match this pattern. Tags are written in angle brackets, as in `"<ADJ\|NOUN>*<NOUN>"`, or use the name of a pattern in `termxtract.pos.POS_PATTERNS`, such as `"noun_phrase"`. None (the default) allows every tag sequence.	`None`
`pos_layer`	`Optional[str]`	The part-of-speech layer used by `pos_pattern`, based on the words layer: None (the default) for `upos` or, if the corpus has no `upos` layer, `pos`.	`None`

`extract_terms_teanga(corpus)`

Extract terms from a Teanga corpus using TF-IDF.

Parameters:

Name	Type	Description	Default
`corpus`	`Corpus`	A Teanga corpus with a `words` span layer.	required

Returns:

Name	Type	Description
`ATEResults`	`ATEResults`	Results with ranked terms and scores, and the character offsets of
	`ATEResults`	each occurrence of each term.

`extract_terms_strings(corpus)`

Extract terms from a list of strings using TF-IDF.

Parameters:

Name	Type	Description	Default
`corpus`	`List[str]`	The documents as plain strings.	required

Returns:

Name	Type	Description
`ATEResults`	`ATEResults`	Results with ranked terms and scores.

`teanga_words(doc)`

Get the words of a Teanga document with their offsets.

Parameters:

Name	Type	Description	Default
`doc`	`Document`	A Teanga document with a words layer (see `words_layer`).	required

Returns:

Type	Description
`list[tuple[int, int, str]]`	List[Tuple[int, int, str]]: A (start, end, text) tuple for each word, where
`list[tuple[int, int, str]]`	text is the word's lemma if `lemma_layer` is set.

`candidate_ngrams_teanga(corpus)`

Find the occurrences of the candidates in a Teanga corpus.

Parameters:

Name	Type	Description	Default
`corpus`	`Corpus`	A Teanga corpus with a words layer (see `words_layer`) and, if `pos_pattern` is set, a part-of-speech layer.	required

Returns:

Type	Description
`dict[str, list[tuple[str, tuple[int, int]]]]`	Dict[str, List[Tuple[str, Tuple[int, int]]]]: For each document ID, every
`dict[str, list[tuple[str, tuple[int, int]]]]`	occurrence of a candidate, with its start and end character offsets.

`candidate_ngrams_strings(docs)`

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
`docs`	`List[List[str]]`	The words of each document.	required

Returns:

Type	Description
`list[list[str]]`	List[List[str]]: Every occurrence of a candidate in each document.

Raises:

Type	Description
`ValueError`	If `pos_pattern` is set, since plain text has no tags.

`candidate_occurrences(docs, tags=None)`

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
`docs`	`List[List[str]]`	The words of each document.	required
`tags`	`Optional[List[List[str]]]`	The part-of-speech tag of each word of each document, needed if `pos_pattern` is set.	`None`

Returns:

Type	Description
`list[list[tuple[str, int, int]]]`	List[List[Tuple[str, int, int]]]: Every occurrence of a candidate in each
`list[list[tuple[str, int, int]]]`	document, with its start and end (exclusive) word index.

Raises:

Type	Description
`ValueError`	If `pos_pattern` is set and there are no tags.

`frequent_candidates(ngrams)`

Find the candidates that occur at least min_frequency times.

Parameters:

Name	Type	Description	Default
`ngrams`	`Iterable[str]`	Every occurrence of a candidate in the corpus.	required

Returns:

Type	Description
`set[str]`	Set[str]: The candidates that occur often enough.

`generate_ngrams_teanga(words_with_offsets)`

Generate n-grams with offsets for a Teanga corpus.

Parameters:

Name	Type	Description	Default
`words_with_offsets`	`List[Tuple[int, int, str]]`	List of (start, end, text) tuples for words.	required

Returns:

Type	Description
`list[tuple[str, tuple[int, int]]]`	List[Tuple[str, Tuple[int, int]]]: List of n-grams with their start and end offsets.

`generate_ngrams_strings(words)`

Generate n-grams for a plain list of strings.

Parameters:

Name	Type	Description	Default
`words`	`List[str]`	List of words in the document.	required

Returns:

Type	Description
`list[str]`	List[str]: List of n-grams.

`compute_tf(ngrams, total=None)`

Compute the relative frequency of each n-gram in a document.

Parameters:

Name	Type	Description	Default
`ngrams`	`List[str]`	The n-grams in the document.	required
`total`	`Optional[int]`	The number to divide the counts by, or None for the number of `ngrams`.	`None`

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: Count of each n-gram divided by the total number of n-grams.

`ngram_total(num_words)`

Count the n-grams of up to n words in a document, candidates or not.

Parameters:

Name	Type	Description	Default
`num_words`	`int`	The number of words of the document.	required

Returns:

Name	Type	Description
`int`	`int`	The number of n-grams.

`compute_idf(corpus_ngrams)`

Compute the inverse document frequency of each n-gram as log(N / (1 + df)).

Parameters:

Name	Type	Description	Default
`corpus_ngrams`	`List[List[str]]`	The n-grams of each document.	required

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: IDF score for each n-gram, where N is the number of documents
`dict[str, float]`	and df is the number of documents containing the n-gram.

`doc_terms(doc_id, scores)`

Build one document's entry in ATEResults.terms.

Parameters:

Name	Type	Description	Default
`doc_id`	`str`	The document ID.	required
`scores`	`Dict[str, float]`	The score of each candidate term.	required

Returns:

Type	Description
`dict[str, Any]`	Dict[str, Any]: `{"doc_id": doc_id, "terms": [{"term": ..., "score": ...}, ...]}`, with
`dict[str, Any]`	terms below the threshold or starting or ending with a stopword removed, and the rest ranked from highest to lowest score.

`doc_offsets(doc_id, ngrams_with_offsets, doc_terms)` `staticmethod`

Build one document's entry in ATEResults.offsets.

Parameters:

Name	Type	Description	Default
`doc_id`	`str`	The document ID.	required
`ngrams_with_offsets`	`Iterable[Tuple[str, Tuple[int, int]]]`	Every n-gram occurrence in the document with its (start, end) character offsets.	required
`doc_terms`	`Dict[str, Any]`	The document's entry in `ATEResults.terms`.	required

Returns:

Type	Description
`dict[str, Any]`	Dict[str, Any]: `{"doc_id": doc_id, "offsets": [{"term": ..., "offset": (start, end)}, ...]}`,
`dict[str, Any]`	listing each occurrence of an extracted term in document order.

TF-IDF

TFIDFTermExtractor(threshold=None, n=1, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)

extract_terms_teanga(corpus)

extract_terms_strings(corpus)

teanga_words(doc)

candidate_ngrams_teanga(corpus)

candidate_ngrams_strings(docs)

candidate_occurrences(docs, tags=None)

frequent_candidates(ngrams)

generate_ngrams_teanga(words_with_offsets)

generate_ngrams_strings(words)

compute_tf(ngrams, total=None)

ngram_total(num_words)

compute_idf(corpus_ngrams)

doc_terms(doc_id, scores)

doc_offsets(doc_id, ngrams_with_offsets, doc_terms) staticmethod

`TFIDFTermExtractor(threshold=None, n=1, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)`

`extract_terms_teanga(corpus)`

`extract_terms_strings(corpus)`

`teanga_words(doc)`

`candidate_ngrams_teanga(corpus)`

`candidate_ngrams_strings(docs)`

`candidate_occurrences(docs, tags=None)`

`frequent_candidates(ngrams)`

`generate_ngrams_teanga(words_with_offsets)`

`generate_ngrams_strings(words)`

`compute_tf(ngrams, total=None)`

`ngram_total(num_words)`

`compute_idf(corpus_ngrams)`

`doc_terms(doc_id, scores)`

`doc_offsets(doc_id, ngrams_with_offsets, doc_terms)` `staticmethod`