Basic

Used by TermExtractor with method="basic".

`BasicTermExtractor(alpha=0.5, threshold=None, n=1, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)`

Bases: NestedTermExtractor

Basic term extraction (Bordea et al., 2013), which favours candidates that are part of many other candidates. Frequencies and nesting are counted over the corpus.

Initialize the Basic extractor.

Parameters:

Name	Type	Description	Default
`alpha`	`float`	Weight for ( e_t ), the number of candidates containing ( t ). ATR4S uses 0.72, from Astrakhantsev (2015), and Bordea et al. (2013) use 3.5; the results of the English benchmark are within a point of those with the default of 0.5.	`0.5`
`threshold`	`Optional[float]`	Minimum score for term inclusion.	`None`
`n`	`int`	Maximum n-gram size (e.g., 1 for unigrams, 2 for bigrams, etc.).	`1`
`stopwords`	`Union[str, Iterable[str], None]`	Drop candidates that start or end with a stopword: a name in `STOPWORD_LISTS`, a list of words, or None.	`'english'`
`words_layer`	`Optional[str]`	For Teanga input, the span layer holding the words, or None for `words` or, failing that, `tokens`.	`None`
`lemma_layer`	`Optional[str]`	For Teanga input, a sequence layer holding the lemma of each word, to count inflected forms such as "cell" and "cells" as one term, or None to use the words as written.	`None`
`min_frequency`	`int`	Drop candidates that occur fewer times in the corpus.	`1`
`word_pattern`	`Optional[str]`	Drop candidates with a word that does not fully match this regular expression or named pattern ("alphanumeric", "atr4s"), or None to keep every word.	`'alphanumeric'`
`pos_pattern`	`Optional[str]`	For Teanga input, count only the occurrences whose part-of-speech tags match this pattern, or None.	`None`
`pos_layer`	`Optional[str]`	The part-of-speech layer for `pos_pattern`, or None for `upos` or, failing that, `pos`.	`None`

`score_terms(frequencies, containing)`

Compute the Basic score of each candidate: ( |t| \log f(t) + \alpha e_t ), where ( e_t ) is the number of candidates that contain ( t ), or 0 if ( t ) is a single word.

Parameters:

Name	Type	Description	Default
`frequencies`	`Counter[str]`	The number of occurrences of each candidate in the corpus.	required
`containing`	`Dict[str, Set[str]]`	For each candidate, the other candidates it is nested in.	required

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: Basic score for each candidate.

`extract_terms_teanga(corpus)`

Extract terms from a Teanga corpus.

Parameters:

Name	Type	Description	Default
`corpus`	`Corpus`	A Teanga Corpus object.	required

Returns:

Name	Type	Description
`ATEResults`	`ATEResults`	Results with ranked terms and scores, and term offsets.

`extract_terms_strings(corpus)`

Extract terms from a plain list of strings.

Parameters:

Name	Type	Description	Default
`corpus`	`List[str]`	List of documents as strings.	required

Returns:

Name	Type	Description
`ATEResults`	`ATEResults`	Results with ranked terms and scores.

`teanga_words(doc)`

Get the words of a Teanga document with their offsets.

Parameters:

Name	Type	Description	Default
`doc`	`Document`	A Teanga document with a words layer (see `words_layer`).	required

Returns:

Type	Description
`list[tuple[int, int, str]]`	List[Tuple[int, int, str]]: A (start, end, text) tuple for each word, where
`list[tuple[int, int, str]]`	text is the word's lemma if `lemma_layer` is set.

`candidate_ngrams_teanga(corpus)`

Find the occurrences of the candidates in a Teanga corpus.

Parameters:

Name	Type	Description	Default
`corpus`	`Corpus`	A Teanga corpus with a words layer (see `words_layer`) and, if `pos_pattern` is set, a part-of-speech layer.	required

Returns:

Type	Description
`dict[str, list[tuple[str, tuple[int, int]]]]`	Dict[str, List[Tuple[str, Tuple[int, int]]]]: For each document ID, every
`dict[str, list[tuple[str, tuple[int, int]]]]`	occurrence of a candidate, with its start and end character offsets.

`candidate_ngrams_strings(docs)`

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
`docs`	`List[List[str]]`	The words of each document.	required

Returns:

Type	Description
`list[list[str]]`	List[List[str]]: Every occurrence of a candidate in each document.

Raises:

Type	Description
`ValueError`	If `pos_pattern` is set, since plain text has no tags.

`candidate_occurrences(docs, tags=None)`

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
`docs`	`List[List[str]]`	The words of each document.	required
`tags`	`Optional[List[List[str]]]`	The part-of-speech tag of each word of each document, needed if `pos_pattern` is set.	`None`

Returns:

Type	Description
`list[list[tuple[str, int, int]]]`	List[List[Tuple[str, int, int]]]: Every occurrence of a candidate in each
`list[list[tuple[str, int, int]]]`	document, with its start and end (exclusive) word index.

Raises:

Type	Description
`ValueError`	If `pos_pattern` is set and there are no tags.

`frequent_candidates(ngrams)`

Find the candidates that occur at least min_frequency times.

Parameters:

Name	Type	Description	Default
`ngrams`	`Iterable[str]`	Every occurrence of a candidate in the corpus.	required

Returns:

Type	Description
`set[str]`	Set[str]: The candidates that occur often enough.

`generate_ngrams_teanga(words_with_offsets)`

Generate n-grams with offsets for a Teanga corpus.

Parameters:

Name	Type	Description	Default
`words_with_offsets`	`List[Tuple[int, int, str]]`	List of (start, end, text) tuples for words.	required

Returns:

Type	Description
`list[tuple[str, tuple[int, int]]]`	List[Tuple[str, Tuple[int, int]]]: List of n-grams with their start and end offsets.

`generate_ngrams_strings(words)`

Generate n-grams for a plain list of strings.

Parameters:

Name	Type	Description	Default
`words`	`List[str]`	List of words in the document.	required

Returns:

Type	Description
`list[str]`	List[str]: List of n-grams.

`compute_tf(ngrams, total=None)`

Compute the relative frequency of each n-gram in a document.

Parameters:

Name	Type	Description	Default
`ngrams`	`List[str]`	The n-grams in the document.	required
`total`	`Optional[int]`	The number to divide the counts by, or None for the number of `ngrams`.	`None`

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: Count of each n-gram divided by the total number of n-grams.

`ngram_total(num_words)`

Count the n-grams of up to n words in a document, candidates or not.

Parameters:

Name	Type	Description	Default
`num_words`	`int`	The number of words of the document.	required

Returns:

Name	Type	Description
`int`	`int`	The number of n-grams.

`compute_idf(corpus_ngrams)`

Compute the inverse document frequency of each n-gram as log(N / (1 + df)).

Parameters:

Name	Type	Description	Default
`corpus_ngrams`	`List[List[str]]`	The n-grams of each document.	required

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: IDF score for each n-gram, where N is the number of documents
`dict[str, float]`	and df is the number of documents containing the n-gram.

`doc_terms(doc_id, scores)`

Build one document's entry in ATEResults.terms.

Parameters:

Name	Type	Description	Default
`doc_id`	`str`	The document ID.	required
`scores`	`Dict[str, float]`	The score of each candidate term.	required

Returns:

Type	Description
`dict[str, Any]`	Dict[str, Any]: `{"doc_id": doc_id, "terms": [{"term": ..., "score": ...}, ...]}`, with
`dict[str, Any]`	terms below the threshold or starting or ending with a stopword removed, and the rest ranked from highest to lowest score.

`doc_offsets(doc_id, ngrams_with_offsets, doc_terms)` `staticmethod`

Build one document's entry in ATEResults.offsets.

Parameters:

Name	Type	Description	Default
`doc_id`	`str`	The document ID.	required
`ngrams_with_offsets`	`Iterable[Tuple[str, Tuple[int, int]]]`	Every n-gram occurrence in the document with its (start, end) character offsets.	required
`doc_terms`	`Dict[str, Any]`	The document's entry in `ATEResults.terms`.	required

Returns:

Type	Description
`dict[str, Any]`	Dict[str, Any]: `{"doc_id": doc_id, "offsets": [{"term": ..., "offset": (start, end)}, ...]}`,
`dict[str, Any]`	listing each occurrence of an extracted term in document order.

`nesting(candidates)`

Find the candidates that each candidate is nested in.

Parameters:

Name	Type	Description	Default
`candidates`	`Set[str]`	The candidates, with words separated by spaces.	required

Returns:

Type	Description
`dict[str, set[str]]`	Dict[str, Set[str]]: For each candidate, the longer candidates that contain
`dict[str, set[str]]`	its words as a contiguous sequence.

`corpus_scores(corpus_ngrams)`

Count the candidates of a corpus and score them with score_terms.

Parameters:

Name	Type	Description	Default
`corpus_ngrams`	`List[List[str]]`	The occurrences of candidates in each document, from `candidate_ngrams_teanga` or `candidate_ngrams_strings`.	required

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: The score of each candidate.

Basic

BasicTermExtractor(alpha=0.5, threshold=None, n=1, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)

score_terms(frequencies, containing)

extract_terms_teanga(corpus)

extract_terms_strings(corpus)

teanga_words(doc)

candidate_ngrams_teanga(corpus)

candidate_ngrams_strings(docs)

candidate_occurrences(docs, tags=None)

frequent_candidates(ngrams)

generate_ngrams_teanga(words_with_offsets)

generate_ngrams_strings(words)

compute_tf(ngrams, total=None)

ngram_total(num_words)

compute_idf(corpus_ngrams)

doc_terms(doc_id, scores)

doc_offsets(doc_id, ngrams_with_offsets, doc_terms) staticmethod

nesting(candidates)

corpus_scores(corpus_ngrams)

`BasicTermExtractor(alpha=0.5, threshold=None, n=1, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)`

`score_terms(frequencies, containing)`

`extract_terms_teanga(corpus)`

`extract_terms_strings(corpus)`

`teanga_words(doc)`

`candidate_ngrams_teanga(corpus)`

`candidate_ngrams_strings(docs)`

`candidate_occurrences(docs, tags=None)`

`frequent_candidates(ngrams)`

`generate_ngrams_teanga(words_with_offsets)`

`generate_ngrams_strings(words)`

`compute_tf(ngrams, total=None)`

`ngram_total(num_words)`

`compute_idf(corpus_ngrams)`

`doc_terms(doc_id, scores)`

`doc_offsets(doc_id, ngrams_with_offsets, doc_terms)` `staticmethod`

`nesting(candidates)`

`corpus_scores(corpus_ngrams)`