RAKE

Used by TermExtractor with method="rake".

RAKETermExtractor(stoplist, phrase_delimiters, threshold=None)

Bases: BaseTermExtractor

RAKE-based term extraction with support for n-grams and keyword linking.

Initialize the RAKE extractor.

Parameters:

Name	Type	Description	Default
stoplist	List[str]	List of stop words (required).	required
phrase_delimiters	List[str]	List of delimiters to split phrases (e.g., punctuation) (required).	required
threshold	Optional[float]	Minimum score for term inclusion.	None

generate_candidate_keywords(text)

Generate candidate keywords by splitting on stop words and phrase delimiters.

Parameters:

Name	Type	Description	Default
text	str	Document text.	required

Returns:

Type	Description
list[list[str]]	List[List[str]]: Candidate keywords as lists of words.

compute_word_scores(candidates)

Compute scores for individual words based on degree and frequency.

Parameters:

Name	Type	Description	Default
candidates	List[List[str]]	Candidate keywords as lists of words.	required

Returns:

Type	Description
dict[str, float]	Dict[str, float]: Word scores.

compute_candidate_scores(candidates, word_scores)

Compute scores for candidate keywords.

Parameters:

Name	Type	Description	Default
candidates	List[List[str]]	Candidate keywords as lists of words.	required
word_scores	Dict[str, float]	Scores for individual words.	required

Returns:

Type	Description
dict[str, float]	Dict[str, float]: Candidate keyword scores.

link_adjoining_keywords(text, candidate_scores)

Link adjoining keywords with interior stop words.

Parameters:

Name	Type	Description	Default
text	str	Document text.	required
candidate_scores	Dict[str, float]	Scores for original candidate keywords.	required

Returns:

Type	Description
dict[str, float]	Dict[str, float]: Updated keyword scores including linked keywords.

extract_terms_teanga(corpus)

Extract terms from a Teanga corpus using RAKE.

Parameters:

Name	Type	Description	Default
corpus	Corpus	A Teanga Corpus object.	required

Returns:

Name	Type	Description
ATEResults	ATEResults	Results with ranked terms and scores, and the character offsets of
	ATEResults	each occurrence of each term.

extract_terms_strings(corpus)

Extract terms from a plain list of strings using RAKE.

Parameters:

Name	Type	Description	Default
corpus	List[str]	List of documents as strings.	required

Returns:

Name	Type	Description
ATEResults	ATEResults	Results with ranked terms and scores.

teanga_words(doc)

Get the words of a Teanga document with their offsets.

Parameters:

Name	Type	Description	Default
doc	Document	A Teanga document with a words layer (see words_layer).	required

Returns:

Type	Description
list[tuple[int, int, str]]	List[Tuple[int, int, str]]: A (start, end, text) tuple for each word, where
list[tuple[int, int, str]]	text is the word's lemma if lemma_layer is set.

candidate_ngrams_teanga(corpus)

Find the occurrences of the candidates in a Teanga corpus.

Parameters:

Name	Type	Description	Default
corpus	Corpus	A Teanga corpus with a words layer (see words_layer) and, if pos_pattern is set, a part-of-speech layer.	required

Returns:

Type	Description
dict[str, list[tuple[str, tuple[int, int]]]]	Dict[str, List[Tuple[str, Tuple[int, int]]]]: For each document ID, every
dict[str, list[tuple[str, tuple[int, int]]]]	occurrence of a candidate, with its start and end character offsets.

candidate_ngrams_strings(docs)

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
docs	List[List[str]]	The words of each document.	required

Returns:

Type	Description
list[list[str]]	List[List[str]]: Every occurrence of a candidate in each document.

Raises:

Type	Description
ValueError	If pos_pattern is set, since plain text has no tags.

candidate_occurrences(docs, tags=None)

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
docs	List[List[str]]	The words of each document.	required
tags	Optional[List[List[str]]]	The part-of-speech tag of each word of each document, needed if pos_pattern is set.	None

Returns:

Type	Description
list[list[tuple[str, int, int]]]	List[List[Tuple[str, int, int]]]: Every occurrence of a candidate in each
list[list[tuple[str, int, int]]]	document, with its start and end (exclusive) word index.

Raises:

Type	Description
ValueError	If pos_pattern is set and there are no tags.

frequent_candidates(ngrams)

Find the candidates that occur at least min_frequency times.

Parameters:

Name	Type	Description	Default
ngrams	Iterable[str]	Every occurrence of a candidate in the corpus.	required

Returns:

Type	Description
set[str]	Set[str]: The candidates that occur often enough.

generate_ngrams_teanga(words_with_offsets)

Generate n-grams with offsets for a Teanga corpus.

Parameters:

Name	Type	Description	Default
words_with_offsets	List[Tuple[int, int, str]]	List of (start, end, text) tuples for words.	required

Returns:

Type	Description
list[tuple[str, tuple[int, int]]]	List[Tuple[str, Tuple[int, int]]]: List of n-grams with their start and end offsets.

generate_ngrams_strings(words)

Generate n-grams for a plain list of strings.

Parameters:

Name	Type	Description	Default
words	List[str]	List of words in the document.	required

Returns:

Type	Description
list[str]	List[str]: List of n-grams.

compute_tf(ngrams, total=None)

Compute the relative frequency of each n-gram in a document.

Parameters:

Name	Type	Description	Default
ngrams	List[str]	The n-grams in the document.	required
total	Optional[int]	The number to divide the counts by, or None for the number of ngrams.	None

Returns:

Type	Description
dict[str, float]	Dict[str, float]: Count of each n-gram divided by the total number of n-grams.

ngram_total(num_words)

Count the n-grams of up to n words in a document, candidates or not.

Parameters:

Name	Type	Description	Default
num_words	int	The number of words of the document.	required

Returns:

Name	Type	Description
int	int	The number of n-grams.

compute_idf(corpus_ngrams)

Compute the inverse document frequency of each n-gram as log(N / (1 + df)).

Parameters:

Name	Type	Description	Default
corpus_ngrams	List[List[str]]	The n-grams of each document.	required

Returns:

Type	Description
dict[str, float]	Dict[str, float]: IDF score for each n-gram, where N is the number of documents
dict[str, float]	and df is the number of documents containing the n-gram.

doc_terms(doc_id, scores)

Build one document's entry in ATEResults.terms.

Parameters:

Name	Type	Description	Default
doc_id	str	The document ID.	required
scores	Dict[str, float]	The score of each candidate term.	required

Returns:

Type	Description
dict[str, Any]	Dict[str, Any]: {"doc_id": doc_id, "terms": [{"term": ..., "score": ...}, ...]}, with
dict[str, Any]	terms below the threshold or starting or ending with a stopword removed, and the rest ranked from highest to lowest score.

doc_offsets(doc_id, ngrams_with_offsets, doc_terms) staticmethod

Build one document's entry in ATEResults.offsets.

Parameters:

Name	Type	Description	Default
doc_id	str	The document ID.	required
ngrams_with_offsets	Iterable[Tuple[str, Tuple[int, int]]]	Every n-gram occurrence in the document with its (start, end) character offsets.	required
doc_terms	Dict[str, Any]	The document's entry in ATEResults.terms.	required

Returns:

Type	Description
dict[str, Any]	Dict[str, Any]: {"doc_id": doc_id, "offsets": [{"term": ..., "offset": (start, end)}, ...]},
dict[str, Any]	listing each occurrence of an extracted term in document order.