Weirdness

Used by TermExtractor with method="weirdness".

`WeirdnessTermExtractor(reference_corpus, threshold=None, n=1, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)`

Bases: ReferenceCorpusTermExtractor

Weirdness-based term extraction with n-gram support (Ahmad et al., 1999).

Each n-gram is scored as p_target(t) / p_ref(t): how many times more frequent it is in the target corpus than in the reference corpus. Frequencies are relative to the number of n-grams in each corpus, and the reference count is add-one smoothed, p_ref(t) = (count_ref(t) + 1) / N_ref, so n-grams missing from the reference corpus get a finite score.

Initialize the extractor.

Parameters:

Name	Type	Description	Default
`reference_corpus`	`Union[Corpus, List[str], List[Union[Corpus, List[str]]]]`	General-language reference corpus, of the same type as the corpus the terms are extracted from, or a list of reference corpora.	required
`threshold`	`Optional[float]`	Minimum score for term inclusion.	`None`
`n`	`int`	Maximum n-gram size (e.g., 1 for unigrams, 2 for bigrams, etc.).	`1`
`stopwords`	`Union[str, Iterable[str], None]`	Drop candidates that start or end with a stopword: a name in `STOPWORD_LISTS`, a list of words, or None.	`'english'`
`words_layer`	`Optional[str]`	For Teanga input, the span layer holding the words, or None for `words` or, failing that, `tokens`.	`None`
`lemma_layer`	`Optional[str]`	For Teanga input, a sequence layer holding the lemma of each word, to count inflected forms such as "cell" and "cells" as one term, or None to use the words as written.	`None`
`min_frequency`	`int`	Drop candidates that occur fewer times in the corpus.	`1`
`word_pattern`	`Optional[str]`	Drop candidates with a word that does not fully match this regular expression or named pattern ("alphanumeric", "atr4s"), or None to keep every word.	`'alphanumeric'`
`pos_pattern`	`Optional[str]`	For Teanga input, count only the occurrences whose part-of-speech tags match this pattern, or None.	`None`
`pos_layer`	`Optional[str]`	The part-of-speech layer for `pos_pattern`, or None for `upos` or, failing that, `pos`.	`None`

`score_terms(target_ngrams, ref_ngrams)`

Score the n-grams of the target corpus by Weirdness.

Parameters:

Name	Type	Description	Default
`target_ngrams`	`List[List[str]]`	The n-grams of each target document.	required
`ref_ngrams`	`List[List[str]]`	The n-grams of each reference document.	required

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: The Weirdness of every n-gram in the target corpus.

`extract_terms_teanga(corpus)`

Extract terms from a Teanga corpus.

Parameters:

Name	Type	Description	Default
`corpus`	`Corpus`	A Teanga corpus with a `words` span layer. The reference corpus must also be a Teanga corpus with a `words` layer.	required

Returns:

Name	Type	Description
`ATEResults`	`ATEResults`	Results with ranked terms and scores, and term offsets.

Raises:

Type	Description
`TypeError`	If the reference corpus is a list of strings.

`extract_terms_strings(corpus)`

Extract terms from a list of strings.

Parameters:

Name	Type	Description	Default
`corpus`	`List[str]`	The documents as plain strings. The reference corpus must also be a list of strings.	required

Returns:

Name	Type	Description
`ATEResults`	`ATEResults`	Results with ranked terms and scores.

Raises:

Type	Description
`TypeError`	If the reference corpus is a Teanga corpus.

`teanga_words(doc)`

Get the words of a Teanga document with their offsets.

Parameters:

Name	Type	Description	Default
`doc`	`Document`	A Teanga document with a words layer (see `words_layer`).	required

Returns:

Type	Description
`list[tuple[int, int, str]]`	List[Tuple[int, int, str]]: A (start, end, text) tuple for each word, where
`list[tuple[int, int, str]]`	text is the word's lemma if `lemma_layer` is set.

`candidate_ngrams_teanga(corpus)`

Find the occurrences of the candidates in a Teanga corpus.

Parameters:

Name	Type	Description	Default
`corpus`	`Corpus`	A Teanga corpus with a words layer (see `words_layer`) and, if `pos_pattern` is set, a part-of-speech layer.	required

Returns:

Type	Description
`dict[str, list[tuple[str, tuple[int, int]]]]`	Dict[str, List[Tuple[str, Tuple[int, int]]]]: For each document ID, every
`dict[str, list[tuple[str, tuple[int, int]]]]`	occurrence of a candidate, with its start and end character offsets.

`candidate_ngrams_strings(docs)`

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
`docs`	`List[List[str]]`	The words of each document.	required

Returns:

Type	Description
`list[list[str]]`	List[List[str]]: Every occurrence of a candidate in each document.

Raises:

Type	Description
`ValueError`	If `pos_pattern` is set, since plain text has no tags.

`candidate_occurrences(docs, tags=None)`

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
`docs`	`List[List[str]]`	The words of each document.	required
`tags`	`Optional[List[List[str]]]`	The part-of-speech tag of each word of each document, needed if `pos_pattern` is set.	`None`

Returns:

Type	Description
`list[list[tuple[str, int, int]]]`	List[List[Tuple[str, int, int]]]: Every occurrence of a candidate in each
`list[list[tuple[str, int, int]]]`	document, with its start and end (exclusive) word index.

Raises:

Type	Description
`ValueError`	If `pos_pattern` is set and there are no tags.

`frequent_candidates(ngrams)`

Find the candidates that occur at least min_frequency times.

Parameters:

Name	Type	Description	Default
`ngrams`	`Iterable[str]`	Every occurrence of a candidate in the corpus.	required

Returns:

Type	Description
`set[str]`	Set[str]: The candidates that occur often enough.

`generate_ngrams_teanga(words_with_offsets)`

Generate n-grams with offsets for a Teanga corpus.

Parameters:

Name	Type	Description	Default
`words_with_offsets`	`List[Tuple[int, int, str]]`	List of (start, end, text) tuples for words.	required

Returns:

Type	Description
`list[tuple[str, tuple[int, int]]]`	List[Tuple[str, Tuple[int, int]]]: List of n-grams with their start and end offsets.

`generate_ngrams_strings(words)`

Generate n-grams for a plain list of strings.

Parameters:

Name	Type	Description	Default
`words`	`List[str]`	List of words in the document.	required

Returns:

Type	Description
`list[str]`	List[str]: List of n-grams.

`compute_tf(ngrams, total=None)`

Compute the relative frequency of each n-gram in a document.

Parameters:

Name	Type	Description	Default
`ngrams`	`List[str]`	The n-grams in the document.	required
`total`	`Optional[int]`	The number to divide the counts by, or None for the number of `ngrams`.	`None`

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: Count of each n-gram divided by the total number of n-grams.

`ngram_total(num_words)`

Count the n-grams of up to n words in a document, candidates or not.

Parameters:

Name	Type	Description	Default
`num_words`	`int`	The number of words of the document.	required

Returns:

Name	Type	Description
`int`	`int`	The number of n-grams.

`compute_idf(corpus_ngrams)`

Compute the inverse document frequency of each n-gram as log(N / (1 + df)).

Parameters:

Name	Type	Description	Default
`corpus_ngrams`	`List[List[str]]`	The n-grams of each document.	required

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: IDF score for each n-gram, where N is the number of documents
`dict[str, float]`	and df is the number of documents containing the n-gram.

`doc_terms(doc_id, scores)`

Build one document's entry in ATEResults.terms.

Parameters:

Name	Type	Description	Default
`doc_id`	`str`	The document ID.	required
`scores`	`Dict[str, float]`	The score of each candidate term.	required

Returns:

Type	Description
`dict[str, Any]`	Dict[str, Any]: `{"doc_id": doc_id, "terms": [{"term": ..., "score": ...}, ...]}`, with
`dict[str, Any]`	terms below the threshold or starting or ending with a stopword removed, and the rest ranked from highest to lowest score.

`doc_offsets(doc_id, ngrams_with_offsets, doc_terms)` `staticmethod`

Build one document's entry in ATEResults.offsets.

Parameters:

Name	Type	Description	Default
`doc_id`	`str`	The document ID.	required
`ngrams_with_offsets`	`Iterable[Tuple[str, Tuple[int, int]]]`	Every n-gram occurrence in the document with its (start, end) character offsets.	required
`doc_terms`	`Dict[str, Any]`	The document's entry in `ATEResults.terms`.	required

Returns:

Type	Description
`dict[str, Any]`	Dict[str, Any]: `{"doc_id": doc_id, "offsets": [{"term": ..., "offset": (start, end)}, ...]}`,
`dict[str, Any]`	listing each occurrence of an extracted term in document order.

`score_corpora(target_ngrams, ref_ngrams)`

Score the n-grams of the target corpus against one or more reference corpora.

By default, the reference corpora are read as one corpus and passed to score_terms.

Parameters:

Name	Type	Description	Default
`target_ngrams`	`List[List[str]]`	The n-grams of each target document.	required
`ref_ngrams`	`List[List[List[str]]]`	The n-grams of each document of each reference corpus.	required

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: The score of every n-gram in the target corpus.

`count_ngrams(corpus_ngrams)` `staticmethod`

Count the n-grams of a corpus.

Parameters:

Name	Type	Description	Default
`corpus_ngrams`	`List[List[str]]`	The n-grams of each document.	required

Returns:

Type	Description
`tuple[Counter, int]`	Tuple[Counter, int]: The count of each n-gram, and the total number of n-grams.

Weirdness

WeirdnessTermExtractor(reference_corpus, threshold=None, n=1, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)

score_terms(target_ngrams, ref_ngrams)

extract_terms_teanga(corpus)

extract_terms_strings(corpus)

teanga_words(doc)

candidate_ngrams_teanga(corpus)

candidate_ngrams_strings(docs)

candidate_occurrences(docs, tags=None)

frequent_candidates(ngrams)

generate_ngrams_teanga(words_with_offsets)

generate_ngrams_strings(words)

compute_tf(ngrams, total=None)

ngram_total(num_words)

compute_idf(corpus_ngrams)

doc_terms(doc_id, scores)

doc_offsets(doc_id, ngrams_with_offsets, doc_terms) staticmethod

score_corpora(target_ngrams, ref_ngrams)

count_ngrams(corpus_ngrams) staticmethod

`WeirdnessTermExtractor(reference_corpus, threshold=None, n=1, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)`

`score_terms(target_ngrams, ref_ngrams)`

`extract_terms_teanga(corpus)`

`extract_terms_strings(corpus)`

`teanga_words(doc)`

`candidate_ngrams_teanga(corpus)`

`candidate_ngrams_strings(docs)`

`candidate_occurrences(docs, tags=None)`

`frequent_candidates(ngrams)`

`generate_ngrams_teanga(words_with_offsets)`

`generate_ngrams_strings(words)`

`compute_tf(ngrams, total=None)`

`ngram_total(num_words)`

`compute_idf(corpus_ngrams)`

`doc_terms(doc_id, scores)`

`doc_offsets(doc_id, ngrams_with_offsets, doc_terms)` `staticmethod`

`score_corpora(target_ngrams, ref_ngrams)`

`count_ngrams(corpus_ngrams)` `staticmethod`