Domain Pertinence

Used by TermExtractor with method="domainpertinence".

`DomainPertinenceTermExtractor(reference_corpus, threshold=None, n=1, tie_break='ratio', stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)`

Bases: ReferenceCorpusTermExtractor

Domain Pertinence (Navigli & Velardi, 2004), which compares a candidate's frequency in the target corpus with its frequency in the reference corpus where it is most frequent.

Each candidate is scored as ( p_T(t) / \max(p_T(t), \max_j p_j(t)) ), where ( p_T(t) ) and ( p_j(t) ) are its frequencies relative to the number of n-grams in the target corpus and in the ( j )-th reference corpus. Navigli & Velardi use one reference corpus per contrasting domain. The score is 1 for every candidate at least as frequent in the target corpus as in any reference corpus, including those missing from them all, so with a single reference corpus most candidates tie at 1. tie_break orders those candidates, by adding a number between 0 and 1 to their score:

"ratio": by ( p_T(t) / \max_j p_j(t) ), with the reference counts smoothed by adding one, as in weirdness. With a single reference corpus, candidates are then ranked as weirdness ranks them.
"frequency": by their frequency in the target corpus.
None: leave them tied, so they are ranked in arbitrary order.

Initialize the Domain Pertinence extractor.

Parameters:

Name	Type	Description	Default
`reference_corpus`	`Union[Corpus, List[str], List[Union[Corpus, List[str]]]]`	The reference corpus, of the same type as the corpus the terms are extracted from, or a list of reference corpora, such as one per contrasting domain.	required
`threshold`	`Optional[float]`	Minimum score for term inclusion.	`None`
`n`	`int`	Maximum n-gram size (e.g., 1 for unigrams, 2 for bigrams, etc.).	`1`
`tie_break`	`Optional[str]`	How to order the candidates that score 1: "ratio", "frequency" or None (see the class description).	`'ratio'`
`stopwords`	`Union[str, Iterable[str], None]`	Drop candidates that start or end with a stopword: a name in `STOPWORD_LISTS`, a list of words, or None.	`'english'`
`words_layer`	`Optional[str]`	For Teanga input, the span layer holding the words, or None for `words` or, failing that, `tokens`.	`None`
`lemma_layer`	`Optional[str]`	For Teanga input, a sequence layer holding the lemma of each word, to count inflected forms such as "cell" and "cells" as one term, or None to use the words as written.	`None`
`min_frequency`	`int`	Drop candidates that occur fewer times in the corpus.	`1`
`word_pattern`	`Optional[str]`	Drop candidates with a word that does not fully match this regular expression or named pattern ("alphanumeric", "atr4s"), or None to keep every word.	`'alphanumeric'`
`pos_pattern`	`Optional[str]`	For Teanga input, count only the occurrences whose part-of-speech tags match this pattern, or None.	`None`
`pos_layer`	`Optional[str]`	The part-of-speech layer for `pos_pattern`, or None for `upos` or, failing that, `pos`.	`None`

Raises:

Type	Description
`ValueError`	If `tie_break` is not one of `TIE_BREAKS` or None.

`score_terms(target_ngrams, ref_ngrams)`

Score the n-grams of the target corpus by Domain Pertinence against one reference corpus.

Parameters:

Name	Type	Description	Default
`target_ngrams`	`List[List[str]]`	The n-grams of each target document.	required
`ref_ngrams`	`List[List[str]]`	The n-grams of each reference document.	required

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: The Domain Pertinence of every n-gram in the target corpus.

`score_corpora(target_ngrams, ref_ngrams)`

Score the n-grams of the target corpus by Domain Pertinence.

Parameters:

Name	Type	Description	Default
`target_ngrams`	`List[List[str]]`	The n-grams of each target document.	required
`ref_ngrams`	`List[List[List[str]]]`	The n-grams of each document of each reference corpus.	required

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: The Domain Pertinence of every n-gram in the target corpus,
`dict[str, float]`	between 0 and 1, plus the tie-breaking value for those at 1.

`extract_terms_teanga(corpus)`

Extract terms from a Teanga corpus.

Parameters:

Name	Type	Description	Default
`corpus`	`Corpus`	A Teanga corpus with a `words` span layer. The reference corpus must also be a Teanga corpus with a `words` layer.	required

Returns:

Name	Type	Description
`ATEResults`	`ATEResults`	Results with ranked terms and scores, and term offsets.

Raises:

Type	Description
`TypeError`	If the reference corpus is a list of strings.

`extract_terms_strings(corpus)`

Extract terms from a list of strings.

Parameters:

Name	Type	Description	Default
`corpus`	`List[str]`	The documents as plain strings. The reference corpus must also be a list of strings.	required

Returns:

Name	Type	Description
`ATEResults`	`ATEResults`	Results with ranked terms and scores.

Raises:

Type	Description
`TypeError`	If the reference corpus is a Teanga corpus.

`teanga_words(doc)`

Get the words of a Teanga document with their offsets.

Parameters:

Name	Type	Description	Default
`doc`	`Document`	A Teanga document with a words layer (see `words_layer`).	required

Returns:

Type	Description
`list[tuple[int, int, str]]`	List[Tuple[int, int, str]]: A (start, end, text) tuple for each word, where
`list[tuple[int, int, str]]`	text is the word's lemma if `lemma_layer` is set.

`candidate_ngrams_teanga(corpus)`

Find the occurrences of the candidates in a Teanga corpus.

Parameters:

Name	Type	Description	Default
`corpus`	`Corpus`	A Teanga corpus with a words layer (see `words_layer`) and, if `pos_pattern` is set, a part-of-speech layer.	required

Returns:

Type	Description
`dict[str, list[tuple[str, tuple[int, int]]]]`	Dict[str, List[Tuple[str, Tuple[int, int]]]]: For each document ID, every
`dict[str, list[tuple[str, tuple[int, int]]]]`	occurrence of a candidate, with its start and end character offsets.

`candidate_ngrams_strings(docs)`

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
`docs`	`List[List[str]]`	The words of each document.	required

Returns:

Type	Description
`list[list[str]]`	List[List[str]]: Every occurrence of a candidate in each document.

Raises:

Type	Description
`ValueError`	If `pos_pattern` is set, since plain text has no tags.

`candidate_occurrences(docs, tags=None)`

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
`docs`	`List[List[str]]`	The words of each document.	required
`tags`	`Optional[List[List[str]]]`	The part-of-speech tag of each word of each document, needed if `pos_pattern` is set.	`None`

Returns:

Type	Description
`list[list[tuple[str, int, int]]]`	List[List[Tuple[str, int, int]]]: Every occurrence of a candidate in each
`list[list[tuple[str, int, int]]]`	document, with its start and end (exclusive) word index.

Raises:

Type	Description
`ValueError`	If `pos_pattern` is set and there are no tags.

`frequent_candidates(ngrams)`

Find the candidates that occur at least min_frequency times.

Parameters:

Name	Type	Description	Default
`ngrams`	`Iterable[str]`	Every occurrence of a candidate in the corpus.	required

Returns:

Type	Description
`set[str]`	Set[str]: The candidates that occur often enough.

`generate_ngrams_teanga(words_with_offsets)`

Generate n-grams with offsets for a Teanga corpus.

Parameters:

Name	Type	Description	Default
`words_with_offsets`	`List[Tuple[int, int, str]]`	List of (start, end, text) tuples for words.	required

Returns:

Type	Description
`list[tuple[str, tuple[int, int]]]`	List[Tuple[str, Tuple[int, int]]]: List of n-grams with their start and end offsets.

`generate_ngrams_strings(words)`

Generate n-grams for a plain list of strings.

Parameters:

Name	Type	Description	Default
`words`	`List[str]`	List of words in the document.	required

Returns:

Type	Description
`list[str]`	List[str]: List of n-grams.

`compute_tf(ngrams, total=None)`

Compute the relative frequency of each n-gram in a document.

Parameters:

Name	Type	Description	Default
`ngrams`	`List[str]`	The n-grams in the document.	required
`total`	`Optional[int]`	The number to divide the counts by, or None for the number of `ngrams`.	`None`

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: Count of each n-gram divided by the total number of n-grams.

`ngram_total(num_words)`

Count the n-grams of up to n words in a document, candidates or not.

Parameters:

Name	Type	Description	Default
`num_words`	`int`	The number of words of the document.	required

Returns:

Name	Type	Description
`int`	`int`	The number of n-grams.

`compute_idf(corpus_ngrams)`

Compute the inverse document frequency of each n-gram as log(N / (1 + df)).

Parameters:

Name	Type	Description	Default
`corpus_ngrams`	`List[List[str]]`	The n-grams of each document.	required

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: IDF score for each n-gram, where N is the number of documents
`dict[str, float]`	and df is the number of documents containing the n-gram.

`doc_terms(doc_id, scores)`

Build one document's entry in ATEResults.terms.

Parameters:

Name	Type	Description	Default
`doc_id`	`str`	The document ID.	required
`scores`	`Dict[str, float]`	The score of each candidate term.	required

Returns:

Type	Description
`dict[str, Any]`	Dict[str, Any]: `{"doc_id": doc_id, "terms": [{"term": ..., "score": ...}, ...]}`, with
`dict[str, Any]`	terms below the threshold or starting or ending with a stopword removed, and the rest ranked from highest to lowest score.

`doc_offsets(doc_id, ngrams_with_offsets, doc_terms)` `staticmethod`

Build one document's entry in ATEResults.offsets.

Parameters:

Name	Type	Description	Default
`doc_id`	`str`	The document ID.	required
`ngrams_with_offsets`	`Iterable[Tuple[str, Tuple[int, int]]]`	Every n-gram occurrence in the document with its (start, end) character offsets.	required
`doc_terms`	`Dict[str, Any]`	The document's entry in `ATEResults.terms`.	required

Returns:

Type	Description
`dict[str, Any]`	Dict[str, Any]: `{"doc_id": doc_id, "offsets": [{"term": ..., "offset": (start, end)}, ...]}`,
`dict[str, Any]`	listing each occurrence of an extracted term in document order.

`count_ngrams(corpus_ngrams)` `staticmethod`

Count the n-grams of a corpus.

Parameters:

Name	Type	Description	Default
`corpus_ngrams`	`List[List[str]]`	The n-grams of each document.	required

Returns:

Type	Description
`tuple[Counter, int]`	Tuple[Counter, int]: The count of each n-gram, and the total number of n-grams.

Domain Pertinence