Domain Coherence

Used by TermExtractor with method="domaincoherence".

`DomainCoherenceTermExtractor(threshold=None, n=1, window_size=5, seed_terms=200, context_words=50, stopwords='english', words_layer=None, lemma_layer=None, min_frequency=1, word_pattern='alphanumeric', pos_pattern=None, pos_layer=None)`

Bases: BaseTermExtractor

Domain coherence (Bordea, Buitelaar & Polajnar, 2013), which scores candidates by how strongly they are associated with words that characterise the domain.

The seed_terms best candidates by Basic are taken as a model of the domain.
The context words are the content words that occur in at least a quarter of the documents. The context_words of them with the highest average normalised PMI with the seed terms are kept.
Each candidate is scored by its average normalised PMI with those context words.

Normalised PMI is computed from co-occurrence within window_size words on either side of each occurrence of a candidate: ( \log \frac{c(t, w) N}{f(t) f(w)} / -\log \frac{c(t, w)}{N} ), where ( N ) is the number of words of the corpus, or -1 if ( t ) and ( w ) never co-occur.

Content words are those that are not stopwords, that match word_pattern and, for a Teanga corpus with a part-of-speech layer, whose tag is in CONTENT_TAGS.

Initialize the DomainCoherence extractor.

Parameters:

Name	Type	Description	Default
`threshold`	`Optional[float]`	Minimum score for term inclusion.	`None`
`n`	`int`	Maximum n-gram size for candidate terms.	`1`
`window_size`	`int`	The number of words on either side of a candidate that count as co-occurring with it.	`5`
`seed_terms`	`int`	The number of best candidates by Basic used to choose the context words.	`200`
`context_words`	`int`	The number of context words that candidates are scored against.	`50`
`stopwords`	`Union[str, Iterable[str], None]`	Drop candidates that start or end with a stopword: a name in `STOPWORD_LISTS`, a list of words, or None.	`'english'`
`words_layer`	`Optional[str]`	For Teanga input, the span layer holding the words, or None for `words` or, failing that, `tokens`.	`None`
`lemma_layer`	`Optional[str]`	For Teanga input, a sequence layer holding the lemma of each word, to count inflected forms such as "cell" and "cells" as one term, or None to use the words as written.	`None`
`min_frequency`	`int`	Drop candidates that occur fewer times in the corpus.	`1`
`word_pattern`	`Optional[str]`	Drop candidates with a word that does not fully match this regular expression or named pattern ("alphanumeric", "atr4s"), or None to keep every word.	`'alphanumeric'`
`pos_pattern`	`Optional[str]`	For Teanga input, count only the occurrences whose part-of-speech tags match this pattern, or None.	`None`
`pos_layer`	`Optional[str]`	The part-of-speech layer for `pos_pattern` and for choosing the context words, or None for `upos` or, failing that, `pos`.	`None`

`extract_terms_teanga(corpus)`

Extract terms from a Teanga corpus using DomainCoherence.

`extract_terms_strings(corpus)`

Extract terms from a plain list of strings using DomainCoherence.

`corpus_scores(docs, occurrences, tags=None)`

Score the candidates of a corpus by domain coherence.

Parameters:

Name	Type	Description	Default
`docs`	`List[List[str]]`	The words of each document.	required
`occurrences`	`List[List[Tuple[str, int, int]]]`	The occurrences of the candidates in each document, from `candidate_occurrences`.	required
`tags`	`Optional[List[List[str]]]`	The part-of-speech tag of each word of each document, to restrict the context words to `CONTENT_TAGS`, or None.	`None`

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: The score of each candidate.

`teanga_words(doc)`

Get the words of a Teanga document with their offsets.

Parameters:

Name	Type	Description	Default
`doc`	`Document`	A Teanga document with a words layer (see `words_layer`).	required

Returns:

Type	Description
`list[tuple[int, int, str]]`	List[Tuple[int, int, str]]: A (start, end, text) tuple for each word, where
`list[tuple[int, int, str]]`	text is the word's lemma if `lemma_layer` is set.

`candidate_ngrams_teanga(corpus)`

Find the occurrences of the candidates in a Teanga corpus.

Parameters:

Name	Type	Description	Default
`corpus`	`Corpus`	A Teanga corpus with a words layer (see `words_layer`) and, if `pos_pattern` is set, a part-of-speech layer.	required

Returns:

Type	Description
`dict[str, list[tuple[str, tuple[int, int]]]]`	Dict[str, List[Tuple[str, Tuple[int, int]]]]: For each document ID, every
`dict[str, list[tuple[str, tuple[int, int]]]]`	occurrence of a candidate, with its start and end character offsets.

`candidate_ngrams_strings(docs)`

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
`docs`	`List[List[str]]`	The words of each document.	required

Returns:

Type	Description
`list[list[str]]`	List[List[str]]: Every occurrence of a candidate in each document.

Raises:

Type	Description
`ValueError`	If `pos_pattern` is set, since plain text has no tags.

`candidate_occurrences(docs, tags=None)`

Find the occurrences of the candidates in documents split into words.

Parameters:

Name	Type	Description	Default
`docs`	`List[List[str]]`	The words of each document.	required
`tags`	`Optional[List[List[str]]]`	The part-of-speech tag of each word of each document, needed if `pos_pattern` is set.	`None`

Returns:

Type	Description
`list[list[tuple[str, int, int]]]`	List[List[Tuple[str, int, int]]]: Every occurrence of a candidate in each
`list[list[tuple[str, int, int]]]`	document, with its start and end (exclusive) word index.

Raises:

Type	Description
`ValueError`	If `pos_pattern` is set and there are no tags.

`frequent_candidates(ngrams)`

Find the candidates that occur at least min_frequency times.

Parameters:

Name	Type	Description	Default
`ngrams`	`Iterable[str]`	Every occurrence of a candidate in the corpus.	required

Returns:

Type	Description
`set[str]`	Set[str]: The candidates that occur often enough.

`generate_ngrams_teanga(words_with_offsets)`

Generate n-grams with offsets for a Teanga corpus.

Parameters:

Name	Type	Description	Default
`words_with_offsets`	`List[Tuple[int, int, str]]`	List of (start, end, text) tuples for words.	required

Returns:

Type	Description
`list[tuple[str, tuple[int, int]]]`	List[Tuple[str, Tuple[int, int]]]: List of n-grams with their start and end offsets.

`generate_ngrams_strings(words)`

Generate n-grams for a plain list of strings.

Parameters:

Name	Type	Description	Default
`words`	`List[str]`	List of words in the document.	required

Returns:

Type	Description
`list[str]`	List[str]: List of n-grams.

`compute_tf(ngrams, total=None)`

Compute the relative frequency of each n-gram in a document.

Parameters:

Name	Type	Description	Default
`ngrams`	`List[str]`	The n-grams in the document.	required
`total`	`Optional[int]`	The number to divide the counts by, or None for the number of `ngrams`.	`None`

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: Count of each n-gram divided by the total number of n-grams.

`ngram_total(num_words)`

Count the n-grams of up to n words in a document, candidates or not.

Parameters:

Name	Type	Description	Default
`num_words`	`int`	The number of words of the document.	required

Returns:

Name	Type	Description
`int`	`int`	The number of n-grams.

`compute_idf(corpus_ngrams)`

Compute the inverse document frequency of each n-gram as log(N / (1 + df)).

Parameters:

Name	Type	Description	Default
`corpus_ngrams`	`List[List[str]]`	The n-grams of each document.	required

Returns:

Type	Description
`dict[str, float]`	Dict[str, float]: IDF score for each n-gram, where N is the number of documents
`dict[str, float]`	and df is the number of documents containing the n-gram.

`doc_terms(doc_id, scores)`

Build one document's entry in ATEResults.terms.

Parameters:

Name	Type	Description	Default
`doc_id`	`str`	The document ID.	required
`scores`	`Dict[str, float]`	The score of each candidate term.	required

Returns:

Type	Description
`dict[str, Any]`	Dict[str, Any]: `{"doc_id": doc_id, "terms": [{"term": ..., "score": ...}, ...]}`, with
`dict[str, Any]`	terms below the threshold or starting or ending with a stopword removed, and the rest ranked from highest to lowest score.

`doc_offsets(doc_id, ngrams_with_offsets, doc_terms)` `staticmethod`

Build one document's entry in ATEResults.offsets.

Parameters:

Name	Type	Description	Default
`doc_id`	`str`	The document ID.	required
`ngrams_with_offsets`	`Iterable[Tuple[str, Tuple[int, int]]]`	Every n-gram occurrence in the document with its (start, end) character offsets.	required
`doc_terms`	`Dict[str, Any]`	The document's entry in `ATEResults.terms`.	required

Returns:

Type	Description
`dict[str, Any]`	Dict[str, Any]: `{"doc_id": doc_id, "offsets": [{"term": ..., "offset": (start, end)}, ...]}`,
`dict[str, Any]`	listing each occurrence of an extracted term in document order.

Domain Coherence