Filters and layers

Candidate filtering, used by the stopwords, word_pattern, pos_pattern and min_frequency parameters, and the helpers that read words and tags from Teanga corpora.

Candidates

MARKS = _marks() module-attribute

A character class of the Unicode combining marks, such as the vowel signs and virama of Devanagari. Python's \w does not match them, but they are part of words: शिक्षा is श, ि, क, ्, ष and ा.

WORD = re.compile(f'(?:\\w|{MARKS})+') module-attribute

A word of plain text: a run of letters, digits, underscores and combining marks.

WORD_PATTERNS = {'alphanumeric': '.*[^\\W_].*', 'atr4s': f'(?:[^\\W_]|-|{MARKS}){{3,}}'} module-attribute

Named word patterns for word_pattern, as regular expressions that every word of a candidate must fully match.

alphanumeric matches words with at least one letter or digit, so candidates such as "95 % ci" are dropped. atr4s is the noise filter of ATR4S (Astrakhantsev, 2018): words of at least three characters, each a letter, a digit, a hyphen or a combining mark.

split_words(text)

Split plain text into lower-case words.

Parameters:

Name	Type	Description	Default
text	str	The text.	required

Returns:

Type	Description
list[str]	List[str]: The runs of letters, digits, underscores and combining marks.

compile_word_pattern(pattern)

Compile a word pattern.

Parameters:

Name	Type	Description	Default
pattern	str	A regular expression, or the name of one of WORD_PATTERNS.	required

Returns:

Type	Description
Pattern[str]	re.Pattern: The compiled pattern, to be matched against whole words.

Raises:

Type	Description
ValueError	If the pattern is not a valid regular expression.

candidate_spans(words, n, stopwords, word_pattern=None, tags=None, pos_pattern=None)

Find the occurrences of candidates in a sequence of words.

An occurrence is a sequence of up to n words that neither starts nor ends with a stopword, whose words all fully match word_pattern, and whose part-of-speech tags fully match pos_pattern.

Parameters:

Name	Type	Description	Default
words	List[str]	The words of a document.	required
n	int	The maximum number of words of a candidate.	required
stopwords	FrozenSet[str]	Lower-case stopwords.	required
word_pattern	Optional[Pattern]	A pattern from compile_word_pattern, or None to allow every word.	None
tags	Optional[List[str]]	The part-of-speech tag of each word, needed if pos_pattern is given.	None
pos_pattern	Optional[Pattern]	A pattern from termxtract.pos.compile_pos_pattern, or None to allow every tag sequence.	None

Yields:

Type	Description
tuple[int, int]	Tuple[int, int]: The start and end (exclusive) word index of each occurrence.

Stopwords

ENGLISH_STOPWORDS = frozenset({'a', 'about', 'above', 'after', 'again', 'against', 'ain', 'all', 'am', 'an', 'and', 'any', 'are', 'aren', "aren't", 'as', 'at', 'be', 'because', 'been', 'before', 'being', 'below', 'between', 'both', 'but', 'by', 'can', 'couldn', "couldn't", 'd', 'did', 'didn', "didn't", 'do', 'does', 'doesn', "doesn't", 'doing', 'don', "don't", 'down', 'during', 'each', 'few', 'for', 'from', 'further', 'had', 'hadn', "hadn't", 'has', 'hasn', "hasn't", 'have', 'haven', "haven't", 'having', 'he', "he'd", "he'll", 'her', 'here', 'hers', 'herself', "he's", 'him', 'himself', 'his', 'how', 'i', "i'd", 'if', "i'll", "i'm", 'in', 'into', 'is', 'isn', "isn't", 'it', "it'd", "it'll", "it's", 'its', 'itself', "i've", 'just', 'll', 'm', 'ma', 'me', 'mightn', "mightn't", 'more', 'most', 'mustn', "mustn't", 'my', 'myself', 'needn', "needn't", 'no', 'nor', 'not', 'now', 'o', 'of', 'off', 'on', 'once', 'only', 'or', 'other', 'our', 'ours', 'ourselves', 'out', 'over', 'own', 're', 's', 'same', 'shan', "shan't", 'she', "she'd", "she'll", "she's", 'should', 'shouldn', "shouldn't", "should've", 'so', 'some', 'such', 't', 'than', 'that', "that'll", 'the', 'their', 'theirs', 'them', 'themselves', 'then', 'there', 'these', 'they', "they'd", "they'll", "they're", "they've", 'this', 'those', 'through', 'to', 'too', 'under', 'until', 'up', 've', 'very', 'was', 'wasn', "wasn't", 'we', "we'd", "we'll", "we're", 'were', 'weren', "weren't", "we've", 'what', 'when', 'where', 'which', 'while', 'who', 'whom', 'why', 'will', 'with', 'won', "won't", 'wouldn', "wouldn't", 'y', 'you', "you'd", "you'll", 'your', "you're", 'yours', 'yourself', 'yourselves', "you've"}) module-attribute

load_stopwords(stopwords)

Resolve a stopwords argument to a set of lower-case words.

Parameters:

Name	Type	Description	Default
stopwords	Union[str, Iterable[str], None]	The name of a built-in list in STOPWORD_LISTS, such as "english" or an ISO 639-1 code such as "fr", any iterable of words for a custom list, or None for no stopwords.	required

Returns:

Type	Description
frozenset[str]	FrozenSet[str]: The stopwords, in lower case.

Raises:

Type	Description
ValueError	If stopwords is a string that names no built-in list.

is_candidate(term, stopwords)

Check that a candidate term neither starts nor ends with a stopword.

Stopwords inside a multi-word term are allowed, as in "degree of freedom".

Parameters:

Name	Type	Description	Default
term	str	The candidate, with words separated by spaces.	required
stopwords	FrozenSet[str]	Lower-case stopwords.	required

Returns:

Name	Type	Description
bool	bool	True if the first and last words of the term are not stopwords.

Part-of-speech patterns

POS_PATTERNS = {'noun_phrase': '<ADJ|NOUN|PROPN>*<NOUN|PROPN>(<ADP><DET>?<ADJ|NOUN|PROPN>*<NOUN|PROPN>)?'} module-attribute

Named part-of-speech patterns, using Universal Dependencies tags.

noun_phrase matches nouns with optional adjectives or nouns before them, such as photosynthesis or chemical energy, optionally followed by a preposition and a second such phrase, such as degree of freedom.

compile_pos_pattern(pattern)

Compile a part-of-speech pattern into a regular expression over tag sequences.

Patterns use the syntax of NLTK's RegexpParser: each tag is written in angle brackets and combined with regular-expression operators, as in <ADJ|NOUN>*<NOUN>. Inside angle brackets, | separates alternative tags and . matches any character of a tag, so <N.*> matches NOUN and NUM. Whitespace is ignored.

Parameters:

Name	Type	Description	Default
pattern	str	A pattern, or the name of one of POS_PATTERNS.	required

Returns:

Type	Description
Pattern[str]	re.Pattern: A regular expression that fully matches the tag sequences, written
Pattern[str]	as <TAG1><TAG2>..., that the pattern allows.

Raises:

Type	Description
ValueError	If the pattern is not a valid regular expression.

filter_by_pos(results, corpus, pattern, pos_layer=None, words_layer=None, lemma_layer=None)

Keep the terms that match a part-of-speech pattern in at least one occurrence.

Terms are matched to the word sequences of the corpus case-insensitively. Scores are unchanged, and offsets are kept only for the remaining terms.

Parameters:

Name	Type	Description	Default
results	ATEResults	The results to filter.	required
corpus	Corpus	The Teanga corpus the results were extracted from.	required
pattern	Pattern	A pattern from compile_pos_pattern.	required
pos_layer	Optional[str]	The part-of-speech layer, or None for upos or, failing that, pos.	None
words_layer	Optional[str]	The words layer, or None for words or, failing that, tokens.	None
lemma_layer	Optional[str]	The lemma layer the terms were extracted with, if any, so that terms are matched to the lemmas of the words.	None

Returns:

Name	Type	Description
ATEResults	ATEResults	The filtered results.

Teanga layers

WORDS_LAYERS = ('words', 'tokens') module-attribute

Layers read as the words of a Teanga document when no words_layer is given.

POS_LAYERS = ('upos', 'pos') module-attribute

Layers read as part-of-speech tags when no pos_layer is given.

find_layer(meta, layer, defaults, parameter)

Choose the layer to read from a Teanga corpus.

Parameters:

Name	Type	Description	Default
meta	Dict[str, Any]	The layer metadata of the corpus.	required
layer	Optional[str]	The layer requested by the user, or None for the first of defaults that the corpus has.	required
defaults	Tuple[str, ...]	The layers to try, in order, when layer is None.	required
parameter	str	The name of the parameter that sets the layer, for messages.	required

Returns:

Name	Type	Description
str	str	The name of the layer.

Raises:

Type	Description
ValueError	If the requested layer, or every default layer, is missing.

word_spans(doc, words_layer=None)

Get the character offsets of the words of a Teanga document.

Parameters:

Name	Type	Description	Default
doc	Document	A Teanga document.	required
words_layer	Optional[str]	The span layer holding the words, or None for words or, failing that, tokens.	None

Returns:

Type	Description
list[tuple[int, int]]	List[Tuple[int, int]]: The (start, end) offsets of each word.

words_with_offsets(doc, words_layer=None, lemma_layer=None)

Get the words of a Teanga document with their offsets.

Parameters:

Name	Type	Description	Default
doc	Document	A Teanga document.	required
words_layer	Optional[str]	The span layer holding the words, or None for words or, failing that, tokens.	None
lemma_layer	Optional[str]	A sequence layer, based on the words layer, holding the lemma of each word. If given, each word is its lemma, so that inflected forms are counted together; if None, each word is its text.	None

Returns:

Type	Description
list[tuple[int, int, str]]	List[Tuple[int, int, str]]: A (start, end, text) tuple for each word, where text
list[tuple[int, int, str]]	is the lemma if lemma_layer is given.

word_annotations(doc, layer, defaults, parameter, words_layer=None)

Get the value of a sequence layer, such as tags or lemmas, for each word of a document.

Parameters:

Name	Type	Description	Default
doc	Document	A Teanga document.	required
layer	Optional[str]	The sequence layer, or None for the first of defaults that the corpus has.	required
defaults	Tuple[str, ...]	The layers to try when layer is None.	required
parameter	str	The name of the parameter that sets the layer, for messages.	required
words_layer	Optional[str]	The words layer, as for word_spans.	None

Returns:

Type	Description
list[str]	List[str]: One value per word.

Raises:

Type	Description
ValueError	If the layer is missing or not based on the words layer.

pos_tags(doc, pos_layer=None, words_layer=None)

Get the part-of-speech tag of each word of a Teanga document.

Parameters:

Name	Type	Description	Default
doc	Document	A Teanga document.	required
pos_layer	Optional[str]	The sequence layer holding the tags, or None for upos or, failing that, pos.	None
words_layer	Optional[str]	The words layer, as for word_spans.	None

Returns:

Type	Description
list[str]	List[str]: One tag per word.

Raises:

Type	Description
ValueError	If the tag layer is missing or not based on the words layer.