SupervisedTermExtractor

Entry point for the supervised methods, which are trained from a dictionary of labelled candidate terms.

SupervisedTermExtractor(method, model_name='xlm-roberta-base', n=None, population_size=20, offspring_size=100, generations=100, runs=21, random_state=None, max_negatives=10000, patience=None, n_jobs=None, batch_size=8, epochs=3, learning_rate=2e-05, max_length=512, language='en', stopwords='english', words_layer=None, terms_layer=GOLD_LAYER, sentences_layer=None, pos_pattern=None, pos_layer=None)

A wrapper class for selecting the supervised term extraction method.

SupervisedTermExtractor.METHODS maps each method name to a function returning its extractor class and the parameters passed to it, so list(SupervisedTermExtractor.METHODS) lists the available methods.

Initialize the TermExtractor with the desired supervised method and parameters.

Parameters:

Name	Type	Description	Default
method	str	The term extraction method, one of the keys of SupervisedTermExtractor.METHODS ('roger', 'token-classification').	required
model_name	str	Hugging Face model name for token classification, or the directory of a model saved with save.	'xlm-roberta-base'
n	Optional[int]	Maximum length of n-grams to consider, or None for the method's default: 2 for 'roger' and 6 for 'token-classification'.	None
population_size	int	Number of parents in each generation (only for 'roger').	20
offspring_size	int	Number of offspring in each generation (only for 'roger').	100
generations	int	Number of generations in each run (only for 'roger').	100
runs	int	Number of independent runs to combine (only for 'roger').	21
random_state	Optional[int]	Seed for reproducible scores, or for 'token-classification' for reproducible training.	None
max_negatives	Optional[int]	Train on at most this many candidates labelled 0, sampled for each run (only for 'roger').	10000
patience	Optional[int]	For 'roger', stop a run when it has not improved for this many generations; for 'token-classification', stop training when the F1 on the validation set has not improved for this many epochs.	None
n_jobs	Optional[int]	Number of runs trained in parallel, -1 for one per CPU (only for 'roger').	None
batch_size	int	Training batch size (only for 'token-classification').	8
epochs	int	Number of training epochs (only for 'token-classification').	3
learning_rate	float	Learning rate (only for 'token-classification').	2e-05
max_length	int	Maximum sequence length (only for 'token-classification').	512
language	str	Language code for tokenization (only for 'token-classification').	'en'
stopwords	Union[str, Iterable[str], None]	Drop extracted terms that start or end with a stopword: the name of a built-in list in termxtract.stopwords.STOPWORD_LISTS, such as "english" (the default) or an ISO 639-1 code such as "fr", a list of words, or None to keep every term.	'english'
words_layer	Optional[str]	For Teanga input, the span layer holding the words: None (the default) for words or, if the corpus has no words layer, tokens.	None
terms_layer	str	For Teanga input, the span layer of gold terms that fit trains on when it is given no labels: terms by default, as in the corpora of termxtract.datasets.	GOLD_LAYER
sentences_layer	Optional[str]	For Teanga input, the span layer of the sentences: None (the default) for sentences or, if the corpus has no such layer, splitting after each word that is ".", "!" or "?" (only for 'token-classification').	None
pos_pattern	Optional[str]	For Teanga input, keep only the terms whose part-of-speech tags match this pattern in at least one occurrence. Tags are written in angle brackets, as in "<ADJ|NOUN>*<NOUN>", or use the name of a pattern in termxtract.pos.POS_PATTERNS, such as "noun_phrase". None (the default) keeps every term. Scores are unchanged.	None
pos_layer	Optional[str]	The part-of-speech layer used by pos_pattern, based on the words layer: None (the default) for upos or, if the corpus has no upos layer, pos.	None

Raises:

Type	Description
ValueError	If the method is unknown.
ImportError	If the method is 'token-classification' and the optional 'transformers' dependencies are not installed.

FILTER_PARAMS = ('pos_pattern', 'pos_layer') class-attribute

Parameters applied to the results by the wrapper, not passed to the extractor.

fit(corpus, labels=None, validation=None)

Train the method on a corpus.

Parameters:

Name	Type	Description	Default
corpus	Union[Corpus, List[str]]	Teanga Corpus object or list of strings.	required
labels	Optional[Dict[str, int]]	Dictionary with terms as keys and binary labels {0, 1}. Roger needs examples of both; given only terms, it counts every other candidate of the corpus as a non-term. None, for a Teanga corpus, trains on the gold terms of terms_layer: token classification tags each occurrence of them, and Roger takes them as the terms.	None
validation	Optional[Union[Corpus, List[str]]]	For 'token-classification', a corpus of the same kind, labelled in the same way, to compute the F1 of the tagged terms on after each epoch. The model of the best epoch is kept, and patience can stop training early.	None

Returns:

Name	Type	Description
SupervisedTermExtractor	SupervisedTermExtractor	This extractor, trained, so that fit(...).predict(...)
	SupervisedTermExtractor	can be chained.

Raises:

Type	Description
ValueError	If the corpus is neither a Teanga Corpus nor a list of strings, pos_pattern is set and the corpus is a list of strings, a list of strings is given no labels, or validation is given for Roger.

predict(corpus)

Extract terms from a corpus with the trained method.

The corpus can differ from the one the method was trained on, for example another domain or another split of a dataset.

Parameters:

Name	Type	Description	Default
corpus	Union[Corpus, List[str]]	Teanga Corpus object or list of strings.	required

Returns:

Name	Type	Description
ATEResults	ATEResults	The terms of each document with their scores, and their offsets
	ATEResults	for a Teanga corpus.

Raises:

Type	Description
ValueError	If the corpus is neither a Teanga Corpus nor a list of strings, or pos_pattern is set and the corpus is a list of strings.

extract(corpus, labels=None)

Train the method on a corpus and extract the terms of that same corpus.

To extract terms from another corpus, use fit and then predict.

Parameters:

Name	Type	Description	Default
corpus	Union[Corpus, List[str]]	Teanga Corpus object or list of strings.	required
labels	Optional[Dict[str, int]]	Dictionary with terms as keys and binary labels {0, 1}, or None to train on the gold terms of terms_layer, as for fit.	None

Returns:

Name	Type	Description
ATEResults	ATEResults	Object containing extracted terms with scores.

save(directory)

Save a trained token-classification model, to pass the directory as model_name to extract terms without training again.

Parameters:

Name	Type	Description	Default
directory	Union[str, Path]	The directory to save to.	required

Raises:

Type	Description
ValueError	If the method is not 'token-classification'.
RuntimeError	If the model has not been trained.