Skip to content

Token Classification

Used by SupervisedTermExtractor with method="token-classification". Requires the optional transformers extra: pip install "termxtract[transformers]".

TokenClassificationTermExtractor(model_name='xlm-roberta-base', n=6, batch_size=8, epochs=3, learning_rate=2e-05, max_length=512, language='en', stopwords='english', words_layer=None, random_state=None, patience=None, terms_layer=GOLD_LAYER, sentences_layer=None)

Term extraction using transformer-based token classification models.

Initialize the Token Classification Term Extractor.

Parameters:

Name Type Description Default
model_name str

Hugging Face model to use for token classification, or the directory of a model saved with save, which is already trained.

 'xlm-roberta-base'
n int

Maximum n-gram size to consider.

 6
batch_size int

Batch size for training and evaluation.

 8
epochs int

Number of training epochs.

 3
learning_rate float

Learning rate for training.

 2e-05
max_length int

Maximum sequence length for input tokenization.

 512
language str

Language code for tokenization (en, fr, nl).

 'en'
stopwords Union[str, Iterable[str], None]

Drop extracted terms that start or end with a stopword: a name in STOPWORD_LISTS, a list of words, or None.

 'english'
words_layer Optional[str]

For Teanga input, the span layer holding the words, or None for words or, failing that, tokens.

 None
random_state Optional[int]

Seed for the new classification layer of the model and the order of the training examples, for reproducible training, or None for an unseeded classification layer.

 None
patience Optional[int]

With a validation set, stop training when the F1 on it has not improved for this many epochs. None trains for epochs.

 None
terms_layer str

For Teanga input without labels, the span layer of gold terms to train on.

 GOLD_LAYER
sentences_layer Optional[str]

For Teanga input, the span layer of the sentences, or None for sentences or, if the corpus has no such layer, splitting after each word that is ".", "!" or "?".

 None

preprocess_text(text)

Split text into sentences and tokenize.

Parameters:

Name Type Description Default
text str

Input text to preprocess.

 required

Returns:

Type Description
list[tuple[list[str], str]]

List[Tuple[List[str], str]]: List of tuples (tokens, original sentence).

find_sublist_indices(sublist, full_list)

Find all occurrences of a sublist within a larger list.

Parameters:

Name Type Description Default
sublist List[str]

The sublist to find.

 required
full_list List[str]

The full list to search in.

 required

Returns:

Type Description
list[tuple[int, int]]

List[Tuple[int, int]]: List of (start, end) indices where the sublist occurs.

create_training_data(sentence_list, terms)

Create training data with token-level annotations.

Where labelled terms overlap, the longest is tagged, and of two of the same length the first.

Parameters:

Name Type Description Default
sentence_list List[Tuple[List[str], str]]

List of (tokens, original sentence) tuples.

 required
terms Dict[str, int]

Dictionary with terms as keys and binary labels as values.

 required

Returns:

Type Description
list[tuple[list[str], list[str]]]

List[Tuple[List[str], List[str]]]: List of (tokens, tags) tuples.

create_span_training_data(sentences, terms)

Create training data from the gold term spans of a document.

Each term is tagged on the words within its span. Where terms are nested or overlap, the longest is tagged, as in ACTER's sequential annotations, and of two of the same length the first.

Parameters:

Name Type Description Default
sentences List[Tuple[List[str], List[Tuple[int, int]]]]

The words of each sentence of the document, and their character offsets.

 required
terms List[Tuple[int, int]]

The character offsets of the gold terms.

 required

Returns:

Type Description
list[tuple[list[str], list[str]]]

List[Tuple[List[str], List[str]]]: List of (tokens, tags) tuples, one per
list[tuple[list[str], list[str]]]

sentence.

tokenize_and_align_labels(texts, tags)

Tokenize inputs and align labels for transformer model.

Parameters:

Name Type Description Default
texts List[List[str]]

List of tokenized sentences.

 required
tags List[List[str]]

List of token-level tags.

 required

Returns:

Name Type Description
Dict dict

Tokenized inputs with aligned labels.

create_torch_dataset(tokenized_inputs)

Create a PyTorch dataset from tokenized inputs.

Parameters:

Name Type Description Default
tokenized_inputs Dict

Tokenized inputs with labels.

 required

Returns:

Type Description
Dataset

torch.utils.data.Dataset: Dataset for training/evaluation.

extract_terms_from_predictions(token_predictions, texts)

Extract terms from token-level predictions.

Parameters:

Name Type Description Default
token_predictions List[List[str]]

Token-level predictions.

 required
texts List[List[str]]

Original tokenized texts.

 required

Returns:

Type Description
dict[str, float]

Dict[str, float]: Dictionary of extracted terms with confidence scores.

tagged_terms(tags, tokens)

Find the terms tagged in a sentence: each B with the I tags after it.

Parameters:

Name Type Description Default
tags List[str]

The tag of each token.

 required
tokens List[str]

The tokens of the sentence.

 required

Returns:

Type Description
list[tuple[str, int, int]]

List[Tuple[str, int, int]]: Each term in lower case, with the index of its
list[tuple[str, int, int]]

first token and of the token after its last.

train_model(tokenized_dataset, validation_dataset=None)

Train the token classification model.

Parameters:

Name Type Description Default
tokenized_dataset Dataset

Training dataset.

 required
validation_dataset Dataset | None

Dataset to compute the F1 of the tagged terms on after each epoch, keeping the model of the best epoch, or None to keep the model of the last.

 None

save(directory)

Save the trained model and its tokenizer, to pass the directory as model_name to extract terms without training again.

Parameters:

Name Type Description Default
directory Union[str, Path]

The directory to save to.

 required

Raises:

Type Description
RuntimeError

If the model has not been trained.

predict_sentences(tokenized_dataset, texts)

Make predictions using the trained model, for each sentence.

Parameters:

Name Type Description Default
tokenized_dataset Dataset

Dataset for prediction.

 required
texts List[List[str]]

Original tokenized texts.

 required

Returns:

Type Description
list[dict[str, float]]

List[Dict[str, float]]: The terms extracted from each sentence, with
list[dict[str, float]]

confidence scores.

predict_tags(tokenized_dataset)

Tag the tokens of each sentence with the trained model.

Parameters:

Name Type Description Default
tokenized_dataset Dataset

Dataset for prediction.

 required

Returns:

Type Description
list[list[str]]

List[List[str]]: The tag of each token of each sentence.

predict(tokenized_dataset, texts)

Make predictions using the trained model.

Parameters:

Name Type Description Default
tokenized_dataset Dataset

Dataset for prediction.

 required
texts List[List[str]]

Original tokenized texts.

 required

Returns:

Type Description
dict[str, float]

Dict[str, float]: Dictionary of extracted terms with confidence scores.

prepare_teanga_documents(corpus)

Split each document of a Teanga corpus into sentences.

Parameters:

Name Type Description Default
corpus Corpus

Teanga corpus object.

 required

Returns:

Type Description
list[list[tuple[list[str], str]]]

List[List[Tuple[List[str], str]]]: For each document, its (tokens, original
list[list[tuple[list[str], str]]]

sentence) tuples.

prepare_teanga_corpus(corpus)

Prepare a Teanga corpus for term extraction.

Parameters:

Name Type Description Default
corpus Corpus

Teanga corpus object.

 required

Returns:

Type Description
list[tuple[list[str], str]]

List[Tuple[List[str], str]]: List of (tokens, original sentence) tuples.

extract_documents(corpus, doc_ids, documents, spans=None)

Tag the sentences of each document with the trained model.

Each document lists the terms the model tagged in its own sentences.

Parameters:

Name Type Description Default
corpus Union[Corpus, List[str]]

The input corpus, returned in the results.

 required
doc_ids List[str]

The ID of each document.

 required
documents List[List[Tuple[List[str], str]]]

For each document, its (tokens, original sentence) tuples.

 required
spans Optional[List[List[List[Tuple[int, int]]]]]

The character offsets of each token of each sentence of each document, to give the offsets of the tagged terms, or None.

 None

Returns:

Name Type Description
ATEResults ATEResults

Results object with extracted terms, and their offsets if
ATEResults

spans is given.

fit_strings(corpus, labels, validation=None)

Fine-tune the model on a list of strings, tagging the terms labelled 1.

Parameters:

Name Type Description Default
corpus List[str]

List of text documents.

 required
labels Dict[str, int]

Dictionary with terms as keys and binary labels as values.

 required
validation Optional[List[str]]

Documents tagged with the same labels, to keep the model of the epoch with the best F1 on them, or None.

 None

fit_teanga(corpus, labels=None, validation=None)

Fine-tune the model on a Teanga corpus, tagging its gold terms or the terms labelled 1.

Parameters:

Name Type Description Default
corpus Corpus

Teanga corpus object.

 required
labels Optional[Dict[str, int]]

Dictionary with terms as keys and binary labels as values, or None to tag the gold terms of terms_layer.

 None
validation Optional[Corpus]

A corpus tagged in the same way, to keep the model of the epoch with the best F1 on it, or None.

 None

predict_strings(corpus)

Tag the terms of a list of strings with the trained model.

Parameters:

Name Type Description Default
corpus List[str]

List of text documents.

 required

Returns:

Name Type Description
ATEResults ATEResults

The terms tagged in each document.

predict_teanga(corpus)

Tag the terms of a Teanga corpus with the trained model.

Parameters:

Name Type Description Default
corpus Corpus

Teanga corpus object.

 required

Returns:

Name Type Description
ATEResults ATEResults

The terms tagged in each document, with their offsets.

extract_terms_teanga(corpus, labels=None)

Extract terms from a Teanga corpus, training on it first if labels are given.

Parameters:

Name Type Description Default
corpus Corpus

Teanga corpus object.

 required
labels Optional[Dict[str, int]]

Dictionary with terms as keys and binary labels as values, to train on, or None to use the trained model.

 None

Returns:

Name Type Description
ATEResults ATEResults

Results object with extracted terms.

train_terms_teanga(corpus, labels)

Train the model and extract terms from a Teanga corpus.

Parameters:

Name Type Description Default
corpus Corpus

Teanga corpus object.

 required
labels Dict[str, int]

Dictionary with terms as keys and binary labels as values.

 required

Returns:

Name Type Description
ATEResults ATEResults

Results object with extracted terms.

extract_terms_strings(corpus, labels=None)

Extract terms from a list of strings, training on it first if labels are given.

Parameters:

Name Type Description Default
corpus List[str]

List of text documents.

 required
labels Optional[Dict[str, int]]

Dictionary with terms as keys and binary labels as values, to train on, or None to use the trained model.

 None

Returns:

Name Type Description
ATEResults ATEResults

Results object with extracted terms.

train_terms_strings(corpus, labels)

Train the model and extract terms from a list of strings.

Parameters:

Name Type Description Default
corpus List[str]

List of text documents.

 required
labels Dict[str, int]

Dictionary with terms as keys and binary labels as values.

 required

Returns:

Name Type Description
ATEResults ATEResults

Results object with extracted terms.