Evaluation

evaluate() scores an ATEResults object against a gold standard. It is also available as termxtract.evaluate.

from termxtract import TermExtractor, evaluate

corpus = [
    "Heart failure patients often have high blood pressure.",
    "Blood pressure and ejection fraction predict heart failure.",
]
gold = {"heart failure", "blood pressure", "ejection fraction"}

results = TermExtractor(method="cvalue", n=2).extract(corpus)
scores = evaluate(results, gold, ks=(3,))
print(f"P={scores.precision:.2f} R={scores.recall:.2f} F1={scores.f1:.2f}")
print(f"AvP={scores.average_precision:.2f} P@3={scores.precision_at_k[3]:.2f}")
# P=0.17 R=1.00 F1=0.29
# AvP=0.63 P@3=0.33

top3 = evaluate(results, gold, top_k=3)
print(f"P={top3.precision:.2f} R={top3.recall:.2f} MCC={top3.mcc:.2f}")
# P=0.33 R=0.33 MCC=0.20

Matthews correlation coefficient (mcc) treats the terms the extractor scored as the candidates, so it is informative only when top_k keeps part of the ranking.

Gold = Corpus | Iterable[str] | Iterable[Iterable[str]] | Mapping[str, Iterable[str]] module-attribute

A gold standard:

• a collection of terms for the whole corpus, such as a set or a list of strings,
• one collection of terms per document, in the order of the documents,
• a dictionary from document IDs to collections of terms, or
• a Teanga corpus with a span layer of gold terms.

Evaluation(precision, recall, f1, mcc, true_positives, false_positives, false_negatives, true_negatives, average_precision, r_precision, precision_at_k=dict()) dataclass

Scores of extracted terms against a gold standard, from evaluate.

Attributes:

Name	Type	Description
precision	float	The fraction of extracted terms that are gold terms.
recall	float	The fraction of gold terms that were extracted.
f1	float	The harmonic mean of precision and recall.
mcc	float	Matthews correlation coefficient, between -1 and 1, treating the terms the extractor scored as the candidates; see evaluate.
true_positives	int	The number of extracted terms that are gold terms.
false_positives	int	The number of extracted terms that are not gold terms.
false_negatives	int	The number of gold terms that were not extracted.
true_negatives	int	The number of scored terms that are not gold terms and were not extracted, because they fell outside top_k.
average_precision	float	The mean of the precision at the rank of each gold term in the corpus ranking, counting gold terms that were not extracted as zero.
r_precision	float	The precision of the first R terms of the corpus ranking, where R is the number of gold terms.
precision_at_k	Dict[int, float]	The precision of the first K terms of the corpus ranking, for each K. If fewer than K terms were extracted, the missing terms count as wrong.

evaluate(results, gold, average='corpus', top_k=None, ks=(10, 100, 1000), term_lengths=None, gold_layer='terms', normalize=normalize_term)

Score extracted terms against a gold standard.

Terms match if they are equal after normalize, which by default lowercases them and collapses whitespace. There is no credit for partial matches: if the gold standard has "heart failure patients", extracting "heart failure" is a false positive. Nested gold terms each count as a separate term.

Precision, recall and F1 compare sets of terms, in one of three ways:

• "corpus": the distinct terms extracted from the whole corpus against the distinct gold terms, as in the TermEval shared task.
• "micro": the terms of each document against that document's gold terms, adding up the true positives, false positives and false negatives of all documents.
• "macro": the same per-document comparison, averaging the precision, recall, F1 and MCC of the documents.

Matthews correlation coefficient (MCC) also needs true negatives. The candidates are the terms the extractor scored, plus the gold terms, and the true negatives are the scored terms that are not gold terms and were not extracted. Extractors return every candidate they score, so there are true negatives only when top_k cuts the ranking; without it, MCC is 0 or negative. As for the other scores, an MCC with a zero denominator is taken as 0.

The ranking scores (average precision, R-precision and precision at K) always use the corpus ranking, where each term is ranked by its highest score in any document, and the gold terms of the whole corpus. With per-document gold terms, only the documents in the results count.

Parameters:

Name	Type	Description	Default
results	ATEResults	The extracted terms.	required
gold	Gold	The gold standard. "micro" and "macro" need per-document gold terms. Documents are matched to the results by ID for a dictionary or a Teanga corpus, and by position for a list of collections.	required
average	str	"corpus", "micro" or "macro".	'corpus'
top_k	Optional[int]	If given, only the first top_k terms of the corpus ranking (for "corpus") or of each document (otherwise) count as extracted.	None
ks	Iterable[int]	The values of K for precision_at_k.	(10, 100, 1000)
term_lengths	Optional[Collection[int]]	If given, only extracted and gold terms with these numbers of words are scored, for example {1} for single-word terms.	None
gold_layer	str	For a Teanga gold standard, the span layer of gold terms.	'terms'
normalize	Callable[[str], str]	The function applied to every term before matching.	normalize_term

Returns:

Name	Type	Description
Evaluation	Evaluation	The scores.

Raises:

Type	Description
ValueError	If average is unknown, "micro" or "macro" is used with gold terms for the whole corpus, or the per-document gold terms do not match the documents of the results.

normalize_term(term)

Normalize a term for matching: lowercase it and collapse runs of whitespace.

Parameters:

Name	Type	Description	Default
term	str	A term.	required

Returns:

Name	Type	Description
str	str	The normalized term.

precision_at(ranking, gold, k)

Get the precision of the first k terms of a ranking.

Parameters:

Name	Type	Description	Default
ranking	List[str]	Terms, from best to worst.	required
gold	Collection[str]	The gold terms.	required
k	int	The number of terms to score. If the ranking is shorter, the missing terms count as wrong.	required

Returns:

Name	Type	Description
float	float	The fraction of the first k terms that are gold terms, or 0 if k is 0.

average_precision(ranking, gold)

Get the average precision of a ranking.

This is the sum over the gold terms in the ranking of the precision at their rank, divided by the number of gold terms, so gold terms missing from the ranking count as zero.

Parameters:

Name	Type	Description	Default
ranking	List[str]	Terms, from best to worst.	required
gold	Collection[str]	The gold terms.	required

Returns:

Name	Type	Description
float	float	The average precision, or 0 if there are no gold terms.

matthews_corrcoef(tp, fp, fn, tn)

Get Matthews correlation coefficient from the counts of a binary classification.

Parameters:

Name	Type	Description	Default
tp	int	The number of true positives.	required
fp	int	The number of false positives.	required
fn	int	The number of false negatives.	required
tn	int	The number of true negatives.	required

Returns:

Name	Type	Description
float	float	The coefficient, between -1 and 1, or 0 if its denominator is 0.