EWISER¶
EwiserLinker ports
Bevilacqua & Navigli's
EWISER: a frozen-BERT sentence encoder that classifies every word against
WordNet's entire sense inventory in one shared forward pass, with a
WordNet relation-graph propagation step over the output logits. Four
examples cover it end to end: loading the paper's own checkpoints,
training the encoder from scratch, loading its pretrained sense-embedding
initialization, and a full-corpus run.
- Reproducing the paper's published results
- Verifying the training path
- Loading LMMS/SensEmBERT sense embeddings
- Full-corpus training
Reproducing the paper's published results¶
Like GlossBERT's reproduction,
this runs no training at all -- see
Verifying the training path for that side
instead. EwiserEncoder.from_checkpoint
loads one of the three checkpoints
Bevilacqua & Navigli
released alongside the paper directly, reading the checkpoint's own
baked-in WordNet-relation adjacency straight from its state dict (no graph
construction needed for inference), and evaluates via
EwiserLinker.link, the exact same
production code path a freshly trained model would use.
Unlike GlossBERT's cross-encoder, EWISER encodes a whole sentence once
with a frozen BERT and classifies every word against WordNet's entire
sense inventory in one shared forward pass -- see
ewiser.py's module docstring for how that
reshapes candidate scoring internally.
Requires the three checkpoints (linked from the
reference repo's README)
at ~/Downloads/ewiser.semcor{,+wngt,_base}.pt, that repo's own
res/dictionaries/offsets.txt (the checkpoints' frequency-sorted output
vocabulary order, not reproducible from first principles), and
UFSAC 2.1 extracted to
~/data/ufsac-public-2.1/ -- see the script's own docstring for all
three. Candidates are restricted to each mention's own tagged part of
speech (unlike GlossBERT's deliberate no-POS-filtering), matching the
reference's own candidate generation exactly.
"""Reproduces EWISER's published results using the paper's own checkpoints,
through this repo's `EwiserLinker`/`UfsacDataset` pipeline.
No training happens here -- this loads Bevilacqua & Navigli's own three
released checkpoints (https://github.com/SapienzaNLP/ewiser, ACL 2020)
directly into `EwiserEncoder.from_checkpoint` and evaluates each via
`EwiserLinker.link()`, the exact same production code path a freshly
trained model would use. **This is a hard acceptance gate for issue #40,
not a best-effort verification pass**: measured numbers are expected to
match the published ones (within a stated, verified methodology
difference, not an assumed one).
Setup:
1. The three checkpoints (from the Google Drive links in the EWISER
README's "EWISER English checkpoints" section) at
``~/Downloads/ewiser.semcor.pt``, ``ewiser.semcor+wngt.pt``,
``ewiser.semcor_base.pt`` (or point `_CHECKPOINTS` below elsewhere).
2. EWISER's own ``res/dictionaries/offsets.txt`` (from a checkout of
https://github.com/SapienzaNLP/ewiser) at `_OFFSETS_PATH` below --
the checkpoints' frequency-sorted output-vocabulary order, not
reproducible from first principles (see `SenseVocabulary`).
3. UFSAC 2.1 (https://github.com/getalp/UFSAC#content-of-the-repository)
extracted to `_UFSAC_DIR` below -- this script reads its
``raganato_*.xml`` files, the same Raganato et al. (2017) WSD
evaluation framework data EWISER's own paper reports numbers against.
Checkpoint -> published-row mapping: each checkpoint's own recorded
training args (its ``data`` path and ``save_dir`` name, both readable
straight from the checkpoint file) were cross-referenced against the
paper's Table 3 (`paper/ewiser.pdf`, extracted via ``pdftotext -layout``
for exact digits, not read visually) to identify precisely which trained
configuration each file is:
- ``ewiser.semcor_base.pt``: ``data=.../semcor``, ``save_dir`` names
``hyper`` (no ``hypo``) -- SemCor only, EWISER_hyper, Table 3's
``S-only, hyper`` row.
- ``ewiser.semcor.pt``: ``data=.../semcor+glosses_main.untagged``,
``save_dir`` names ``hyper+hypo`` -- SemCor + untagged glosses,
EWISER_hyper+hypo, Table 3's ``S+G, hyper+hypo`` row.
- ``ewiser.semcor+wngt.pt``: ``data=.../semcor+glosses_main+examples``,
``save_dir`` names ``hyper`` -- SemCor + tagged glosses + WordNet
examples, EWISER_hyper, Table 3's ``S+G+G+/E, hyper`` row.
This orders exactly the same way as the README's own "EWISER English
checkpoints" bullet list (SemCor / SemCor + untagged glosses / SemCor +
tagged glosses + WordNet Examples), which is the confirming cross-check
that this mapping is right, not a guess from filenames alone.
Candidates are restricted to each mention's own tagged part of speech
(``EwiserLinker``'s candidates, filtered by `_PosRestrictedMatch` below) --
unlike `glossbert_reproduction.py`'s deliberate no-POS-filtering (matching
GlossBERT's own paper convention), EWISER's reference candidate generation
*does* restrict to ``(lemma, POS)`` (confirmed directly against
``ewiser/fairseq_ext/data/wsd_dataset.py``'s
``lemma_pos_to_possible_senses``), so matching that is required to
reproduce its published numbers, not optional fidelity.
Measured results (ALL column, the paper's own headline metric) match
published numbers to within 0.1-0.4 points on all three checkpoints:
SemCor 76.9% vs. 77.0%, SemCor+untagged-glosses 78.2% vs. 78.3%,
SemCor+tagged-glosses+examples 79.7% vs. 80.1%. Getting here required
finding and fixing two real bugs, chased down by tracing a single
instance's score, layer by layer, against the reference's own real
forward pass running in its own (Python 3.10, `qbert`/fairseq) environment
(see `~/external/ewiser/.venv`) -- not accepted as a plausible shortfall:
1. Subword tokenization must NOT use
``tokenizer(words, is_split_into_words=True)``: that high-level call
still runs BERT's punctuation-splitting pre-tokenizer on each provided
word, so a word with internal punctuation (e.g. UFSAC's own
``"Oct."``) gets pre-split into ``"Oct"``/``"."`` before wordpiece,
producing a standalone ``"."`` token where the reference (wordpiece
applied directly to the literal string, no pre-splitting) produces a
``"##."`` continuation piece -- a different vocabulary entry, unrelated
embedding. Fixed by
[wordpiece_tokenize_words][linkingtk.algorithms.wsd._ewiser_text.wordpiece_tokenize_words],
a from-scratch WordPiece implementation applied to each word in
isolation.
2. The encoder input must be the **sum of the last 4 BERT hidden-state
layers**, not just the final layer, **despite** every released
checkpoint's own recorded `context_embeddings_use_all_hidden=False`
(which reads as "single layer only"). That flag is dead: the
reference's own `TaggerModel.build_model` computes `use_all_hidden`
from `len(args.context_embeddings_type) > 1` -- a variable-name
mix-up (`context_embeddings_type` is the string `"bert"`, not the
actual layer list) that happens to make `use_all_hidden` always `True`
for any BERT-backed checkpoint, silently overriding the
correctly-named flag. See
[EwiserEncoder][linkingtk.algorithms.wsd.ewiser.EwiserEncoder]'s
`num_summed_layers` docstring.
The remaining small per-dataset gaps (mostly on the smallest eval sets,
e.g. SemEval-2007's 455 instances) are consistent with a real, separately
diagnosed and verified residual: this port's HF `transformers` encoder and
the reference's original `pytorch_pretrained_bert` encoder produce
cosine-~0.99 (not 1.0) mean-pooled word vectors for bit-identical input
and bit-identical weights -- confirmed directly, layer by layer, to be
accumulated cross-library BERT forward-pass numerics (matching
`layer_norm_eps`, matching `hidden_act`, bit-identical embedding weights),
not a further port bug. A checkpoint's `BatchNorm` was calibrated tightly
enough to the original library's exact numerics that this residual
difference can flip a close call on individual instances, with a bigger
relative effect on small eval sets than on `ALL`.
Run with: `uv run python examples/ewiser_reproduction.py`
"""
from __future__ import annotations
from pathlib import Path
import torch
from linkingtk.algorithms.wsd.ewiser import EwiserEncoder, EwiserLinker
from linkingtk.blocking.base import BlockingStrategy
from linkingtk.blocking.exact import ExactMatch
from linkingtk.core.entity import Entity
from linkingtk.core.source import EntitySource
from linkingtk.datasets.ufsac import UfsacDataset
from linkingtk.eval import Evaluator
_CHECKPOINT_DIR = Path.home() / "Downloads"
_OFFSETS_PATH = Path.home() / "external" / "ewiser" / "res" / "dictionaries" / "offsets.txt"
_UFSAC_DIR = Path.home() / "data" / "ufsac-public-2.1"
# (UFSAC filename stem, README column label)
_EVAL_SETS = [
("raganato_semeval2007", "SE07"),
("raganato_senseval2", "SE2"),
("raganato_senseval3", "SE3"),
("raganato_semeval2013", "SE13"),
("raganato_semeval2015", "SE15"),
("raganato_ALL", "ALL"),
]
# (checkpoint filename, README label, published F1 per _EVAL_SETS label --
# see module docstring for how these were matched to Table 3's rows)
_CHECKPOINTS = [
(
"ewiser.semcor_base.pt",
"SemCor",
{"SE07": 71.0, "SE2": 77.5, "SE3": 77.9, "SE13": 76.4, "SE15": 77.8, "ALL": 77.0},
),
(
"ewiser.semcor.pt",
"SemCor + untagged glosses",
{"SE07": 71.0, "SE2": 78.9, "SE3": 78.4, "SE13": 78.9, "SE15": 79.3, "ALL": 78.3},
),
(
"ewiser.semcor+wngt.pt",
"SemCor + tagged glosses + WordNet Examples",
{"SE07": 75.2, "SE2": 80.8, "SE3": 79.0, "SE13": 80.7, "SE15": 81.8, "ALL": 80.1},
),
]
_PENN_TO_WORDNET_POS = {"N": "n", "V": "v", "J": "a", "R": "r"}
def _wordnet_pos_of_synset_id(synset_id: str) -> str | None:
pos = synset_id.rsplit("-", 1)[-1]
if pos == "s": # adjective satellite counts as a plain adjective for POS filtering
return "a"
return pos if pos in "nvar" else None
def _mention_wordnet_pos(mention: Entity) -> str | None:
tag = mention.properties.get("pos")
return _PENN_TO_WORDNET_POS.get(tag[0]) if tag else None
class _PosRestrictedMatch(BlockingStrategy):
"""Wraps `ExactMatch`, keeping only candidates whose WordNet POS matches
the mention's own tagged POS (see module docstring)."""
def __init__(self, top_k: int = 50) -> None:
self._inner = ExactMatch(top_k=top_k)
def candidate_pairs(
self, dataset1: list[Entity], dataset2: list[Entity] | EntitySource
) -> list[tuple[Entity, Entity]]:
pairs = self._inner.candidate_pairs(dataset1, dataset2)
filtered = []
for entity1, entity2 in pairs:
mention_pos = _mention_wordnet_pos(entity1)
if mention_pos is None or _wordnet_pos_of_synset_id(entity2.id) == mention_pos:
filtered.append((entity1, entity2))
return filtered
def main() -> None:
device = "cuda" if torch.cuda.is_available() else "cpu"
blocking = _PosRestrictedMatch(top_k=50)
for filename, label, published in _CHECKPOINTS:
print(f"\n=== {label} ({filename}) ===")
encoder = EwiserEncoder.from_checkpoint(
_CHECKPOINT_DIR / filename, _OFFSETS_PATH, forward_batch_size=32
)
encoder.to(device)
linker = EwiserLinker(encoder)
print(f"{'dataset':<8} {'precision@1':>12} {'published':>10}")
for stem, eval_label in _EVAL_SETS:
mentions, senses, ground_truth = UfsacDataset(
source=str(_UFSAC_DIR / f"{stem}.xml")
).load()
results = linker.link(mentions, senses, blocking=blocking)
predictions = [(result.source_id, result.target_id) for result in results]
report = Evaluator.evaluate(predictions=predictions, ground_truth=ground_truth)
score = report.metrics["precision@1"] * 100
print(f"{eval_label:<8} {score:>11.1f}% {published[eval_label]:>9.1f}%")
if __name__ == "__main__":
main()
Run with:
uv run python examples/ewiser_reproduction.py
=== SemCor (ewiser.semcor_base.pt) ===
dataset precision@1 published
SE07 68.8% 71.0%
SE2 77.3% 77.5%
SE3 77.8% 77.9%
SE13 76.5% 76.4%
SE15 78.1% 77.8%
ALL 76.9% 77.0%
=== SemCor + untagged glosses (ewiser.semcor.pt) ===
dataset precision@1 published
SE07 70.8% 71.0%
SE2 78.9% 78.9%
SE3 77.9% 78.4%
SE13 79.0% 78.9%
SE15 79.5% 79.3%
ALL 78.2% 78.3%
=== SemCor + tagged glosses + WordNet Examples (ewiser.semcor+wngt.pt) ===
dataset precision@1 published
SE07 74.5% 75.2%
SE2 80.7% 80.8%
SE3 77.6% 79.0%
SE13 80.5% 80.7%
SE15 82.5% 81.8%
ALL 79.7% 80.1%
The ALL column -- the paper's own headline metric -- lands within
0.1-0.4 points on all three checkpoints. Getting here took real
diagnosis against a live oracle, not code review alone: the reference's
own Python 3.10 (qbert/fairseq) environment was stood up locally, and a
single instance's score was traced layer by layer against it until the
first point of divergence was found -- twice.
Bugs found and fixed¶
1 -- tokenization must not pre-split punctuation-attached words.
tokenizer(words, is_split_into_words=True) still runs BERT's own
punctuation-splitting pre-tokenizer on each provided word, so a UFSAC
token like "Oct." became two pre-tokens ("Oct", ".") before
wordpiece ever ran, producing a standalone "." token where the
reference (wordpiece applied directly to the literal string, no
pre-splitting) produces a "##." continuation piece instead -- a
different vocabulary entry, unrelated embedding. First measured impact:
61.3% vs. a 71.0% published target on SemEval-2007. Traced by comparing a
single sentence's mean-pooled word vector for "reported" against the
reference's own real BERT forward pass -- the vectors matched almost
perfectly for words without internal punctuation, and diverged sharply
(different vocab id entirely) exactly at "Oct."/"Sept." positions.
Fixed with a from-scratch WordPiece implementation
(wordpiece_tokenize_words) applied to
each word in isolation, matching the reference's own literal per-word
call.
2 -- the encoder input must sum the last 4 BERT layers, not just the
final one. Every released checkpoint's own recorded
context_embeddings_use_all_hidden=False reads, in isolation, as "use
only the final layer" -- a reasonable interpretation of that one config
field, and wrong. The reference's own TaggerModel.build_model computes
its actual internal use_all_hidden flag from
len(args.context_embeddings_type) > 1 -- a variable-name mix-up
(context_embeddings_type is the string "bert", not the real layer
list) that happens to make use_all_hidden unconditionally True for
every BERT-backed checkpoint, silently overriding the correctly-named,
correctly-read flag. Found by extracting the reference's own real
internal representation via model.forward_encode_words(...) directly
and comparing its norm (~82) against a single-layer extraction's norm
(~18) -- an unmistakable, not-subtle divergence once actually measured,
not something a second reading of the config field would have surfaced.
A separate, real memory bug, found while testing on the full ALL
split (7253 instances): EwiserEncoder.score() was holding every
distinct sentence's full [num_words, 117664] logits matrix in memory
simultaneously for the whole call. Fixed by consuming and releasing each
sentence-batch's logits immediately after extracting the scores it
contributes, before encoding the next batch.
The residual gap¶
The small remaining per-dataset gaps (mostly on the smallest eval sets,
e.g. SemEval-2007's 455 instances) are a separately diagnosed and
verified residual, not a further bug: this port's Hugging Face
transformers encoder and the reference's original
pytorch_pretrained_bert encoder produce cosine-~0.99 (not 1.0)
mean-pooled word vectors for bit-identical input and bit-identical
embedding weights -- confirmed directly (matching layer_norm_eps,
matching hidden_act, word-embedding weights diffing by exactly 0.0)
to be accumulated cross-library BERT forward-pass numerics, not a port
bug. A checkpoint's BatchNorm was calibrated tightly enough to the
original library's exact numerics that this residual can flip a close
call on individual instances, with a proportionally bigger effect on
small eval sets than on ALL.
Verifying the training path¶
Reproducing the paper's published results
validates EwiserEncoder's inference path by loading the paper's own
published checkpoints. It never touches
EwiserTrainer at all. This example is the
counterpart: it trains a fresh bert-base-cased
EwiserEncoder from scratch via
EwiserTrainer on a real slice of SemCor, and checks that real learning
happens.
Small subset, and not expected to approach published numbers -- unlike
Reproducing the paper's published results
(a hard acceptance gate for the checkpoint path), this script has no such
bar. EWISER's own published results critically depend on initializing
the output layer (decoder.logits.weight) from externally pretrained
LMMS/SensEmBERT sense embeddings; loaders for those vectors were added in
issue #57 (see
Loading LMMS/SensEmBERT sense embeddings),
but this script doesn't use them -- from-scratch training here starts
that layer randomly. The point is to verify the training code itself --
EwiserTrainer's cross-entropy loss, per-sentence batching, and
freeze-then-thaw schedule -- end to end against a real pretrained encoder
and real data, not to reproduce the paper's numbers.
This script does wire up EWISER's own distinguishing idea, the WordNet
relation-graph propagation step (build_relation_adjacency) -- an
earlier version trained a plain frozen-BERT + FFN classifier with no
graph at all, which meant it wasn't really exercising "EWISER," just a
baseline sense classifier. The graph's practical benefit here is still
capped by construction: SenseVocabulary.from_wn restricts the
vocabulary to gold senses actually observed in this slice, so every
neighbor the graph can propagate from is already a labeled entry -- there
is no truly-unseen synset for the mechanism to generalize to, unlike
training against the full ~117k-synset inventory.
This is also the only place the freeze-then-thaw schedule
(freeze_output_epochs) gets exercised against a real training run, not
just synthetic-data unit tests
(tests/algorithms/wsd/test_ewiser_trainer.py).
"""Trains EwiserEncoder from scratch (real BERT, real gradient descent) on a
real slice of SemCor, and evaluates on held-out documents -- the
training-path counterpart to `examples/ewiser_reproduction.py`, which only
exercises inference against the paper's own published checkpoints.
**Small subset, not the paper's setup, and not expected to approach
published numbers.** Unlike `ewiser_reproduction.py` (a hard acceptance
gate for issue #40), this script has no such bar: EWISER's own published
results critically depend on initializing the output layer
(`decoder.logits.weight`) from externally pretrained LMMS/SensEmBERT
sense embeddings (Section 3.4 of the paper). Issue #57 added loaders for
those vectors (`load_synset_centroid_vectors`/
`build_synset_centroid_vectors_from_lmms` in
`linkingtk.algorithms.wsd._ewiser_sense_embeddings`, exercised against
real downloaded vector files in
`examples/ewiser_pretrained_output_embedding.py`), but this script
doesn't use them -- from-scratch training here starts that layer
randomly, and trains on a small `_TRAIN_DOCS`-document slice of SemCor
for a handful of epochs, not the full corpus. This exists to verify the
*training path itself* --
`EwiserTrainer`'s cross-entropy loss, per-sentence batching, and
freeze/unfreeze schedule all work correctly on real data and a real
pretrained encoder -- not to reproduce the paper's numbers.
This *does* wire up EWISER's own distinguishing idea -- the WordNet
relation-graph propagation step (`build_relation_adjacency`), unlike an
earlier version of this script which trained a plain frozen-BERT + FFN
classifier with no graph at all. Its practical benefit here is still
capped by the restricted vocabulary (see below): structured logits exist
to let the model score synsets it never saw labeled in training, by
propagating from labeled *neighbors* -- but every neighbor this graph can
reach is, by construction, already a labeled vocabulary entry, so there's
no truly-unseen synset for it to generalize to in this setup.
Also exercises the freeze-then-thaw schedule end to end on real data
(`freeze_output_epochs=1`): `decoder.logits.weight` starts frozen for the
first epoch, then unfreezes -- verified here via a real training run, not
just synthetic-data unit tests (see `tests/algorithms/wsd/test_ewiser_trainer.py`
for the synthetic-data version of this same check).
Requires UFSAC 2.1 extracted to `~/data/ufsac-public-2.1/` -- see
`ewiser_reproduction.py`'s docstring.
Run with: `uv run python examples/ewiser_benchmark.py`
"""
from __future__ import annotations
from pathlib import Path
import torch
from linkingtk.algorithms.wsd._ewiser_graph import build_relation_adjacency
from linkingtk.algorithms.wsd._ewiser_vocab import SenseVocabulary
from linkingtk.algorithms.wsd.ewiser import EwiserEncoder, EwiserLinker
from linkingtk.blocking.exact import ExactMatch
from linkingtk.core.entity import label_texts
from linkingtk.datasets.ufsac import UfsacDataset
from linkingtk.eval import Evaluator
from linkingtk.train.arguments import TrainingArguments
from linkingtk.train.ewiser_trainer import EwiserTrainer
_UFSAC_DIR = Path.home() / "data" / "ufsac-public-2.1"
_TRAIN_DOCS = 6
_EVAL_DOCS = 3
_MODEL_NAME = "bert-base-cased" # smaller/faster than the paper's bert-large-cased
def _document_id(mention_id: str) -> str:
# UfsacDataset's positional mention ids (semcor.xml has no native
# per-word id, unlike the raganato_*.xml files) -- "ufsac:{doc}:{sent}:{index}".
return mention_id.split(":")[1]
def main() -> None:
mentions, senses, ground_truth = UfsacDataset(source=str(_UFSAC_DIR / "semcor.xml")).load()
mentions_by_id = {m.id: m for m in mentions}
doc_ids = sorted({_document_id(m.id) for m in mentions})
train_docs = set(doc_ids[:_TRAIN_DOCS])
eval_docs = set(doc_ids[_TRAIN_DOCS : _TRAIN_DOCS + _EVAL_DOCS])
train_gt = [(mid, tid) for mid, tid in ground_truth if _document_id(mid) in train_docs]
eval_gt = [(mid, tid) for mid, tid in ground_truth if _document_id(mid) in eval_docs]
train_mentions = [mentions_by_id[mid] for mid, _tid in train_gt]
eval_mentions = [mentions_by_id[mid] for mid, _tid in eval_gt]
print(
f"{len(train_gt)} train instances ({_TRAIN_DOCS} docs) / "
f"{len(eval_gt)} eval instances ({_EVAL_DOCS} docs)"
)
blocking = ExactMatch(top_k=30)
# Most-frequent-sense baseline: WordNet's own sense order (most-common
# first) with no model at all -- the standard WSD sanity-check floor.
mfs_predictions = [
(mention.id, candidates[0].id)
for mention in eval_mentions
for candidates in [senses.search(label_texts(mention)[0], top_k=1)]
if candidates
]
mfs_report = Evaluator.evaluate(predictions=mfs_predictions, ground_truth=eval_gt)
print(f"Most-frequent-sense baseline: {mfs_report.metrics}")
# From-scratch vocabulary: every gold sense observed in *this slice*
# (train+eval docs only, not all of SemCor's ~33K distinct senses) --
# not the full ~117k WordNet inventory either, since from-scratch
# training has no compatibility need to match it (see
# SenseVocabulary.from_wn).
vocabulary = SenseVocabulary.from_wn([synset_id for _mid, synset_id in train_gt + eval_gt])
print(f"Vocabulary size: {len(vocabulary)}")
# Wires up EWISER's own distinguishing idea -- the WordNet
# graph-propagation step -- so this benchmark actually exercises it,
# not just a plain frozen-BERT + FFN classifier. Built from this
# restricted vocabulary (see the comment above), so its practical
# benefit here is capped: structured logits exist to let the model
# score synsets it never saw labeled by propagating from labeled
# neighbors, but every neighbor this graph can reach is, by
# construction, already a labeled vocabulary entry.
adjacency = build_relation_adjacency(vocabulary)
print(f"Relation graph edges: {adjacency.values().numel()}")
device = "cuda" if torch.cuda.is_available() else "cpu"
encoder = EwiserEncoder(
model_name_or_path=_MODEL_NAME,
vocabulary=vocabulary,
adjacency=adjacency,
max_length=128,
)
encoder.to(device)
linker = EwiserLinker(encoder)
before = Evaluator.evaluate(
predictions=[
(r.source_id, r.target_id)
for r in linker.link(eval_mentions, senses, blocking=blocking)
],
ground_truth=eval_gt,
)
print(f"Untrained precision@1: {before.metrics}")
args = TrainingArguments(
output_dir="./models/ewiser_semcor_subset",
learning_rate=1e-3,
num_epochs=3,
batch_size=8,
device=device,
)
trainer = EwiserTrainer(
model=encoder,
args=args,
train_data=(train_mentions, train_gt),
eval_data=(eval_mentions, eval_gt),
eval_dataset2=senses,
blocking=blocking,
freeze_output_epochs=1,
output_freeze_lr=1e-3,
output_unfreeze_lr=1e-4,
)
trainer.train()
print("Per-epoch held-out Hits@1 (via EwiserTrainer.eval_history):")
for i, epoch_report in enumerate(trainer.eval_history):
print(f" epoch {i + 1}: {epoch_report.metrics}")
after = Evaluator.evaluate(
predictions=[
(r.source_id, r.target_id)
for r in linker.link(eval_mentions, senses, blocking=blocking)
],
ground_truth=eval_gt,
)
print("Trained precision@1 (via EwiserLinker.link, the real production path):")
print(f" {after.metrics}")
if __name__ == "__main__":
main()
Run with:
uv run python examples/ewiser_benchmark.py
2964 train instances (6 docs) / 669 eval instances (3 docs)
Most-frequent-sense baseline: {'precision@1': 0.205, 'recall': 0.205, 'f1': 0.205}
Vocabulary size: 1521
Relation graph edges: 911
Untrained precision@1: {'precision@1': 0.395, 'recall': 0.395, 'f1': 0.395}
Per-epoch held-out Hits@1 (via EwiserTrainer.eval_history):
epoch 1: {'Hits@1': 0.631, 'MRR': 0.767}
epoch 2: {'Hits@1': 0.622, 'MRR': 0.761}
epoch 3: {'Hits@1': 0.641, 'MRR': 0.769}
Trained precision@1 (via EwiserLinker.link, the real production path):
{'precision@1': 0.641, 'recall': 0.641, 'f1': 0.641}
Real, meaningful learning on a real pretrained backbone: 39.5% (untrained)
→ 64.1% (trained, 3 epochs on ~3K instances), clearing the most-frequent-
sense baseline (20.5%) by a wide margin. eval_history's per-epoch
numbers match the independent EwiserLinker.link() check exactly,
confirming the training path and the production inference path agree --
the same regression this repo's own
CrossEncoderTrainer history caught a real bug
on for GlossBERT (see
GlossBERT's training verification).
Hits@1 dips slightly after the first epoch (63.1% → 62.2%) before
recovering -- expected noise on a slice this small (~3K instances, no
pretrained sense-embedding init), not a sign of a training-path bug; the
freeze-then-thaw transition at epoch 1 (decoder.logits.weight unfreezes,
learning rate drops to output_unfreeze_lr) briefly perturbs a model
that had already found a good frozen-output-layer optimum on this small
slice.
Loading LMMS/SensEmBERT sense embeddings¶
Verifying the training path trains
EwiserEncoder from scratch with a randomly initialized output layer,
and notes that the paper's own published numbers depend on initializing
that layer from externally pretrained LMMS/SensEmBERT sense embeddings
instead (Section 4.2, "Output Embeddings", of the paper) -- this example
exercises the issue #57
loaders that make that possible:
load_synset_centroid_vectors and
build_synset_centroid_vectors_from_lmms.
No training happens here. Two real, independent checks:
load_synset_centroid_vectorsagainst the reference's ownres/embeddings/sensembert+lmms.svd512.synset-centroid.vec-- the literal filebin/train-ewiser.sh(in a checkout of SapienzaNLP/ewiser) trains against, already combining SensEmBERT (nouns) with LMMS (other POS) and already SVD-reduced to 512d: the paper's own best ("LMMS + SensEmBERT") configuration from Table 2, baked in. Since this file isSenseVocabulary.from_offsets_file's own token space (both come from the same reference release), coverage should land near 100% -- a real acceptance check, not a fuzzy one.build_synset_centroid_vectors_from_lmmsagainst a raw LMMS release (sensekey-keyed, from danlou/LMMS), building the same shape of tensor from scratch: resolving each sensekey to a synset, centroiding, and reducing with truncated SVD. There's no bit-exact reference to check this against (no raw SensEmBERT file or spec is available -- see the loader module's own docstring for why), so this only confirms the pipeline runs correctly end to end on real data.
Requires res/dictionaries/offsets.txt and
res/embeddings/sensembert+lmms.svd512.synset-centroid.vec from a
checkout of the reference repo, and a raw LMMS vectors file from
danlou/LMMS's "Download Sense
Embeddings" section -- see the script's own docstring for exact paths.
"""Real-data verification of issue #57's sense-embedding loaders
(`linkingtk.algorithms.wsd._ewiser_sense_embeddings`).
No training happens here -- this exercises both loaders against real
downloaded vector files and checks their output against a real
`SenseVocabulary`.
Setup:
1. EWISER's own ``res/dictionaries/offsets.txt`` at `_OFFSETS_PATH` below
(same file `examples/ewiser_reproduction.py` requires).
2. EWISER's own ``res/embeddings/sensembert+lmms.svd512.synset-centroid.vec``
at `_CENTROID_VECTORS_PATH` below -- the literal file
``bin/train-ewiser.sh`` (in a checkout of
https://github.com/SapienzaNLP/ewiser) trains against
(``EMBEDDINGS='../res/embeddings/sensembert+lmms.svd512.synset-centroid.vec'``).
Already combines SensEmBERT (nouns) with LMMS (other POS), already
centroided per synset and SVD-reduced to 512d -- the paper's own best
("LMMS + SensEmBERT") configuration from Table 2.
3. A raw LMMS release vector file (sensekey-keyed, GloVe-style, no
header) at `_RAW_LMMS_VECTORS_PATH` below, e.g. one of the
``lmms-sp-*.vectors.txt`` files from https://github.com/danlou/LMMS's
"Download Sense Embeddings" section.
**Loader 1 check (`load_synset_centroid_vectors`) is a real acceptance
test, not a fuzzy one**: `_CENTROID_VECTORS_PATH` *is* the token space
`SenseVocabulary.from_offsets_file(_OFFSETS_PATH)` was built from (both
come from the same reference release), so coverage should land near
100%, not just "some". This does **not** reproduce a released
checkpoint's own `decoder.logits.weight` bit-for-bit -- the reference's
own two-stage training (`bin/train-ewiser.sh`) freezes this
initialization for stage 1 only, then unfreezes and trains it for 20
more epochs in stage 2, so real, expected drift from this raw
initialization is normal, not a bug to chase.
**Loader 2 check (`build_synset_centroid_vectors_from_lmms`) has no such
bit-exact reference to compare against** (no raw SensEmBERT file/spec is
available -- see the module's own docstring) -- this only checks that
the centroid-and-reduce pipeline runs end to end on real data and
produces coverage/shape/unit-norm results consistent with what the
loader's own unit tests already established on synthetic data.
Run with: `uv run python examples/ewiser_pretrained_output_embedding.py`
"""
from __future__ import annotations
from pathlib import Path
import torch
from linkingtk.algorithms.wsd._ewiser_sense_embeddings import (
build_synset_centroid_vectors_from_lmms,
load_synset_centroid_vectors,
)
from linkingtk.algorithms.wsd._ewiser_vocab import SenseVocabulary
_OFFSETS_PATH = Path.home() / "external" / "ewiser" / "res" / "dictionaries" / "offsets.txt"
_CENTROID_VECTORS_PATH = Path.home() / "Downloads" / "sensembert+lmms.svd512.synset-centroid.vec"
_RAW_LMMS_VECTORS_PATH = Path.home() / "Downloads" / "lmms-sp-wsd.albert-xxlarge-v2.vectors.txt"
_LMMS_TARGET_DIM = 512
def main() -> None:
vocabulary = SenseVocabulary.from_offsets_file(_OFFSETS_PATH)
print(f"SenseVocabulary: {len(vocabulary)} entries (from {_OFFSETS_PATH})")
base = torch.nn.Linear(512, len(vocabulary), bias=False).weight.data.clone()
_vectors, matched = load_synset_centroid_vectors(_CENTROID_VECTORS_PATH, vocabulary, base)
coverage = matched / len(vocabulary)
print(
f"load_synset_centroid_vectors: {matched}/{len(vocabulary)} matched "
f"({coverage:.1%}) from {_CENTROID_VECTORS_PATH}"
)
assert coverage > 0.95, (
"expected near-total coverage (this file is that vocabulary's own "
f"token space), got {coverage:.1%}"
)
raw_base = torch.nn.Linear(_LMMS_TARGET_DIM, len(vocabulary), bias=False).weight.data.clone()
_raw_vectors, raw_matched = build_synset_centroid_vectors_from_lmms(
_RAW_LMMS_VECTORS_PATH, vocabulary, raw_base, target_dim=_LMMS_TARGET_DIM
)
raw_coverage = raw_matched / len(vocabulary)
print(
f"build_synset_centroid_vectors_from_lmms: {raw_matched}/{len(vocabulary)} matched "
f"({raw_coverage:.1%}) from {_RAW_LMMS_VECTORS_PATH}"
)
# A matched row's norm should be ~1.0 (both loaders renormalize after
# reduction) -- pick a real vocabulary entry rather than "any nonzero
# row", since every row already starts nonzero under nn.Linear's own
# random default init.
first_real_index = next(
i for i in range(len(vocabulary)) if vocabulary.synset_id_for(i) is not None
)
sample_norm = raw_base[first_real_index].norm()
print(f" reduced to {_LMMS_TARGET_DIM}d, sample row norm: {sample_norm:.4f}")
if __name__ == "__main__":
main()
Run with:
uv run python examples/ewiser_pretrained_output_embedding.py
SenseVocabulary: 117664 entries (from ~/external/ewiser/res/dictionaries/offsets.txt)
load_synset_centroid_vectors: 117659/117664 matched (100.0%) from ~/Downloads/sensembert+lmms.svd512.synset-centroid.vec
build_synset_centroid_vectors_from_lmms: 117659/117664 matched (100.0%) from ~/Downloads/lmms-sp-wsd.albert-xxlarge-v2.vectors.txt
reduced to 512d, sample row norm: 1.0000
Both loaders match 117,659 of 117,664 vocabulary entries (99.996%) --
as close to total coverage as this vocabulary gets (the handful of
unmatched slots are SenseVocabulary's own reserved special tokens,
which no vector file could ever cover). The raw-LMMS path reaching the
same coverage as the pre-built reference file is notable on its own: this
particular LMMS release (the newer "Reloaded" AIJ 2022 profile, not the
paper's original 2019 one) propagates a vector to essentially every
WordNet sense via gloss embeddings, not just senses observed in annotated
data. The reduced vectors' unit-length norm (1.0000) confirms the
paper's stated final-state invariant actually holds after truncated SVD,
not just before it.
This does not mean a fresh EwiserEncoder built this way reproduces
a released checkpoint's own decoder.logits.weight bit-for-bit: the
reference's own two-stage training freezes this initialization for stage
1 only, then unfreezes and trains it for 20 more epochs in stage 2 --
real, expected drift from this raw initialization, not a bug to chase.
Full-corpus training¶
Verifying the training path only
verifies EwiserTrainer's training code path on a tiny slice (6 docs,
~3K instances, a vocabulary restricted to gold senses observed in that
slice). This is the full-scale counterpart: the real 117,664-entry
WordNet vocabulary, a real from-scratch WordNet relation graph over that
whole vocabulary, and decoder.logits.weight initialized from the real
paper-best "LMMS + SensEmBERT" pretrained sense embeddings (issue #57)
rather than a random init -- trained on all 37,176 sentences of real
SemCor, evaluated on UFSAC's real Raganato et al. framework test sets.
A real bug was fixed to make this possible (issue #58): at the full
117,664-entry vocabulary, StructuredLogits's graph-propagation step
with structured_logits_trainable=True (the default, and what the
reference's own bin/train-ewiser.sh uses in both training stages) used
to OOM -- torch.sparse.mm's default backward materializes a dense
[117664, 117664] gradient (~51.6GB) instead of an O(nnz) one.
Confirmed directly: the OOM's own reported allocation size
(55,379,492,864 bytes) matches 117664**2 * 4 bytes almost exactly, and
was invisible until now (ewiser_benchmark.py's ~1,500-entry vocab
keeps that dense gradient to ~9MB). Fixed with _SparsePropagate, a
custom torch.autograd.Function computing the same gradient via an
O(nnz) indexed gather-multiply-sum -- see
_ewiser_structured_logits.py's own
module docstring, and its test file for a gradcheck-verified
correctness test. Measured directly on real hardware: 53.7ms per
8-sentence batch, 2.9GB peak GPU memory at the full vocabulary with a
trainable graph -- indistinguishable in cost from a non-trainable one.
Acceptance threshold: the closest real published number for this
exact training-data recipe (SemCor only, hypernymy-only relation graph)
is ewiser.semcor_base.pt's own published ALL F1 of 77.0% -- the same
row Reproducing the paper's published results
reproduced to within 0.1 points against the released checkpoint. Not a
hard bit-exact gate here (this script trains its own relation graph from
wn rather than loading the checkpoint's baked-in one -- real, expected
structural differences, documented in _ewiser_graph.py's own
docstring). Two checks, against measured ALL F1: a floor (must clear
the most-frequent-sense baseline by 10 points, or training is actually
broken) and a target (within 5 points of 77.0%, i.e. >= 72.0%,
reported but not asserted -- a multi-hour run's output shouldn't be
discarded on a near miss).
Requires UFSAC 2.1, EWISER's own res/dictionaries/offsets.txt and
res/embeddings/sensembert+lmms.svd512.synset-centroid.vec -- see the
script's own docstring for exact paths.
"""Trains `EwiserEncoder` on the *full* real SemCor corpus (37,176
sentences) with the paper's own architecture and best-known
initialization, then evaluates against UFSAC's real Raganato et al.
framework test sets -- the full-scale counterpart to
`examples/ewiser_benchmark.py`'s small, fast training-path smoke test.
Follow-up to #40 and #57 (LMMS/SensEmBERT output-embedding init): unlike
`ewiser_benchmark.py`, which restricts the vocabulary to gold senses
observed in its own small training slice, this trains a genuinely
**full-inventory** classifier -- the real 117,664-entry WordNet
vocabulary (`SenseVocabulary.from_offsets_file`), a real from-scratch
WordNet relation graph over that whole vocabulary
(`build_relation_adjacency`), and `decoder.logits.weight` initialized
from the real, paper-best "LMMS + SensEmBERT" pretrained sense
embeddings (`load_synset_centroid_vectors`,
`sensembert+lmms.svd512.synset-centroid.vec`) rather than a random init
-- so the graph-propagation step has real unseen synsets to generalize
to, and `EwiserTrainer.freeze_output_epochs` is finally protecting a
real pretrained init, not a no-op.
**Long-running.** Like `glossbert_full_training.py`, meant to run for
hours, unattended (`nohup ... & disown`, not a foreground `uv run`).
**A real bug was fixed to make this possible** (issue #58): at this
vocabulary size, `StructuredLogits`'s graph-propagation step with
`structured_logits_trainable=True` (the default, and what the
reference's own `bin/train-ewiser.sh` uses in both training stages) used
to OOM -- `torch.sparse.mm`'s default backward materializes a dense
``[117664, 117664]`` gradient (~51.6GB) instead of an ``O(nnz)`` one. See
`_ewiser_structured_logits.py`'s module docstring for the fix
(`_SparsePropagate`, a custom `torch.autograd.Function`). With the fix,
measured directly on this sandbox's single RTX 4090: 53.7ms per
8-sentence batch, 2.9GB peak GPU memory, at the *full* vocabulary +
trainable graph -- indistinguishable in cost from a non-trainable graph.
37,176 sentences / batch_size=8 -> ~4,647 batches/epoch -> ~4.2
min/epoch. `bin/train-ewiser.sh`'s own recipe (50 frozen epochs at
lr=1e-4, then 20 thawed epochs at lr=1e-5) is therefore ~5 hours total --
comfortably an overnight run, with margin.
**Hyperparameters, matching the reference's own `bin/train-ewiser.sh`
where GPU-safe**: `decoder_hidden_dim=512` (SVD-reduction target of the
pretrained output embeddings too), `model_name_or_path="bert-large-cased"`
(the paper's own encoder, unlike `ewiser_benchmark.py`'s deliberately
smaller/faster `bert-base-cased` smoke-test choice -- confirmed
comfortably fast enough at this scale by the measurement above),
`freeze_output_epochs=50, output_freeze_lr=1e-4, output_unfreeze_lr=1e-5`.
Candidates are restricted to each mention's own tagged part of speech
(`_PosRestrictedMatch` below, duplicated from `ewiser_reproduction.py` --
each example script here is self-contained), matching the reference's
own candidate generation exactly, for both dev-time monitoring and final
test-time evaluation.
**Acceptance threshold**: the closest real published number for this
exact training-data recipe (SemCor only, hypernymy-only relation graph)
is `ewiser.semcor_base.pt`'s own published ALL F1 of 77.0% (the same row
`ewiser_reproduction.py` reproduced to within 0.1 points against the
released checkpoint). This is **not** a hard bit-exact gate the way
`ewiser_reproduction.py`'s checkpoint reproduction is -- unlike that
script, this one trains its own relation graph from `wn` rather than
loading the checkpoint's baked-in BabelNet-id-based one, and
`_ewiser_graph.py`'s own docstring already documents that a from-scratch
graph "cannot be verified bit-exactly against EWISER's own -- only
structurally". Two checks are reported at the end, against measured ALL
F1:
- **Floor** (asserted): must clear the most-frequent-sense baseline by
at least 10 points -- failing this means training is actually broken,
not just under-tuned.
- **Target** (printed, not asserted -- a multi-hour run's real output
shouldn't be discarded on a near miss): within 5 points of 77.0%
(i.e. >= 72.0%) is reported "met"; a bigger gap is printed plainly, to
be diagnosed rather than silently accepted.
Requires UFSAC 2.1 extracted to `~/data/ufsac-public-2.1/` (same as
`ewiser_reproduction.py`), EWISER's own `res/dictionaries/offsets.txt`
and `res/embeddings/sensembert+lmms.svd512.synset-centroid.vec` (from a
checkout of https://github.com/SapienzaNLP/ewiser) at `_OFFSETS_PATH`/
`_CENTROID_VECTORS_PATH` below.
Run with (from the repo root), detached so it survives the shell exiting:
"""
from __future__ import annotations
import logging
import time
from pathlib import Path
import torch
from linkingtk.algorithms.wsd._ewiser_graph import build_relation_adjacency
from linkingtk.algorithms.wsd._ewiser_sense_embeddings import load_synset_centroid_vectors
from linkingtk.algorithms.wsd._ewiser_vocab import SenseVocabulary
from linkingtk.algorithms.wsd.ewiser import EwiserEncoder, EwiserLinker
from linkingtk.blocking.base import BlockingStrategy
from linkingtk.blocking.exact import ExactMatch
from linkingtk.core.entity import Entity, label_texts
from linkingtk.core.source import EntitySource
from linkingtk.datasets.ufsac import UfsacDataset
from linkingtk.eval import Evaluator
from linkingtk.train.arguments import TrainingArguments
from linkingtk.train.ewiser_trainer import EwiserTrainer
_UFSAC_DIR = Path.home() / "data" / "ufsac-public-2.1"
_OFFSETS_PATH = Path.home() / "external" / "ewiser" / "res" / "dictionaries" / "offsets.txt"
_CENTROID_VECTORS_PATH = Path.home() / "Downloads" / "sensembert+lmms.svd512.synset-centroid.vec"
_OUTPUT_DIR = Path("./models/ewiser_semcor_full")
_DECODER_HIDDEN_DIM = 512
_MODEL_NAME = "bert-large-cased"
_TARGET_ALL_F1 = 0.770
_TARGET_MARGIN = 0.05
_MFS_FLOOR_MARGIN = 0.10
# (UFSAC filename stem, README column label, ewiser.semcor_base.pt's own published F1)
_EVAL_SETS = [
("raganato_senseval2", "SE2", 77.5),
("raganato_senseval3", "SE3", 77.9),
("raganato_semeval2013", "SE13", 76.4),
("raganato_semeval2015", "SE15", 77.8),
("raganato_ALL", "ALL", 77.0),
]
_PENN_TO_WORDNET_POS = {"N": "n", "V": "v", "J": "a", "R": "r"}
def _wordnet_pos_of_synset_id(synset_id: str) -> str | None:
pos = synset_id.rsplit("-", 1)[-1]
if pos == "s": # adjective satellite counts as a plain adjective for POS filtering
return "a"
return pos if pos in "nvar" else None
def _mention_wordnet_pos(mention: Entity) -> str | None:
tag = mention.properties.get("pos")
return _PENN_TO_WORDNET_POS.get(tag[0]) if tag else None
class _PosRestrictedMatch(BlockingStrategy):
"""Wraps `ExactMatch`, keeping only candidates whose WordNet POS matches
the mention's own tagged POS -- see `ewiser_reproduction.py`'s own copy
of this class for the full rationale."""
def __init__(self, top_k: int = 50) -> None:
self._inner = ExactMatch(top_k=top_k)
def candidate_pairs(
self, dataset1: list[Entity], dataset2: list[Entity] | EntitySource
) -> list[tuple[Entity, Entity]]:
pairs = self._inner.candidate_pairs(dataset1, dataset2)
filtered = []
for entity1, entity2 in pairs:
mention_pos = _mention_wordnet_pos(entity1)
if mention_pos is None or _wordnet_pos_of_synset_id(entity2.id) == mention_pos:
filtered.append((entity1, entity2))
return filtered
def main() -> None:
logging.basicConfig(
level=logging.WARNING, format="%(asctime)s %(levelname)s %(message)s", force=True
)
logging.getLogger("linkingtk").setLevel(logging.INFO)
t0 = time.time()
vocabulary = SenseVocabulary.from_offsets_file(_OFFSETS_PATH)
print(f"[{time.time() - t0:.0f}s] SenseVocabulary: {len(vocabulary)} entries")
adjacency = build_relation_adjacency(vocabulary)
print(f"[{time.time() - t0:.0f}s] relation graph: {adjacency.values().numel()} edges")
base = torch.nn.Linear(_DECODER_HIDDEN_DIM, len(vocabulary), bias=False).weight.data.clone()
_vectors, matched = load_synset_centroid_vectors(_CENTROID_VECTORS_PATH, vocabulary, base)
print(
f"[{time.time() - t0:.0f}s] output-embedding init: {matched}/{len(vocabulary)} "
f"matched ({matched / len(vocabulary):.1%})"
)
device = "cuda" if torch.cuda.is_available() else "cpu"
encoder = EwiserEncoder(
model_name_or_path=_MODEL_NAME,
vocabulary=vocabulary,
adjacency=adjacency,
decoder_hidden_dim=_DECODER_HIDDEN_DIM,
structured_logits_trainable=True,
output_embedding_init=base,
max_length=128,
)
encoder.to(device)
linker = EwiserLinker(encoder)
blocking = _PosRestrictedMatch(top_k=50)
train_mentions, senses, train_gt = UfsacDataset(source=str(_UFSAC_DIR / "semcor.xml")).load()
print(f"[{time.time() - t0:.0f}s] loaded {len(train_gt)} SemCor training instances")
# Most-frequent-sense baseline: the same sanity floor ewiser_benchmark.py checks.
mfs_predictions = [
(mention.id, candidates[0].id)
for mention in train_mentions
for candidates in [senses.search(label_texts(mention)[0], top_k=1)]
if candidates
]
mfs_report = Evaluator.evaluate(predictions=mfs_predictions, ground_truth=train_gt)
mfs_f1 = mfs_report.metrics["f1"] or 0.0
print(f"[{time.time() - t0:.0f}s] most-frequent-sense baseline (train set): {mfs_f1:.3f}")
dev_mentions, _senses, dev_gt = UfsacDataset(
source=str(_UFSAC_DIR / "raganato_semeval2007.xml")
).load()
print(f"[{time.time() - t0:.0f}s] loaded {len(dev_gt)} SemEval-2007 dev instances")
args = TrainingArguments(
output_dir=str(_OUTPUT_DIR),
learning_rate=1e-4,
num_epochs=70,
batch_size=8,
device=device,
)
trainer = EwiserTrainer(
model=encoder,
args=args,
train_data=(train_mentions, train_gt),
eval_data=(dev_mentions, dev_gt),
eval_dataset2=senses,
blocking=blocking,
freeze_output_epochs=50,
output_freeze_lr=1e-4,
output_unfreeze_lr=1e-5,
)
print(f"[{time.time() - t0:.0f}s] starting training ({args.num_epochs} epochs)")
trainer.train()
for i, epoch_report in enumerate(trainer.eval_history):
print(f"[{time.time() - t0:.0f}s] epoch {i + 1} dev (SE07): {epoch_report.metrics}")
print(f"[{time.time() - t0:.0f}s] training complete, checkpoint at {_OUTPUT_DIR}/model.pt")
print(f"[{time.time() - t0:.0f}s] evaluating on real held-out test sets")
print(f"{'dataset':<8} {'precision@1':>12} {'published (SemCor)':>20}")
all_f1 = None
for stem, label, published in _EVAL_SETS:
mentions, _senses, ground_truth = UfsacDataset(
source=str(_UFSAC_DIR / f"{stem}.xml")
).load()
results = linker.link(mentions, senses, blocking=blocking)
predictions = [(r.source_id, r.target_id) for r in results]
report = Evaluator.evaluate(predictions=predictions, ground_truth=ground_truth)
score = (report.metrics["precision@1"] or 0.0) * 100
print(f"{label:<8} {score:>11.1f}% {published:>19.1f}%")
if label == "ALL":
all_f1 = report.metrics["f1"] or 0.0
assert all_f1 is not None # "ALL" is always in _EVAL_SETS
print(f"[{time.time() - t0:.0f}s] ALL F1 = {all_f1:.3f}")
assert all_f1 > mfs_f1 + _MFS_FLOOR_MARGIN, (
f"floor check failed: ALL F1 ({all_f1:.3f}) doesn't clear the MFS baseline "
f"({mfs_f1:.3f}) by {_MFS_FLOOR_MARGIN} -- training is likely broken, not just under-tuned"
)
if all_f1 >= _TARGET_ALL_F1 - _TARGET_MARGIN:
print(
f"[{time.time() - t0:.0f}s] target met: ALL F1 within {_TARGET_MARGIN} of "
f"{_TARGET_ALL_F1} (published ewiser.semcor_base.pt)"
)
else:
gap = _TARGET_ALL_F1 - all_f1
print(
f"[{time.time() - t0:.0f}s] below target: ALL F1 {gap:.3f} short of "
f"{_TARGET_ALL_F1} (published ewiser.semcor_base.pt) -- worth diagnosing"
)
print(f"[{time.time() - t0:.0f}s] done")
if __name__ == "__main__":
main()
Run with (detached, since this takes hours):
nohup uv run python -u examples/ewiser_full_training.py \
> /tmp/ewiser_full_training.log 2>&1 & disown
Real run, this sandbox's single RTX 4090: 70 epochs completed in 18,341s (~5.1 hours), matching the ~5-hour planning estimate closely. Loss dropped from 4.99 (epoch 1) to 0.049 (epoch 70), with a visible drop right at the freeze-to-thaw transition (epoch 51, lr 1e-4 -> 1e-5) as the previously-frozen output layer started fine-tuning. Dev (SE07) Hits@1 climbed from 0.669 (epoch 1) to a stable ~0.70-0.71 band by the last ~15 epochs.
Final evaluation on the real held-out UFSAC test sets:
dataset precision@1 published (SemCor)
SE2 76.6% 77.5%
SE3 76.2% 77.9%
SE13 75.3% 76.4%
SE15 75.8% 77.8%
ALL 75.8% 77.0%
ALL F1 = 0.758
target met: ALL F1 within 0.05 of 0.77 (published ewiser.semcor_base.pt)
Both acceptance checks passed. The floor check (ALL F1 must clear
the most-frequent-sense baseline by 10 points) passed without needing to
print anything (an assert, not a verdict line). The target check
passed with real margin: 75.8% is 1.2 points short of the published
77.0%, well inside the 5-point (72.0%) bar -- 3.8 points of margin to
spare. The gap is consistent across every test set (0.9-2.0 points, no
outliers), matching what's expected from the already-documented
structural differences from the reference's own recipe (a from-scratch
WordNet relation graph via wn, not the checkpoint's baked-in
BabelNet-id-based one -- see _ewiser_graph.py's own docstring) rather
than pointing at a bug to chase.