Skip to content

ESC

EscLinker ports Barba, Pasini & Navigli's ESC, which reframes WSD as extractive span comprehension over BART-large. Two examples cover it: loading the paper's own checkpoint, and training the encoder from scratch.

Reproducing the paper's published results

Like EWISER's reproduction, this runs no training at all -- see Verifying the training path for that side instead. EscEncoder.from_checkpoint loads Barba, Pasini & Navigli's own published SemCor checkpoint directly, and evaluates via EscLinker.link, the exact same production code path a freshly trained model would use.

ESC reframes WSD as extractive span comprehension: a mention's context sentence (with the target word bracketed by literal <classify>/ </classify> markers) is paired with every candidate sense's gloss, concatenated into one sequence, and a standard extractive-QA head (AutoModelForQuestionAnswering, BART-large) predicts which candidate's gloss span answers the "question" -- one forward pass per mention, jointly observing every one of that mention's candidates at once. See esc.py's module docstring for the real architectural details confirmed directly from the checkpoint (the plain AutoModelForQuestionAnswering path, not the reference's custom XLNet/ SQuAD-head/special-token variants; the full BART encoder-decoder stack, not encoder-only).

Requires the paper's own SemCor checkpoint (linked from the reference repo's README) at ~/Downloads/escher_semcor_best.ckpt, and UFSAC 2.1 extracted to ~/data/ufsac-public-2.1/ -- see the script's own docstring. Candidates are restricted to each mention's own tagged part of speech, matching the reference's own WordNetDataset.init_dataset (confirmed directly against esc/esc_dataset.py's wn_offsets_from_lemmapos) -- the same convention EWISER's reproduction already uses for EWISER's own candidate generation.

"""Reproduces ESC's published results using the paper's own checkpoint,
through this repo's `EscLinker`/`UfsacDataset` pipeline.

No training happens here -- this loads Barba, Pasini & Navigli's own
released SemCor checkpoint (https://github.com/SapienzaNLP/esc, NAACL
2021) directly into `EscEncoder.from_checkpoint` and evaluates it via
`EscLinker.link()`, the exact same production code path a freshly trained
model would use. **This is a hard acceptance gate for issue #41, not a
best-effort verification pass**: measured numbers are expected to match
the published ones (within a stated, verified methodology difference, not
an assumed one).

Setup:

1. The paper's own SemCor checkpoint (the Google Drive link in the ESC
   README's "Checkpoints" section) at
   ``~/Downloads/escher_semcor_best.ckpt`` (or point `_CHECKPOINT_PATH`
   below elsewhere).
2. UFSAC 2.1 (https://github.com/getalp/UFSAC#content-of-the-repository)
   extracted to `_UFSAC_DIR` below -- same Raganato et al. (2017) WSD
   evaluation framework data ESC's own paper reports numbers against (same
   convention as ``ewiser_reproduction.py``).

Only SE07 and ALL are published in the reference's own README (``SE07:
76.3 | ALL: 80.7`` for the SemCor-only checkpoint) -- the other Raganato
splits are reported below for reference, without a stated target.

Candidates are restricted to each mention's own tagged part of speech,
matching the reference's own `WordNetDataset.init_dataset`
(``wn_offsets_from_lemmapos``, confirmed directly against
``esc/esc_dataset.py``) -- the same convention `ewiser_reproduction.py`
already uses for EWISER's own (separately confirmed) POS-restricted
candidate generation.

Run with: `uv run python examples/esc_reproduction.py`
"""

from __future__ import annotations

from pathlib import Path

import torch

from linkingtk.algorithms.wsd.esc import EscEncoder, EscLinker
from linkingtk.blocking.base import BlockingStrategy
from linkingtk.blocking.exact import ExactMatch
from linkingtk.core.entity import Entity
from linkingtk.core.source import EntitySource
from linkingtk.datasets.ufsac import UfsacDataset
from linkingtk.eval import Evaluator

_CHECKPOINT_PATH = Path.home() / "Downloads" / "escher_semcor_best.ckpt"
_UFSAC_DIR = Path.home() / "data" / "ufsac-public-2.1"

# (UFSAC filename stem, README column label, published F1 or None)
_EVAL_SETS = [
    ("raganato_semeval2007", "SE07", 76.3),
    ("raganato_senseval2", "SE2", None),
    ("raganato_senseval3", "SE3", None),
    ("raganato_semeval2013", "SE13", None),
    ("raganato_semeval2015", "SE15", None),
    ("raganato_ALL", "ALL", 80.7),
]

_PENN_TO_WORDNET_POS = {"N": "n", "V": "v", "J": "a", "R": "r"}


def _wordnet_pos_of_synset_id(synset_id: str) -> str | None:
    pos = synset_id.rsplit("-", 1)[-1]
    if pos == "s":  # adjective satellite counts as a plain adjective for POS filtering
        return "a"
    return pos if pos in "nvar" else None


def _mention_wordnet_pos(mention: Entity) -> str | None:
    tag = mention.properties.get("pos")
    return _PENN_TO_WORDNET_POS.get(tag[0]) if tag else None


class _PosRestrictedMatch(BlockingStrategy):
    """Wraps `ExactMatch`, keeping only candidates whose WordNet POS matches
    the mention's own tagged POS (see module docstring)."""

    def __init__(self, top_k: int = 50) -> None:
        self._inner = ExactMatch(top_k=top_k)

    def candidate_pairs(
        self, dataset1: list[Entity], dataset2: list[Entity] | EntitySource
    ) -> list[tuple[Entity, Entity]]:
        pairs = self._inner.candidate_pairs(dataset1, dataset2)
        filtered = []
        for entity1, entity2 in pairs:
            mention_pos = _mention_wordnet_pos(entity1)
            if mention_pos is None or _wordnet_pos_of_synset_id(entity2.id) == mention_pos:
                filtered.append((entity1, entity2))
        return filtered


def main() -> None:
    device = "cuda" if torch.cuda.is_available() else "cpu"
    blocking = _PosRestrictedMatch(top_k=50)

    encoder = EscEncoder.from_checkpoint(_CHECKPOINT_PATH, forward_batch_size=8)
    encoder.to(device)
    linker = EscLinker(encoder)

    print(f"{'dataset':<8} {'precision@1':>12} {'published':>10}")
    for stem, eval_label, published in _EVAL_SETS:
        mentions, senses, ground_truth = UfsacDataset(source=str(_UFSAC_DIR / f"{stem}.xml")).load()

        results = linker.link(mentions, senses, blocking=blocking)
        predictions = [(result.source_id, result.target_id) for result in results]
        report = Evaluator.evaluate(predictions=predictions, ground_truth=ground_truth)
        score = report.metrics["precision@1"] * 100

        published_str = f"{published:.1f}%" if published is not None else "n/a"
        print(f"{eval_label:<8} {score:>11.1f}% {published_str:>10}")


if __name__ == "__main__":
    main()

Run with:

uv run python examples/esc_reproduction.py

dataset   precision@1  published
SE07            76.3%      76.3%
SE2             81.5%        n/a
SE3             77.8%        n/a
SE13            82.2%        n/a
SE15            83.0%        n/a
ALL             80.6%      80.7%

SE07 matches the published number exactly; ALL -- the paper's own headline metric -- lands 0.1 points off. Only SE07 and ALL are published in the reference's own README; the other splits are reported for reference without a stated target.

Unlike EWISER (#40) and GlossBERT (#39), this reproduction needed no bug hunting at all: reading the checkpoint's own hyper_parameters and state_dict directly (not just the paper text) identified the exact architecture up front -- facebook/bart-large, squad_head=False, use_special_tokens=False, the plain HF AutoModelForQuestionAnswering path -- and porting the joint-sequence tokenization (build_joint_sequence) against that confirmed architecture landed within the checkpoint's own real numbers on the first real run.

Verifying the training path

Reproducing the paper's published results validates EscEncoder's inference path by loading the paper's own published checkpoint. It never touches EscTrainer at all. This example is the counterpart: it trains a fresh facebook/bart-base EscEncoder from scratch via EscTrainer on a real slice of SemCor, and checks that real learning happens.

Small subset and a smaller backbone, not expected to approach published numbers -- unlike Reproducing the paper's published results (a hard acceptance gate for the checkpoint path), this script has no such bar. The reference's own full recipe trains facebook/bart-large for far longer than a handful of epochs on a 9-document slice, and also injects Poisson-sampled distractor-gloss noise during training (add_glosses_noise) -- this uses facebook/bart-base instead (same "smaller/faster backbone for the training-path check" convention EWISER's training verification already uses for EWISER's bert-base-cased vs. the paper's bert-large-cased) and no noise augmentation. The point is to verify the training code itself -- EscTrainer's extractive-QA span-position loss, per-mention candidate resolution/shuffling, and batching -- end to end against a real pretrained encoder and real data, not to reproduce the paper's numbers.

"""Trains `EscEncoder` from scratch (real BART, real gradient descent) on a
real slice of SemCor, and evaluates on held-out documents -- the
training-path counterpart to `examples/esc_reproduction.py`, which only
exercises inference against the paper's own published checkpoint.

**Small subset and a smaller backbone, not the paper's setup, and not
expected to approach published numbers.** Unlike `esc_reproduction.py` (a
hard acceptance gate for issue #41), this script has no such bar: this
exists to verify the *training path itself* -- `EscTrainer`'s extractive
-QA span-position loss, per-mention candidate resolution/shuffling, and
batching all work correctly on real data and a real pretrained encoder --
not to reproduce the paper's numbers. The reference's own full recipe also
does Poisson-sampled distractor-gloss noise during training
(`add_glosses_noise`) and trains ``facebook/bart-large`` for far longer
than a handful of epochs on a 9-document slice; this uses
``facebook/bart-base`` instead (same "smaller/faster backbone for the
training-path check" convention `ewiser_benchmark.py` already uses for
EWISER's `bert-base-cased` vs. the paper's `bert-large-cased`).

Requires UFSAC 2.1 extracted to `~/data/ufsac-public-2.1/` -- see
`esc_reproduction.py`'s docstring.

Run with: `uv run python examples/esc_benchmark.py`
"""

from __future__ import annotations

from pathlib import Path

import torch

from linkingtk.algorithms.wsd.esc import EscEncoder, EscLinker
from linkingtk.blocking.exact import ExactMatch
from linkingtk.core.entity import label_texts
from linkingtk.datasets.ufsac import UfsacDataset
from linkingtk.eval import Evaluator
from linkingtk.train.arguments import TrainingArguments
from linkingtk.train.esc_trainer import EscTrainer

_UFSAC_DIR = Path.home() / "data" / "ufsac-public-2.1"
_TRAIN_DOCS = 6
_EVAL_DOCS = 3
_MODEL_NAME = "facebook/bart-base"  # smaller/faster than the paper's facebook/bart-large


def _document_id(mention_id: str) -> str:
    # UfsacDataset's positional mention ids (semcor.xml has no native
    # per-word id, unlike the raganato_*.xml files) -- "ufsac:{doc}:{sent}:{index}".
    return mention_id.split(":")[1]


def main() -> None:
    mentions, senses, ground_truth = UfsacDataset(source=str(_UFSAC_DIR / "semcor.xml")).load()
    mentions_by_id = {m.id: m for m in mentions}

    doc_ids = sorted({_document_id(m.id) for m in mentions})
    train_docs = set(doc_ids[:_TRAIN_DOCS])
    eval_docs = set(doc_ids[_TRAIN_DOCS : _TRAIN_DOCS + _EVAL_DOCS])
    train_gt = [(mid, tid) for mid, tid in ground_truth if _document_id(mid) in train_docs]
    eval_gt = [(mid, tid) for mid, tid in ground_truth if _document_id(mid) in eval_docs]
    train_mentions = [mentions_by_id[mid] for mid, _tid in train_gt]
    eval_mentions = [mentions_by_id[mid] for mid, _tid in eval_gt]
    print(
        f"{len(train_gt)} train instances ({_TRAIN_DOCS} docs) / "
        f"{len(eval_gt)} eval instances ({_EVAL_DOCS} docs)"
    )

    blocking = ExactMatch(top_k=30)

    # Most-frequent-sense baseline: WordNet's own sense order (most-common
    # first) with no model at all -- the standard WSD sanity-check floor.
    mfs_predictions = [
        (mention.id, candidates[0].id)
        for mention in eval_mentions
        for candidates in [senses.search(label_texts(mention)[0], top_k=1)]
        if candidates
    ]
    mfs_report = Evaluator.evaluate(predictions=mfs_predictions, ground_truth=eval_gt)
    print(f"Most-frequent-sense baseline: {mfs_report.metrics}")

    device = "cuda" if torch.cuda.is_available() else "cpu"
    encoder = EscEncoder(model_name_or_path=_MODEL_NAME, max_length=512, forward_batch_size=4)
    encoder.to(device)
    linker = EscLinker(encoder)

    before = Evaluator.evaluate(
        predictions=[
            (r.source_id, r.target_id)
            for r in linker.link(eval_mentions, senses, blocking=blocking)
        ],
        ground_truth=eval_gt,
    )
    print(f"Untrained precision@1: {before.metrics}")

    args = TrainingArguments(
        output_dir="./models/esc_semcor_subset",
        learning_rate=3e-5,
        num_epochs=5,
        batch_size=4,
        device=device,
    )
    trainer = EscTrainer(
        model=encoder,
        args=args,
        train_data=(train_mentions, train_gt),
        senses=senses,
        eval_data=(eval_mentions, eval_gt),
        blocking=blocking,
    )
    trainer.train()
    print("Per-epoch training loss (via EscTrainer.loss_history):")
    for i, loss in enumerate(trainer.loss_history):
        print(f"  epoch {i + 1}: {loss:.4f}")
    print("Per-epoch held-out Hits@1 (via EscTrainer.eval_history):")
    for i, epoch_report in enumerate(trainer.eval_history):
        print(f"  epoch {i + 1}: {epoch_report.metrics}")

    after = Evaluator.evaluate(
        predictions=[
            (r.source_id, r.target_id)
            for r in linker.link(eval_mentions, senses, blocking=blocking)
        ],
        ground_truth=eval_gt,
    )
    print("Trained precision@1 (via EscLinker.link, the real production path):")
    print(f"  {after.metrics}")


if __name__ == "__main__":
    main()

Run with:

uv run python examples/esc_benchmark.py

2964 train instances (6 docs) / 669 eval instances (3 docs)
Most-frequent-sense baseline: {'precision@1': 0.205, 'recall': 0.205, 'f1': 0.205}
Untrained precision@1: {'precision@1': 0.172, 'recall': 0.172, 'f1': 0.172}
Per-epoch training loss (via EscTrainer.loss_history):
  epoch 1: 1.6357
  epoch 2: 1.1363
  epoch 3: 0.8366
  epoch 4: 0.6036
  epoch 5: 0.4051
Per-epoch held-out Hits@1 (via EscTrainer.eval_history):
  epoch 1: {'Hits@1': 0.426, 'MRR': 0.605}
  epoch 2: {'Hits@1': 0.495, 'MRR': 0.652}
  epoch 3: {'Hits@1': 0.505, 'MRR': 0.660}
  epoch 4: {'Hits@1': 0.505, 'MRR': 0.664}
  epoch 5: {'Hits@1': 0.489, 'MRR': 0.657}
Trained precision@1 (via EscLinker.link, the real production path):
  {'precision@1': 0.489, 'recall': 0.489, 'f1': 0.489}

Real, meaningful learning on a real pretrained backbone: the QA span-position loss drops monotonically every epoch (1.64 → 0.41), and held-out precision@1 climbs from 17.2% (untrained) to 48.9% (trained, 5 epochs on ~3K instances) -- clearing the most-frequent-sense baseline (20.5%) by a wide margin. eval_history's final epoch matches the independent EscLinker.link() check exactly (48.9% both ways), confirming the training path and the production inference path agree -- the same regression this repo's own CrossEncoderTrainer history caught a real bug on for GlossBERT (see GlossBERT's training verification).

Hits@1 dips slightly after the peak at epochs 3-4 (50.5% → 48.9%) -- expected noise from continuing to fine-tune a small backbone on a slice this small (~3K instances, 5 epochs, no early stopping), not a sign of a training-path bug; loss_history keeps decreasing every epoch throughout, confirming the optimizer itself is behaving correctly even as held-out generalization plateaus.