Datasets

See also the Datasets page for usage examples.

base

Abstract interface for dataset loaders.

DatasetLoader

Bases: ABC

Loads a linking dataset as a pair of entity lists plus ground truth.

Concrete loaders fetch data from its original source where possible, falling back to a republished copy on the Hugging Face Hub.

load abstractmethod

load() -> tuple[
    list[Entity],
    list[Entity] | EntitySource,
    list[tuple[str, str]],
]

Load the dataset.

Returns:

Type	Description
list[Entity]	A tuple of (dataset1, dataset2, ground_truth) where
list[Entity] | EntitySource	ground_truth is a list of (source_id, target_id) pairs.
list[tuple[str, str]]	dataset2 is an EntitySource
tuple[list[Entity], list[Entity] | EntitySource, list[tuple[str, str]]]	rather than a materialized list for loaders whose targets have
tuple[list[Entity], list[Entity] | EntitySource, list[tuple[str, str]]]	no fixed candidate list to enumerate (e.g. a WSD corpus tagged
tuple[list[Entity], list[Entity] | EntitySource, list[tuple[str, str]]]	against a full dictionary) -- see
tuple[list[Entity], list[Entity] | EntitySource, list[tuple[str, str]]]	SemCorDataset and
tuple[list[Entity], list[Entity] | EntitySource, list[tuple[str, str]]]	UfsacDataset.

Source code in src/linkingtk/datasets/base.py

@abstractmethod
def load(self) -> tuple[list[Entity], list[Entity] | EntitySource, list[tuple[str, str]]]:
    """Load the dataset.

    Returns:
        A tuple of ``(dataset1, dataset2, ground_truth)`` where
        ``ground_truth`` is a list of ``(source_id, target_id)`` pairs.
        ``dataset2`` is an [EntitySource][linkingtk.core.source.EntitySource]
        rather than a materialized list for loaders whose targets have
        no fixed candidate list to enumerate (e.g. a WSD corpus tagged
        against a full dictionary) -- see
        [SemCorDataset][linkingtk.datasets.semcor.SemCorDataset] and
        [UfsacDataset][linkingtk.datasets.ufsac.UfsacDataset].
    """

GraphDatasetLoader

Bases: DatasetLoader

A DatasetLoader whose datasets carry a supporting relational graph.

Implemented by the graph-heavy real-world EA dataset loaders (DBP15K, OpenEA, ICEWS, WordNet-Wikidata) that back KGE-based linking -- unlike plain DatasetLoader, load() alone isn't enough for those, since the relational structure (not just entity labels) is the point.

load_graphs abstractmethod

load_graphs() -> tuple[Graph, Graph]

Load each side's supporting relational graph.

Returns:

Type	Description
Graph	(graph1, graph2), each in the same shape accepted by
Graph	link's graph
tuple[Graph, Graph]	parameter -- here always a list[Triple] of
tuple[Graph, Graph]	(subject_id, predicate_id, object_id).

Source code in src/linkingtk/datasets/base.py

@abstractmethod
def load_graphs(self) -> tuple[Graph, Graph]:
    """Load each side's supporting relational graph.

    Returns:
        ``(graph1, graph2)``, each in the same shape accepted by
        [link][linkingtk.algorithms.base.BaseLinker.link]'s ``graph``
        parameter -- here always a ``list[Triple]`` of
        ``(subject_id, predicate_id, object_id)``.
    """

_util

Helpers shared across dataset loader modules.

fetch_cached

fetch_cached(
    url: str,
    cache_dir: Path | None = None,
    headers: dict[str, str] | None = None,
) -> bytes

Fetch url's bytes, caching to disk under cache_dir.

file:// URLs are read directly and never cached -- they're already local, used to point tests at small fixture files/archives instead of the real (multi-MB) hosted datasets.

Parameters:

Name	Type	Description	Default
url	str	The URL to fetch.	required
cache_dir	Path | None	Directory to cache downloads in. Defaults to ~/.cache/linkingtk/downloads.	None
headers	dict[str, str] | None	Extra request headers, e.g. a User-Agent some hosts require (Wikipedia's API 403s on urllib's default one -- see fetch_wikipedia_extracts).	None

Returns:

Type	Description
bytes	The response body.

Source code in src/linkingtk/datasets/_util.py

def fetch_cached(
    url: str, cache_dir: Path | None = None, headers: dict[str, str] | None = None
) -> bytes:
    """Fetch ``url``'s bytes, caching to disk under ``cache_dir``.

    ``file://`` URLs are read directly and never cached -- they're already
    local, used to point tests at small fixture files/archives instead of
    the real (multi-MB) hosted datasets.

    Args:
        url: The URL to fetch.
        cache_dir: Directory to cache downloads in. Defaults to
            ``~/.cache/linkingtk/downloads``.
        headers: Extra request headers, e.g. a ``User-Agent`` some hosts
            require (Wikipedia's API 403s on urllib's default one -- see
            [fetch_wikipedia_extracts][linkingtk.datasets.aida_conll.fetch_wikipedia_extracts]).

    Returns:
        The response body.
    """
    if url.startswith("file://"):
        with urlopen(url) as response:
            data: bytes = response.read()
        return data

    cache_dir = cache_dir if cache_dir is not None else _DEFAULT_CACHE_DIR
    cache_path = cache_dir / hashlib.sha256(url.encode()).hexdigest()
    if cache_path.exists():
        return cache_path.read_bytes()

    with urlopen(Request(url, headers=headers or {})) as response:
        fetched: bytes = response.read()
    cache_dir.mkdir(parents=True, exist_ok=True)
    cache_path.write_bytes(fetched)
    return fetched

label_from_raw

label_from_raw(raw: str) -> str

A human-readable label from a raw dataset value.

URIs are reduced to their local name with underscores humanized (e.g. http://dbpedia.org/resource/Big_Cat -> Big Cat); anything else (already-plain-text entries, as some datasets have) passes through unchanged.

Source code in src/linkingtk/datasets/_util.py

def label_from_raw(raw: str) -> str:
    """A human-readable label from a raw dataset value.

    URIs are reduced to their local name with underscores humanized (e.g.
    ``http://dbpedia.org/resource/Big_Cat`` -> ``Big Cat``); anything else
    (already-plain-text entries, as some datasets have) passes through
    unchanged.
    """
    if raw.startswith(("http://", "https://")):
        return local_name(raw).replace("_", " ")
    return raw

local_name

local_name(uri: str) -> str

The local (fragment/path-final) part of a URI, e.g. .../Big_Cat -> Big_Cat.

Source code in src/linkingtk/datasets/_util.py

def local_name(uri: str) -> str:
    """The local (fragment/path-final) part of a URI, e.g. ``.../Big_Cat`` -> ``Big_Cat``."""
    return uri.rsplit("#", 1)[-1].rsplit("/", 1)[-1]

parse_literal_value

parse_literal_value(raw: str) -> str

Strip an RDF literal's surrounding quotes and ^^<datatype>/@lang suffix.

E.g. '"3.451E8"^^<http://dbpedia.org/datatype/usDollar>' -> "3.451E8". Plain unquoted values (as this format also has, e.g. "Left") pass through unchanged.

Source code in src/linkingtk/datasets/_util.py

def parse_literal_value(raw: str) -> str:
    """Strip an RDF literal's surrounding quotes and ``^^<datatype>``/``@lang`` suffix.

    E.g. ``'"3.451E8"^^<http://dbpedia.org/datatype/usDollar>'`` ->
    ``"3.451E8"``. Plain unquoted values (as this format also has, e.g.
    ``"Left"``) pass through unchanged.
    """
    match = _LITERAL_RE.match(raw)
    return match.group(1) if match else raw

Toy datasets

toy

Small, hand-curated toy datasets for quick end-to-end testing.

Unlike the other loaders in this package, these don't fetch from an external source: they're bundled directly in the code so every algorithm in the MVP milestone can be exercised without network access. Each one picks classically ambiguous words/mentions with two senses/candidates apiece, so a context-blind baseline (e.g. picking the first candidate) would score no better than chance.

ToyEADataset

Bases: DatasetLoader

Three same-city entities across two knowledge graphs (6 entities).

kg2's labels append the country name (e.g. "Paris, France"), so an exact label match misses every pair — a blocking strategy tolerant of partial overlap (e.g. LabelOverlap) is needed to find the candidates.

ToyELDataset

Bases: DatasetLoader

Six ambiguous-name mentions against two knowledge-base entries apiece (18 entities).

ToyWSADataset

Bases: DatasetLoader

Four ambiguous-word senses across two differently-worded dictionaries (8 entities).

Each dictionary glosses the same four senses of "mouse" and "bat" in different words, so a context-blind baseline would score no better than chance between the two same-label candidates.

ToyWSDDataset

Bases: DatasetLoader

Four ambiguous-word mentions against two senses apiece (12 entities).

(The classic "bank" example is deliberately left out here since it's already the walkthrough example in examples/lesk_wsd.py and the docs — this dataset covers different words so the two don't just duplicate each other.)

Naisc ontology-matching datasets

naisc

Loaders for Naisc's ontology-matching toy benchmarks.

Source: https://github.com/insight-centre/naisc/tree/master/datasets

Each dataset is a pair of small OWL ontologies (left.rdf/right.rdf) plus a reference alignment (align.rdf) listing known-correct class pairs between them. Both were originally used in the OAEI ontology matching campaigns; only the classes (not object/data properties) are loaded as entities here, which keeps both sides comfortably under 100 entities for ConferenceDataset and lets Entity.description stay unused (these ontologies rarely carry rdfs:comment) in favor of just rdfs:label.

AnatomyDataset

AnatomyDataset(base_url: str | None = None)

Bases: _NaiscToyDataset

The OAEI 'anatomy' track: mouse-anatomy vs. NCI Thesaurus ontologies.

Despite being listed alongside ConferenceDataset as a toy dataset in Naisc, this is the full anatomy track (thousands of classes per side) — use ConferenceDataset if you need a dataset small enough for quick, exhaustive testing.

Source code in src/linkingtk/datasets/naisc.py

def __init__(self, base_url: str | None = None) -> None:
    """Create the loader.

    Args:
        base_url: Override for where ``left.rdf``/``right.rdf``/``align.rdf``
            are fetched from (a URL or ``file://`` path). Defaults to
            this dataset's directory in the Naisc GitHub repository.
    """
    self.base_url = base_url if base_url is not None else f"{_BASE_URL}/{self._name}"

ConferenceDataset

ConferenceDataset(base_url: str | None = None)

Bases: _NaiscToyDataset

The OAEI 'conference' track: matching academic-conference ontologies.

Well under 100 classes per side, making it suitable as a genuinely small Entity Alignment / Word Sense Alignment toy benchmark.

Source code in src/linkingtk/datasets/naisc.py

def __init__(self, base_url: str | None = None) -> None:
    """Create the loader.

    Args:
        base_url: Override for where ``left.rdf``/``right.rdf``/``align.rdf``
            are fetched from (a URL or ``file://`` path). Defaults to
            this dataset's directory in the Naisc GitHub repository.
    """
    self.base_url = base_url if base_url is not None else f"{_BASE_URL}/{self._name}"

Shared zip-archive dataset base

kg_zip

Shared base for datasets hosted as a zip of numeric-id KG files.

Neither DBP15K's canonical host (Google Drive, via the JAPE paper's site) nor OpenEA's (Dropbox/Figshare) is a stable, programmatically fetchable URL. Two GitHub repos rehost the same data as plain zips in an identical format instead -- github.com/DexterZeng/EntMatcher (DBP15K plus OpenEA's EN-FR/EN-DE/D-W/D-Y-15K sets) and github.com/jxh4945777/Simple-HHEA (ICEWS) -- which is what the concrete loaders in linkingtk.datasets.dbp15k, linkingtk.datasets.openea and linkingtk.datasets.icews point at.

_KGZipDataset

_KGZipDataset(
    zip_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: GraphDatasetLoader

Base loader for the ent_ids/triples zip format.

Within the zip, a dataset lives under _folder with per-side ent_ids_N (id<TAB>uri_or_label) and triples_N (whitespace- separated columns; the first three are always subject_id predicate_id object_id -- extra trailing columns, as ICEWS has for event timestamps, are ignored) files, plus one or more _ground_truth_files listing known-correct id1<TAB>id2 alignment pairs (concatenated if there's more than one, e.g. separate train/test files with no single combined file).

Some datasets additionally ship a native supervised train/test/ validation split of that same ground truth (as separate files) -- where _train_ground_truth_file/_test_ground_truth_file are set, load_splits exposes that split directly, instead of load()'s single concatenated list.

Create the loader.

Parameters:

Name	Type	Description	Default
zip_url	str | None	Override for where the dataset zip is fetched from (a URL or file:// path). Defaults to this dataset's usual hosted archive.	None
cache_dir	Path | None	Override for the download cache directory. Ignored for file:// URLs, which are never cached.	None

Source code in src/linkingtk/datasets/kg_zip.py

def __init__(self, zip_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        zip_url: Override for where the dataset zip is fetched from (a
            URL or ``file://`` path). Defaults to this dataset's usual
            hosted archive.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.zip_url = zip_url if zip_url is not None else self._zip_url
    self.cache_dir = cache_dir

load_splits

load_splits() -> tuple[
    list[tuple[str, str]],
    list[tuple[str, str]],
    list[tuple[str, str]],
]

Load this dataset's native train/test/validation ground-truth split.

Unlike load()'s single concatenated ground_truth, this preserves the dataset's own supervised split -- the train pairs are meant to be added to the training graph as seed alignment triples (see KGELinker.fit), with the test (and, where available, validation) pairs held out for ranked evaluation.

Returns:

Type	Description
list[tuple[str, str]]	(train_pairs, test_pairs, val_pairs). val_pairs is
list[tuple[str, str]]	[] for datasets with no native validation split.

Source code in src/linkingtk/datasets/kg_zip.py

def load_splits(
    self,
) -> tuple[list[tuple[str, str]], list[tuple[str, str]], list[tuple[str, str]]]:
    """Load this dataset's native train/test/validation ground-truth split.

    Unlike ``load()``'s single
    concatenated ``ground_truth``, this preserves the dataset's own
    supervised split -- the train pairs are meant to be added to the
    training graph as seed alignment triples (see
    [KGELinker.fit][linkingtk.algorithms.ea.kge.KGELinker.fit]), with
    the test (and, where available, validation) pairs held out for
    ranked evaluation.

    Returns:
        ``(train_pairs, test_pairs, val_pairs)``. ``val_pairs`` is
        ``[]`` for datasets with no native validation split.
    """
    with self._open_zip() as archive:
        _, ids1 = self._entities(archive, 1)
        _, ids2 = self._entities(archive, 2)
        train = self._pairs_from_file(archive, ids1, ids2, self._train_ground_truth_file)
        test = self._pairs_from_file(archive, ids1, ids2, self._test_ground_truth_file)
        val = (
            self._pairs_from_file(archive, ids1, ids2, self._val_ground_truth_file)
            if self._val_ground_truth_file is not None
            else []
        )
    return train, test, val

DBP15K

dbp15k

Loaders for DBP15K, the classic cross-lingual DBpedia EA benchmark.

Source: rehosted (see linkingtk.datasets.kg_zip for why) at https://github.com/DexterZeng/EntMatcher's data.zip, under data/{zh_en,ja_en,fr_en}/.

DBP15KFrEnDataset

DBP15KFrEnDataset(
    zip_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _KGZipDataset

DBP15K's French-English pair (~15K entities per side).

Source code in src/linkingtk/datasets/kg_zip.py

def __init__(self, zip_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        zip_url: Override for where the dataset zip is fetched from (a
            URL or ``file://`` path). Defaults to this dataset's usual
            hosted archive.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.zip_url = zip_url if zip_url is not None else self._zip_url
    self.cache_dir = cache_dir

DBP15KJaEnDataset

DBP15KJaEnDataset(
    zip_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _KGZipDataset

DBP15K's Japanese-English pair (~15K entities per side).

Source code in src/linkingtk/datasets/kg_zip.py

def __init__(self, zip_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        zip_url: Override for where the dataset zip is fetched from (a
            URL or ``file://`` path). Defaults to this dataset's usual
            hosted archive.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.zip_url = zip_url if zip_url is not None else self._zip_url
    self.cache_dir = cache_dir

DBP15KZhEnDataset

DBP15KZhEnDataset(
    zip_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _KGZipDataset

DBP15K's Chinese-English pair (~15K entities per side).

Source code in src/linkingtk/datasets/kg_zip.py

def __init__(self, zip_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        zip_url: Override for where the dataset zip is fetched from (a
            URL or ``file://`` path). Defaults to this dataset's usual
            hosted archive.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.zip_url = zip_url if zip_url is not None else self._zip_url
    self.cache_dir = cache_dir

OpenEA

openea

Loaders for OpenEA's sampled multilingual and homogeneous EA benchmarks.

Source: rehosted (see linkingtk.datasets.kg_zip for why) at https://github.com/DexterZeng/EntMatcher's data.zip, under data/{en_fr_15k_V1,en_de_15k_V1,dbp_wd_15k_V1,dbp_yg_15k_V1}/.

Only the 15K/V1 size is available from this mirror -- OpenEA's name-bias-fixed V2 release and the EN-FR/EN-DE 100K sizes are Figshare-only (no stable fetchable URL, see linkingtk.datasets.kg_zip's module docstring); D-W/D-Y do have a 100K variant in this same zip (dbp_wd_100/dbp_yg_100) but it's not exposed here to keep the multilingual and homogeneous sets available at consistent sizes.

DbpediaWikidata15KDataset

DbpediaWikidata15KDataset(
    zip_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _KGZipDataset

OpenEA's D-W-15K-V1: DBpedia-Wikidata (homogeneous, ~15K/side).

Source code in src/linkingtk/datasets/kg_zip.py

def __init__(self, zip_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        zip_url: Override for where the dataset zip is fetched from (a
            URL or ``file://`` path). Defaults to this dataset's usual
            hosted archive.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.zip_url = zip_url if zip_url is not None else self._zip_url
    self.cache_dir = cache_dir

DbpediaYago15KDataset

DbpediaYago15KDataset(
    zip_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _KGZipDataset

OpenEA's D-Y-15K-V1: DBpedia-YAGO (homogeneous, ~15K/side).

Source code in src/linkingtk/datasets/kg_zip.py

def __init__(self, zip_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        zip_url: Override for where the dataset zip is fetched from (a
            URL or ``file://`` path). Defaults to this dataset's usual
            hosted archive.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.zip_url = zip_url if zip_url is not None else self._zip_url
    self.cache_dir = cache_dir

EnDe15KDataset

EnDe15KDataset(
    zip_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _KGZipDataset

OpenEA's EN-DE-15K-V1: English-German DBpedia (multilingual, ~15K/side).

Source code in src/linkingtk/datasets/kg_zip.py

def __init__(self, zip_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        zip_url: Override for where the dataset zip is fetched from (a
            URL or ``file://`` path). Defaults to this dataset's usual
            hosted archive.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.zip_url = zip_url if zip_url is not None else self._zip_url
    self.cache_dir = cache_dir

EnFr15KDataset

EnFr15KDataset(
    zip_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _KGZipDataset

OpenEA's EN-FR-15K-V1: English-French DBpedia (multilingual, ~15K/side).

Source code in src/linkingtk/datasets/kg_zip.py

def __init__(self, zip_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        zip_url: Override for where the dataset zip is fetched from (a
            URL or ``file://`` path). Defaults to this dataset's usual
            hosted archive.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.zip_url = zip_url if zip_url is not None else self._zip_url
    self.cache_dir = cache_dir

OpenEA (native format, with attributes)

openea_native

Loaders for OpenEA's own native per-triple format, including attribute triples.

Source: rehosted on the Hugging Face Hub by the matchbench project (https://huggingface.co/matchbench) as plain zips of OpenEA's own raw-URI rel_triples_N/attr_triples_N/train_links/valid_links/ test_links files -- unlike linkingtk.datasets.openea's EntMatcher- rehosted, numeric-id, structural-only loaders, this format ships attribute triples too, needed by EA methods that use attribute signal (JAPE #28, and later AttrE/IMUSE/MultiKE).

Not a drop-in replacement for linkingtk.datasets.openea's loaders: cross-checked while investigating #28 by downloading both zips and comparing entity URI sets -- only ~10% of EnFr15KDataset's (EntMatcher rehost) entity URIs coincide with EnFr15KAttrDataset's (this module's) entities. They're two independently-sampled cuts of "EN-FR-15K" that happen to share the same split-size convention (OpenEA's standard 20/10/70% ratio), not the same entity roster. So these loaders are entirely self-contained (own entity ids, own relation/attribute triples, own train/valid/test split, all from one internally-consistent source) rather than joined onto linkingtk.datasets.openea's loaders via URI matching, which would only cover a small fraction of entities. Use this module specifically when attribute triples are needed; use linkingtk.datasets.openea otherwise (already used/verified by MTransE #26, IPTransE #27).

DbpediaWikidata15KAttrDataset

DbpediaWikidata15KAttrDataset(
    zip_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _OpenEANativeDataset

OpenEA's D-W-15K-V1, native format with attributes (DBpedia-Wikidata).

Source code in src/linkingtk/datasets/openea_native.py

def __init__(self, zip_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        zip_url: Override for where the dataset zip is fetched from (a
            URL or ``file://`` path). Defaults to this dataset's usual
            hosted archive.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.zip_url = zip_url if zip_url is not None else self._zip_url
    self.cache_dir = cache_dir

DbpediaYago15KAttrDataset

DbpediaYago15KAttrDataset(
    zip_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _OpenEANativeDataset

OpenEA's D-Y-15K-V1, native format with attributes (DBpedia-YAGO).

Source code in src/linkingtk/datasets/openea_native.py

def __init__(self, zip_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        zip_url: Override for where the dataset zip is fetched from (a
            URL or ``file://`` path). Defaults to this dataset's usual
            hosted archive.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.zip_url = zip_url if zip_url is not None else self._zip_url
    self.cache_dir = cache_dir

EnDe15KAttrDataset

EnDe15KAttrDataset(
    zip_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _OpenEANativeDataset

OpenEA's EN-DE-15K-V1, native format with attributes (English-German DBpedia).

Source code in src/linkingtk/datasets/openea_native.py

def __init__(self, zip_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        zip_url: Override for where the dataset zip is fetched from (a
            URL or ``file://`` path). Defaults to this dataset's usual
            hosted archive.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.zip_url = zip_url if zip_url is not None else self._zip_url
    self.cache_dir = cache_dir

EnFr15KAttrDataset

EnFr15KAttrDataset(
    zip_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _OpenEANativeDataset

OpenEA's EN-FR-15K-V1, native format with attributes (English-French DBpedia).

Source code in src/linkingtk/datasets/openea_native.py

def __init__(self, zip_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        zip_url: Override for where the dataset zip is fetched from (a
            URL or ``file://`` path). Defaults to this dataset's usual
            hosted archive.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.zip_url = zip_url if zip_url is not None else self._zip_url
    self.cache_dir = cache_dir

_OpenEANativeDataset

_OpenEANativeDataset(
    zip_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: GraphDatasetLoader

Base loader for OpenEA's native per-triple format (raw URIs, with attributes).

Entities are implicit (no dedicated id file, unlike linkingtk.datasets.kg_zip's ent_ids_N): each side's roster is the sorted set of every subject/object URI appearing in that side's rel_triples_N, enumerated into {prefix}{index} ids. Attribute triples or split pairs referencing a URI absent from rel_triples_N are dropped (tolerated, same posture as linkingtk.datasets.kg_zip's dangling-reference handling). The zip has no folder prefix -- files live at its root.

Create the loader.

Parameters:

Name	Type	Description	Default
zip_url	str | None	Override for where the dataset zip is fetched from (a URL or file:// path). Defaults to this dataset's usual hosted archive.	None
cache_dir	Path | None	Override for the download cache directory. Ignored for file:// URLs, which are never cached.	None

Source code in src/linkingtk/datasets/openea_native.py

def __init__(self, zip_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        zip_url: Override for where the dataset zip is fetched from (a
            URL or ``file://`` path). Defaults to this dataset's usual
            hosted archive.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.zip_url = zip_url if zip_url is not None else self._zip_url
    self.cache_dir = cache_dir

load_attribute_triples

load_attribute_triples() -> tuple[
    list[Triple], list[Triple]
]

Load each side's (entity_id, predicate_uri, value) attribute triples.

value has RDF literal quoting/^^<datatype>/@lang suffixes stripped via parse_literal_value. Not part of the GraphDatasetLoader interface -- only this loader family provides it so far; promote to a proper mixin/ABC once a second attribute-bearing loader family exists.

Returns:

Type	Description
list[Triple]	(attribute_triples1, attribute_triples2). Triples whose
list[Triple]	subject URI doesn't appear in that side's rel_triples_N
tuple[list[Triple], list[Triple]]	are dropped.

Source code in src/linkingtk/datasets/openea_native.py

def load_attribute_triples(self) -> tuple[list[Triple], list[Triple]]:
    """Load each side's ``(entity_id, predicate_uri, value)`` attribute triples.

    ``value`` has RDF literal quoting/``^^<datatype>``/``@lang``
    suffixes stripped via
    [parse_literal_value][linkingtk.datasets._util.parse_literal_value].
    Not part of the [GraphDatasetLoader][linkingtk.datasets.base.GraphDatasetLoader]
    interface -- only this loader family provides it so far; promote
    to a proper mixin/ABC once a second attribute-bearing loader
    family exists.

    Returns:
        ``(attribute_triples1, attribute_triples2)``. Triples whose
        subject URI doesn't appear in that side's ``rel_triples_N``
        are dropped.
    """
    with self._open_zip() as archive:
        ids1, ids2 = self._entity_ids(archive, 1), self._entity_ids(archive, 2)
        return self._attr_triples(archive, 1, ids1), self._attr_triples(archive, 2, ids2)

load_splits

load_splits() -> tuple[
    list[tuple[str, str]],
    list[tuple[str, str]],
    list[tuple[str, str]],
]

Load this dataset's native train/test/validation ground-truth split.

Returns:

Type	Description
list[tuple[str, str]]	(train_pairs, test_pairs, val_pairs) -- see
list[tuple[str, str]]	_KGZipDataset.load_splits
list[tuple[str, str]]	for how these are meant to be used.

Source code in src/linkingtk/datasets/openea_native.py

def load_splits(
    self,
) -> tuple[list[tuple[str, str]], list[tuple[str, str]], list[tuple[str, str]]]:
    """Load this dataset's native train/test/validation ground-truth split.

    Returns:
        ``(train_pairs, test_pairs, val_pairs)`` -- see
        [_KGZipDataset.load_splits][linkingtk.datasets.kg_zip._KGZipDataset.load_splits]
        for how these are meant to be used.
    """
    with self._open_zip() as archive:
        ids1, ids2 = self._entity_ids(archive, 1), self._entity_ids(archive, 2)
        train = self._pairs_from_file(archive, ids1, ids2, "train_links")
        test = self._pairs_from_file(archive, ids1, ids2, "test_links")
        val = self._pairs_from_file(archive, ids1, ids2, "valid_links")
    return train, test, val

ICEWS

icews

Loader for ICEWS-Wiki, a heterogeneous event-KG-to-Wikipedia EA benchmark.

Source: https://github.com/jxh4945777/Simple-HHEA's data/icews_wiki.zip -- same ent_ids/triples format as linkingtk.datasets.dbp15k/linkingtk.datasets.openea (triples_N here has two extra trailing timestamp-id columns, ignored by the shared parser's load_graphs(); see load_temporal_graphs for a loader that keeps them).

icews_yago (the other dataset in that repo) is intentionally not implemented: it's split across a 3-part archive (icews_yago.z01/.z02/.zip) rather than a single zip, which would need real multi-part-archive reassembly rather than the single fetch_cached() + zipfile this module and its siblings share.

IcewsWikiDataset

IcewsWikiDataset(
    zip_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _KGZipDataset

ICEWS (event KG) aligned to Wikipedia entries.

Source code in src/linkingtk/datasets/kg_zip.py

def __init__(self, zip_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        zip_url: Override for where the dataset zip is fetched from (a
            URL or ``file://`` path). Defaults to this dataset's usual
            hosted archive.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.zip_url = zip_url if zip_url is not None else self._zip_url
    self.cache_dir = cache_dir

load_relation_labels

load_relation_labels() -> dict[str, str]

Local predicate_id -> its human-readable relation label.

Merges rel_ids_1/rel_ids_2 -- confirmed by direct inspection to be globally unique across sides (side 2 continues numbering from side 1's max), so unlike ent_ids_N's per-side ids, no per-side prefix is needed here. Needed for ChatEALinker's prompts, which render real relation names (e.g. "member of") rather than the raw numeric ids load_graphs/ load_temporal_graphs keep in each triple's relation slot.

Source code in src/linkingtk/datasets/icews.py

def load_relation_labels(self) -> dict[str, str]:
    """Local ``predicate_id`` -> its human-readable relation label.

    Merges ``rel_ids_1``/``rel_ids_2`` -- confirmed by direct
    inspection to be globally unique across sides (side 2 continues
    numbering from side 1's max), so unlike ``ent_ids_N``'s per-side
    ids, no per-side prefix is needed here. Needed for
    [ChatEALinker][linkingtk.algorithms.ea.chatea.ChatEALinker]'s
    prompts, which render real relation names (e.g. ``"member of"``)
    rather than the raw numeric ids ``load_graphs``/
    [load_temporal_graphs][linkingtk.datasets.icews.IcewsWikiDataset.load_temporal_graphs]
    keep in each triple's relation slot.
    """
    with self._open_zip() as archive:
        labels: dict[str, str] = {}
        for side in (1, 2):
            for line in self._member(archive, f"rel_ids_{side}").splitlines():
                local_id, raw = line.split("\t", 1)
                labels[local_id] = raw
        return labels

load_temporal_graphs

load_temporal_graphs() -> tuple[
    list[TemporalTriple], list[TemporalTriple]
]

Like load_graphs, but keeps each triple's two timestamp columns instead of dropping them.

Returns:

Type	Description
list[TemporalTriple]	(subject_id, relation_id, object_id, start_label, end_label)
list[TemporalTriple]	tuples per side, start_label/end_label a "YYYY-MM"
tuple[list[TemporalTriple], list[TemporalTriple]]	string or None (unresolvable/missing timestamp) -- feed
tuple[list[TemporalTriple], list[TemporalTriple]]	directly to
tuple[list[TemporalTriple], list[TemporalTriple]]	SimpleHHEALinker.fit's
tuple[list[TemporalTriple], list[TemporalTriple]]	temporal_triples.

Source code in src/linkingtk/datasets/icews.py

def load_temporal_graphs(self) -> tuple[list[TemporalTriple], list[TemporalTriple]]:
    """Like [load_graphs][linkingtk.datasets.base.GraphDatasetLoader.load_graphs],
    but keeps each triple's two timestamp columns instead of dropping them.

    Returns:
        `(subject_id, relation_id, object_id, start_label, end_label)`
        tuples per side, `start_label`/`end_label` a `"YYYY-MM"`
        string or `None` (unresolvable/missing timestamp) -- feed
        directly to
        [SimpleHHEALinker.fit][linkingtk.algorithms.ea.simple_hhea.SimpleHHEALinker.fit]'s
        `temporal_triples`.
    """
    with self._open_zip() as archive:
        time_labels = self._time_labels(archive)
        return (
            self._temporal_triples(archive, 1, time_labels),
            self._temporal_triples(archive, 2, time_labels),
        )

WordNet-Wikidata

wordnet_wikidata

Loaders for WordNet-Wikidata, a heterogeneous taxonomic EA benchmark.

Source: https://github.com/jmccrae/wn-wd-entity-align's data/{subset}/jape/ folders -- a URI-keyed relative of the ent_ids/triples zip format shared by linkingtk.datasets.dbp15k/linkingtk.datasets.openea/ linkingtk.datasets.icews (see linkingtk.datasets.kg_zip's module docstring), fetched per-file like linkingtk.datasets.naisc rather than as a zip, since these files are individually small enough not to need it.

Per that export's own NOTES.md, s_labels/t_labels' third column is a same-language duplicate of the second (a cross-lingual-translation slot DBP15K's format expects that doesn't apply here) -- only the second column is used below.

WordNetWikidataLanguagesDataset

WordNetWikidataLanguagesDataset(
    base_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _WordNetWikidataDataset

~200 language senses/items (smallest subset -- a good default).

Source code in src/linkingtk/datasets/wordnet_wikidata.py

def __init__(self, base_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        base_url: Override for where ``s_labels``/``s_triples``/etc.
            are fetched from (a URL or ``file://`` path). Defaults to
            this subset's directory in the source repository.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.base_url = base_url if base_url is not None else f"{_BASE_URL}/{self._subset}/jape"
    self.cache_dir = cache_dir

WordNetWikidataLocationsDataset

WordNetWikidataLocationsDataset(
    base_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _WordNetWikidataDataset

Place-name senses/items.

Source code in src/linkingtk/datasets/wordnet_wikidata.py

def __init__(self, base_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        base_url: Override for where ``s_labels``/``s_triples``/etc.
            are fetched from (a URL or ``file://`` path). Defaults to
            this subset's directory in the source repository.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.base_url = base_url if base_url is not None else f"{_BASE_URL}/{self._subset}/jape"
    self.cache_dir = cache_dir

WordNetWikidataOrganismsDataset

WordNetWikidataOrganismsDataset(
    base_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _WordNetWikidataDataset

Species/taxon senses/items, keyed on binomial (scientific) names.

Source code in src/linkingtk/datasets/wordnet_wikidata.py

def __init__(self, base_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        base_url: Override for where ``s_labels``/``s_triples``/etc.
            are fetched from (a URL or ``file://`` path). Defaults to
            this subset's directory in the source repository.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.base_url = base_url if base_url is not None else f"{_BASE_URL}/{self._subset}/jape"
    self.cache_dir = cache_dir

WordNetWikidataOrganismsHardDataset

WordNetWikidataOrganismsHardDataset(
    base_url: str | None = None,
    cache_dir: Path | None = None,
)

Bases: _WordNetWikidataDataset

A harder variant of WordNetWikidataOrganismsDataset.

See that source repository's README for what makes it harder.

Source code in src/linkingtk/datasets/wordnet_wikidata.py

def __init__(self, base_url: str | None = None, cache_dir: Path | None = None) -> None:
    """Create the loader.

    Args:
        base_url: Override for where ``s_labels``/``s_triples``/etc.
            are fetched from (a URL or ``file://`` path). Defaults to
            this subset's directory in the source repository.
        cache_dir: Override for the download cache directory. Ignored
            for ``file://`` URLs, which are never cached.
    """
    self.base_url = base_url if base_url is not None else f"{_BASE_URL}/{self._subset}/jape"
    self.cache_dir = cache_dir

AIDA-CoNLL

aida_conll

AIDA-CoNLL Entity Linking dataset loader.

AIDA-CoNLL (Hoffart et al., 2011) links named-entity mentions in Reuters newswire text (the CoNLL 2003 NER corpus) to Wikipedia/Wikidata entities. The original release's licensing is murky (the underlying CoNLL 2003 text has its own Reuters-derived redistribution history); this loader instead uses a community republish already redistributed under a clear license: the Hugging Face Hub dataset cyanic-selkie/aida-conll-yago-wikidata (cc-by-sa-3.0), which maps the original AIDA-YAGO2 annotations onto Wikidata QIDs. Loaded via the datasets library (already a hard dependency of this package -- no new install).

Each document's gold entities carry a Wikipedia page title but no description; this loader fetches one via Wikipedia's MediaWiki API (action=query&prop=extracts, batched and disk-cached through fetch_cached) so KB entities have real label+description text, not just a bare title.

AidaConllDataset

AidaConllDataset(
    source: str = _SOURCE,
    cache_dir: Path | None = None,
    description_fetcher: DescriptionFetcher | None = None,
)

Bases: DatasetLoader

AIDA-CoNLL: named-entity mentions in Reuters newswire, linked to Wikidata.

dataset1 (mentions) carries context=(text, start, end) -- the full document text plus the mention's character span, matching algorithms.el's "mentions carry context, no description" convention. dataset2 (KB entries) carries a label (the Wikipedia page title, humanized) and a description (a fetched Wikipedia lead-paragraph extract). Mentions with no valid gold entity (NIL) are skipped entirely, matching the AIDA-CoNLL literature's own "InKB" evaluation convention.

Parameters:

Name	Type	Description	Default
source	str	A Hugging Face Hub dataset id, or a local path to a directory of train/validation/test parquet files in the same shape -- used by tests to point at a tiny local fixture instead of the real (946+216+231 document) Hub dataset.	_SOURCE
cache_dir	Path | None	Override for the Wikipedia-extract download cache directory. Ignored if description_fetcher is given.	None
description_fetcher	DescriptionFetcher | None	Overrides how KB entity descriptions are sourced, given the list of unique Wikipedia titles this split needs. Defaults to fetch_wikipedia_extracts -- pass a fake here in tests to avoid a real network call.	None

Source code in src/linkingtk/datasets/aida_conll.py

def __init__(
    self,
    source: str = _SOURCE,
    cache_dir: Path | None = None,
    description_fetcher: DescriptionFetcher | None = None,
) -> None:
    self.source = source
    self.cache_dir = cache_dir
    self.description_fetcher = (
        description_fetcher
        if description_fetcher is not None
        else lambda titles: fetch_wikipedia_extracts(titles, self.cache_dir)
    )

load_splits

load_splits() -> tuple[
    list[tuple[str, str]],
    list[tuple[str, str]],
    list[tuple[str, str]],
]

Load this dataset's native train/test/validation ground-truth split.

Unlike load(), this skips description fetching entirely (only mention/entity ids are needed) -- same shape and argument order as _KGZipDataset.load_splits (kg_zip.py), the established precedent for "native supervised split" loaders.

Returns:

Type	Description
tuple[list[tuple[str, str]], list[tuple[str, str]], list[tuple[str, str]]]	(train_pairs, test_pairs, val_pairs).

Source code in src/linkingtk/datasets/aida_conll.py

def load_splits(
    self,
) -> tuple[list[tuple[str, str]], list[tuple[str, str]], list[tuple[str, str]]]:
    """Load this dataset's native train/test/validation ground-truth split.

    Unlike ``load()``, this skips description fetching entirely (only
    mention/entity ids are needed) -- same shape and argument order as
    ``_KGZipDataset.load_splits`` (``kg_zip.py``), the established
    precedent for "native supervised split" loaders.

    Returns:
        ``(train_pairs, test_pairs, val_pairs)``.
    """
    _, _, _, ground_truth_by_split = self._build(_SPLITS, fetch_descriptions=False)
    return (
        ground_truth_by_split["train"],
        ground_truth_by_split["test"],
        ground_truth_by_split["validation"],
    )

fetch_wikipedia_extracts

fetch_wikipedia_extracts(
    titles: list[str], cache_dir: Path | None = None
) -> dict[str, str]

Batch-fetch each title's Wikipedia lead-paragraph extract, cached to disk.

Uses MediaWiki's action=query&prop=extracts API in batches of _EXTRACT_BATCH_SIZE titles per request -- not one request per entity, since a full AIDA-CoNLL split has thousands of unique gold entities. A batch of extracts this size is too large for MediaWiki to answer in one response: it returns as many pages as fit plus a continue token, silently omitting the extract field (not an empty string -- the key is simply absent) for whichever pages didn't fit. Measured concretely on AIDA-CoNLL's real title list: the batch containing "United Kingdom", "Spain", "Jimi Hendrix" and 27 other well-known pages returned real extracts for only 20 of 50 -- silently leaving the rest empty, not an error, before this method followed continue (see issue #45's benchmark investigation). Each request (initial and every continuation) is cached via fetch_cached, keyed by its full URL.

Parameters:

Name	Type	Description	Default
titles	list[str]	Wikipedia page titles (underscored, as stored by the source dataset).	required
cache_dir	Path | None	Forwarded to fetch_cached.	None

Returns:

Type	Description
dict[str, str]	title -> extract for every input title, in the same order.
dict[str, str]	A title with no extract available (e.g. a deleted page) maps to
dict[str, str]	"".

Source code in src/linkingtk/datasets/aida_conll.py

def fetch_wikipedia_extracts(titles: list[str], cache_dir: Path | None = None) -> dict[str, str]:
    """Batch-fetch each title's Wikipedia lead-paragraph extract, cached to disk.

    Uses MediaWiki's ``action=query&prop=extracts`` API in batches of
    ``_EXTRACT_BATCH_SIZE`` titles per request -- not one request per
    entity, since a full AIDA-CoNLL split has thousands of unique gold
    entities. A batch of ``extracts`` this size is too large for MediaWiki
    to answer in one response: it returns as many pages as fit plus a
    ``continue`` token, silently *omitting* the ``extract`` field (not an
    empty string -- the key is simply absent) for whichever pages didn't
    fit. Measured concretely on AIDA-CoNLL's real title list: the batch
    containing "United Kingdom", "Spain", "Jimi Hendrix" and 27 other
    well-known pages returned real extracts for only 20 of 50 -- silently
    leaving the rest empty, not an error, before this method followed
    ``continue`` (see issue #45's benchmark investigation). Each request
    (initial and every continuation) is cached via ``fetch_cached``, keyed
    by its full URL.

    Args:
        titles: Wikipedia page titles (underscored, as stored by the
            source dataset).
        cache_dir: Forwarded to ``fetch_cached``.

    Returns:
        ``title -> extract`` for every input title, in the same order.
        A title with no extract available (e.g. a deleted page) maps to
        ``""``.
    """
    unique_titles = list(dict.fromkeys(titles))
    extracts: dict[str, str] = {}
    for start in range(0, len(unique_titles), _EXTRACT_BATCH_SIZE):
        batch = unique_titles[start : start + _EXTRACT_BATCH_SIZE]
        base_params = {
            "action": "query",
            "format": "json",
            "prop": "extracts",
            "exintro": "1",
            "explaintext": "1",
            "redirects": "1",
            "titles": "|".join(batch),
        }
        continue_params: dict[str, str] = {}
        while True:
            url = f"{_WIKIPEDIA_API}?{urlencode({**base_params, **continue_params})}"
            payload = json.loads(fetch_cached(url, cache_dir, headers={"User-Agent": _USER_AGENT}))
            query = payload.get("query", {})
            pages = query.get("pages", {})
            # MediaWiki resolves input titles through normalization, then
            # redirects, before landing on a final page -- reverse both maps
            # to recover which *input* title each returned page corresponds to.
            source_by_normalized = {
                item["to"]: item["from"] for item in query.get("normalized", [])
            }
            source_by_redirect = {item["to"]: item["from"] for item in query.get("redirects", [])}
            for page in pages.values():
                if "extract" not in page:
                    continue  # this page didn't fit in this response; it'll come in a later one
                resolved_title = page.get("title", "")
                original = source_by_redirect.get(resolved_title, resolved_title)
                original = source_by_normalized.get(original, original)
                extracts[original] = page["extract"]
            if "continue" not in payload:
                break
            continue_params = payload["continue"]
    return {title: extracts.get(title, "") for title in titles}

Zeshel

zeshel

Zeshel (Zero-shot Entity Linking) dataset loader.

Zeshel (Logeswaran et al., "Zero-Shot Entity Linking by Reading Entity Descriptions", 2019, https://github.com/lajanugen/zeshel) links named mentions inside a Wikia (fandom) page's text to other Wikia pages, in domains ("worlds") disjoint between train/validation/test -- entities the model must resolve from description text alone, never seen at train time. This is BLINK's own "Zero-shot EL" benchmark (Wu et al. 2020, Scalable Zero-shot Entity Linking with Dense Entity Retrieval, Table 1): its bi-encoder-only Recall@64 numbers (93.12/91.44/82.06 train/val/ test) are the closest thing BLINK's own paper has to an apples-to-apples bi-encoder target -- unlike TAC-KBP2010 (LDC-licensed, not freely downloadable) or WikilinksNED Unseen-Mentions (no stable public host). See BlinkLinker.

The original release is Google-Drive-hosted (not automatable); this loader instead uses a community republish on the Hugging Face Hub, naist-nlp/zeshel (mirrors the original CC-BY-SA Wikia content) via the datasets library (already a hard dependency -- no new install). Its train/validation/test splits already encode the paper's zero-shot domain-disjoint protocol (each split's subset values are disjoint domains). Split sizes don't exactly match the paper's original mention counts (the original's exact split was never republished in an automatable form), so treat any published-number comparison as in-spirit, not a byte-identical reproduction.

Each mention is a hyperlink inside some entity's own Wikia page text, pointing at a different entity -- so mentions and the KB come from two separate configs of the same release: "dataset" (annotated documents) and "dictionary" (each domain's own entity KB, name + description).

ZeshelDataset

ZeshelDataset(
    mentions_source: str = _SOURCE, kb_source: str = _SOURCE
)

Bases: DatasetLoader

Loads Zeshel's zero-shot EL mentions and per-domain entity KB.

Parameters:

Name	Type	Description	Default
mentions_source	str	Hugging Face Hub id (or local directory, for tests) for the annotated-documents config.	_SOURCE
kb_source	str	Hugging Face Hub id (or local directory, for tests) for the entity-dictionary config.	_SOURCE

Source code in src/linkingtk/datasets/zeshel.py

def __init__(self, mentions_source: str = _SOURCE, kb_source: str = _SOURCE) -> None:
    self.mentions_source = mentions_source
    self.kb_source = kb_source

load_splits

load_splits() -> tuple[
    list[tuple[str, str]],
    list[tuple[str, str]],
    list[tuple[str, str]],
]

Load this dataset's native train/test/validation ground-truth split.

Unlike load(), this skips the (~1.2GB) entity-dictionary download entirely -- only mention/entity ids are needed. Same shape and argument order as AidaConllDataset.load_splits.

Returns:

Type	Description
tuple[list[tuple[str, str]], list[tuple[str, str]], list[tuple[str, str]]]	(train_pairs, test_pairs, val_pairs).

Source code in src/linkingtk/datasets/zeshel.py

def load_splits(
    self,
) -> tuple[list[tuple[str, str]], list[tuple[str, str]], list[tuple[str, str]]]:
    """Load this dataset's native train/test/validation ground-truth split.

    Unlike ``load()``, this skips the (~1.2GB) entity-dictionary
    download entirely -- only mention/entity ids are needed. Same
    shape and argument order as
    [AidaConllDataset.load_splits][linkingtk.datasets.aida_conll.AidaConllDataset.load_splits].

    Returns:
        ``(train_pairs, test_pairs, val_pairs)``.
    """
    _mentions_by_split, ground_truth_by_split, _domains = self._build_mentions(_SPLITS)
    return (
        ground_truth_by_split["train"],
        ground_truth_by_split["test"],
        ground_truth_by_split["validation"],
    )

DaMuEL

damuel

DaMuEL multilingual dataset loader.

DaMuEL (Kubeša & Straka, "DaMuEL: A Large Multilingual Dataset for Entity Linking", 2023, https://arxiv.org/abs/2306.09288) links named-entity mentions inside Wikipedia articles (53 languages) to Wikidata QIDs. Published on LINDAT/CLARIAH-CZ (item uuid 10e1cc03-b24b-4e41-9df9-b0fe4324ccbe, handle 11234/1-5047), a stable, freely downloadable host -- unlike TAC KBP (LDC-licensed, no free redistribution at all -- see zeshel.py's docstring), this needed no substitute host.

Each of DaMuEL's 53 languages is published as one bitstream, damuel_1.0_<lang>.tar (278MB-26.3GB -- English alone is 26.3GB), an uncompressed outer tar containing exactly 500 part-NNNNN.xz members in shuffled order (confirmed by inspection: streaming the first bytes of a real language's tar yields part numbers like 247, then 325, not 0 then 1). Each part decompresses (xz) to JSON Lines, one Wikidata entity per line, e.g.::

{"qid": "Q504261", "lang": "da", "label": "Bräcke Kommune",
 "aliases": ["Bræcke kommun"], "description": "kommune i ...",
 "wiki": {"title": "...", "text": "...",
          "tokens": [{"start": 1, "end": 7, ...}, ...],
          "links": [{"start": 16, "end": 17, "origin": "wiki",
                     "title": "Bräcke", "qid": "Q995545"}, ...]}}

Most lines have no "wiki" key (no Wikipedia article in that language) -- they're still valid KB entries (real negatives/candidates for linking), just never a source of mentions. wiki.tokens[i]'s start/end are character offsets into wiki.text; wiki.links[i]'s start/end are token-index offsets (small integers, e.g. 16/17) that must be translated through tokens to get a character span for Entity.context. Each link's origin distinguishes the one real, hand-authored Wikipedia hyperlink per entity mention ("wiki") from several automatically detected extra mentions of the same entities elsewhere in the article ("title", "label", "alias", "redirects", "anchors", and lemmatized variants -- the paper: "Wikipedia documents deliberately annotate only a single mention for every entity present; we further automatically detect all mentions"). This loader defaults to origin == "wiki" only -- gold, not silver, mentions -- via the mention_origins constructor argument.

Since even one language's full tar can be tens of GB, this loader streams it (tarfile.open(fileobj=..., mode="r|"), no seek) and stops after max_parts members -- since part order is already shuffled, this is a random sample, not a biased prefix. This is a partial read by design, so unlike every other loader in this package it is not routed through fetch_cached (which reads a whole response into memory before caching it -- exactly the multi-GB download this loader exists to avoid); re-running re-streams the same small prefix, cheap by design. There is no native train/dev/test split published for DaMuEL (unlike AIDA-CoNLL/Zeshel) -- only the per-language and a language-agnostic-KB (damuel_1.0_wikidata.tar, Wikidata claims/named-entity-type per QID, out of scope here -- label+description already fits every EL algorithm's Entity shape) bitstreams, so this loader has no load_splits().

DamuelDataset

DamuelDataset(
    language: str = "en",
    max_parts: int = 2,
    mention_origins: tuple[str, ...] = ("wiki",),
    source: str | None = None,
    cache_dir: Path | None = None,
)

Bases: DatasetLoader

DaMuEL: Wikipedia mentions (53 languages) linked to Wikidata QIDs.

See the module docstring for the real archive format and why this loader streams and samples rather than materializing a full language.

Parameters:

Name	Type	Description	Default
language	str	A DaMuEL language code (e.g. "en", "da" -- the two-letter codes the source itself uses).	'en'
max_parts	int	How many of the language's 500 shuffled .xz shards to stream and parse. A mention's target entity can land in any of the 500 parts (independent of which part its source article is in), so ground_truth density scales roughly with max_parts / 500, not with the mention count alone -- confirmed on real data: 2 English parts (~29s) resolved 2571 real mention/target pairs out of a much larger KB (114669 entities), while 2 Danish parts (a far smaller language) resolved only 41. Low-resource languages need a larger max_parts for a non-empty ground_truth (e.g. wo, the smallest language, resolved 0 pairs at max_parts=3). Defaults to 2 -- fast and already useful for a higher-resource language; raise it for denser ground truth or a lower-resource one. The full corpus is impractically large to materialize in memory in one load() call.	2
mention_origins	tuple[str, ...]	Which wiki.links[].origin values count as real mentions. Defaults to ("wiki",) -- DaMuEL's one real, hand-authored hyperlink per entity per article, not its additional automatically-detected mentions (see module docstring). Widen this (e.g. add "anchors") for more mentions at the cost of silver- rather than gold-standard precision.	('wiki',)
source	str | None	Override for the language tar's location -- a local path or http(s):// URL, used by tests to point at a tiny local fixture instead of resolving and streaming the real (278MB-26GB) release.	None
cache_dir	Path | None	Override for the small bitstream-listing-lookup cache directory. Ignored if source is given directly (no lookup needed), and never covers the tar stream itself, which is never cached (see module docstring).	None

Unlike AidaConllDataset and ZeshelDataset, DaMuEL publishes no native train/dev/test split, so there's no load_splits() here.

Source code in src/linkingtk/datasets/damuel.py

def __init__(
    self,
    language: str = "en",
    max_parts: int = 2,
    mention_origins: tuple[str, ...] = ("wiki",),
    source: str | None = None,
    cache_dir: Path | None = None,
) -> None:
    self.language = language
    self.max_parts = max_parts
    self.mention_origins = mention_origins
    self.source = source
    self.cache_dir = cache_dir

LCQuAD 2.0

lcquad2

LCQuAD 2.0 Entity Linking dataset loader.

LCQuAD 2.0 (Dubey et al., 2019, https://figshare.com/projects/LCQuAD_2_0/62270) is a large-scale KGQA benchmark: ~30,000 crowd-sourced natural-language questions, each paired with a SPARQL query over Wikidata (and a DBpedia translation, unused here). Hosted as two public figshare articles (CC BY 4.0, direct HTTPS download, no auth): "training set for lcquad 2.0" (article 8479055, 24180 rows) and "test set for lcquad 2.0" (article 8479052, 6046 rows). LCQuAD 2.0 has no separate validation split.

The release publishes no explicit entity-mention span annotations. Many rows carry an NNQT_question template with curly-/angle-brace- delimited slots, but a slot's alignment to a specific KG id is template-specific and not reliably recoverable by position alone -- verified directly: a slot's left-to-right order in NNQT_question does not track its corresponding id's order inside sparql_wikidata (e.g. an answer-type constraint's slot can appear before its subject entity's, following that row's template pattern rather than SPARQL triple order).

This loader instead derives mentions directly from the SPARQL query: every wd:Qxxx reference inside sparql_wikidata is a gold entity for that question. Its text span is located by a literal (case-insensitive) substring search for the entity's Wikidata label, tried first against the question field then the paraphrased_question fallback. Measured on a live 300-question sample of the real test set: ~81% of (question, entity) references resolve this way (341/421) -- the rest are skipped (some Wikidata labels have drifted or the item has been merged/deleted since the 2019 release; others are genuine paraphrases using different words than the canonical label, e.g. "Ebola virus" vs. the label "Ebola hemorrhagic fever"). This is the same "skip what can't be resolved, document plainly" convention as AidaConllDataset's NIL-mention skip, just applied at label-matching time instead of NIL-annotation time. Property (wdt:/p:/ps:/pq: Pxxx) references are deliberately out of scope -- this loader links entities, not relations, matching every other EL loader in this package.

The release carries only bare QIDs, no entity metadata -- KB entity labels/descriptions are fetched live from Wikidata's wbgetentities API (batched 50 ids/request, its own per-request limit; disk-cached through fetch_cached; retried with backoff on 429/503, since a full run needs on the order of ~450 batches for the real dataset's ~23,000 unique referenced QIDs -- easy to trip Wikidata's anonymous rate limit without it).

Unlike AidaConllDataset or ZeshelDataset, load_splits() here can't skip the network fetch: resolving a mention's span itself requires each candidate entity's Wikidata label, not just its id, so the label lookup is unavoidable even for ground-truth-only access. It's still provided, for interface parity with those loaders, but only skips building the full KB Entity list -- not the label fetch itself.

Lcquad2Dataset

Lcquad2Dataset(
    train_source: str = _TRAIN_URL,
    test_source: str = _TEST_URL,
    cache_dir: Path | None = None,
    label_fetcher: LabelFetcher | None = None,
)

Bases: DatasetLoader

LCQuAD 2.0: crowd-sourced KGQA questions, entities linked to Wikidata.

See the module docstring for how mentions are derived (no native span annotations exist -- spans are recovered by matching each SPARQL entity's Wikidata label against the question text) and why load_splits() can't avoid the label-lookup network cost the way sibling loaders' can.

dataset1 (mentions) carries context=(text, start, end) -- matching algorithms.el's "mentions carry context, no description" convention. dataset2 (KB entries) carries a label + description fetched live from Wikidata.

Parameters:

Name	Type	Description	Default
train_source	str	URL or local path to the training-split JSON file.	_TRAIN_URL
test_source	str	URL or local path to the test-split JSON file.	_TEST_URL
cache_dir	Path | None	Override for the Wikidata-label/JSON-file download cache directory. Ignored if label_fetcher is given.	None
label_fetcher	LabelFetcher | None	Overrides how KB entity labels/descriptions are sourced, given the list of unique QIDs referenced anywhere in the loaded rows. Defaults to fetch_wikidata_labels -- pass a fake here in tests to avoid a real network call.	None

Source code in src/linkingtk/datasets/lcquad2.py

def __init__(
    self,
    train_source: str = _TRAIN_URL,
    test_source: str = _TEST_URL,
    cache_dir: Path | None = None,
    label_fetcher: LabelFetcher | None = None,
) -> None:
    self.train_source = train_source
    self.test_source = test_source
    self.cache_dir = cache_dir
    self.label_fetcher = (
        label_fetcher
        if label_fetcher is not None
        else lambda qids: fetch_wikidata_labels(qids, self.cache_dir)
    )

load_splits

load_splits() -> tuple[
    list[tuple[str, str]],
    list[tuple[str, str]],
    list[tuple[str, str]],
]

Load this dataset's native train/test ground-truth split.

LCQuAD 2.0 has no native validation split -- the third element is always empty, matching the shape of AidaConllDataset.load_splits. Unlike that method, this still performs the full label lookup (see the module docstring) -- it only skips building dataset2's Entity objects.

Returns:

Type	Description
tuple[list[tuple[str, str]], list[tuple[str, str]], list[tuple[str, str]]]	(train_pairs, test_pairs, []).

Source code in src/linkingtk/datasets/lcquad2.py

def load_splits(
    self,
) -> tuple[list[tuple[str, str]], list[tuple[str, str]], list[tuple[str, str]]]:
    """Load this dataset's native train/test ground-truth split.

    LCQuAD 2.0 has no native validation split -- the third element is
    always empty, matching the shape of
    [AidaConllDataset.load_splits][linkingtk.datasets.aida_conll.AidaConllDataset.load_splits].
    Unlike that method, this still performs the full label lookup (see
    the module docstring) -- it only skips building `dataset2`'s
    `Entity` objects.

    Returns:
        ``(train_pairs, test_pairs, [])``.
    """
    _mentions, _kb, _ground_truth, ground_truth_by_split = self._build(build_kb=False)
    return ground_truth_by_split["train"], ground_truth_by_split["test"], []

fetch_wikidata_labels

fetch_wikidata_labels(
    qids: list[str], cache_dir: Path | None = None
) -> dict[str, tuple[str, str]]

Batch-fetch each QID's English label + description, cached to disk.

Uses wbgetentities in batches of _LABEL_BATCH_SIZE (Wikidata's own per-request id limit). Each request is retried with exponential backoff on a 429/503 response -- LCQuAD 2.0's real train+test split references ~23,000 unique QIDs (~450 batches), enough to reliably hit Wikidata's anonymous rate limit without backoff.

Parameters:

Name	Type	Description	Default
qids	list[str]	Wikidata QIDs ("Q...").	required
cache_dir	Path | None	Forwarded to fetch_cached.	None

Returns:

Type	Description
dict[str, tuple[str, str]]	qid -> (label, description). A QID with no English label (a
dict[str, tuple[str, str]]	page merged/deleted since the 2019 release) maps to ("", "").

Source code in src/linkingtk/datasets/lcquad2.py

def fetch_wikidata_labels(
    qids: list[str], cache_dir: Path | None = None
) -> dict[str, tuple[str, str]]:
    """Batch-fetch each QID's English label + description, cached to disk.

    Uses ``wbgetentities`` in batches of ``_LABEL_BATCH_SIZE`` (Wikidata's
    own per-request id limit). Each request is retried with exponential
    backoff on a 429/503 response -- LCQuAD 2.0's real train+test split
    references ~23,000 unique QIDs (~450 batches), enough to reliably hit
    Wikidata's anonymous rate limit without backoff.

    Args:
        qids: Wikidata QIDs (``"Q..."``).
        cache_dir: Forwarded to ``fetch_cached``.

    Returns:
        ``qid -> (label, description)``. A QID with no English label (a
        page merged/deleted since the 2019 release) maps to ``("", "")``.
    """
    unique_qids = list(dict.fromkeys(qids))
    result: dict[str, tuple[str, str]] = {}
    for start in range(0, len(unique_qids), _LABEL_BATCH_SIZE):
        batch = unique_qids[start : start + _LABEL_BATCH_SIZE]
        params = {
            "action": "wbgetentities",
            "format": "json",
            "props": "labels|descriptions",
            "languages": "en",
            "ids": "|".join(batch),
        }
        url = f"{_WIKIDATA_API}?{urlencode(params)}"
        payload = json.loads(_fetch_with_retry(url, cache_dir))
        for qid, entity in payload.get("entities", {}).items():
            label = entity.get("labels", {}).get("en", {}).get("value", "")
            description = entity.get("descriptions", {}).get("en", {}).get("value", "")
            result[qid] = (label, description)
    return {qid: result.get(qid, ("", "")) for qid in qids}

SemCor

semcor

SemCor 2026 Word Sense Disambiguation dataset loader.

SemCor 2026 is a fork and reannotation of the classic Brown-Corpus-based SemCor corpus, kept aligned with the current Open English WordNet release. Unlike the original SemCor release (a Princeton-hosted tarball, WordNet 1.6 sense keys), this fork is distributed directly as per-document files in its GitHub repository -- one Teanga YAML file per Brown Corpus document, storing raw text, token character spans, lemmas, POS tags, and three parallel sense-key layers (wn16_key, wn30_key, oewn_key). This loader uses oewn_key: it's already in wn's own synset-id format (e.g. "oewn-08437235-n"), so no sense-key translation is needed -- unlike UfsacDataset's WordNet 3.0 sense keys, see sensekey_to_synset_id.

Since a sense-tagged corpus has no fixed candidate-sense list to enumerate (every open-class word can in principle be any synset in the dictionary), dataset2 is a WnEntitySource rather than a materialized list[Entity] -- this is the loader-side half of that design; see EntitySource.

Each mention's labels is its lemmas layer entry, not its literal surface text -- candidate generation (ExactMatch querying a WnEntitySource by label) needs the dictionary form (e.g. "catch"), since an inflected surface form (e.g. "caught") won't exact-match WordNet's own lemma index. The surface form is still recoverable from context's span for anything that needs the literal text.

SemCorDataset

SemCorDataset(
    source: str = _SOURCE,
    ref: str = _REF,
    categories: list[str] | None = None,
    cache_dir: Path | None = None,
    lexicon: str = "oewn:2025+",
)

Bases: DatasetLoader

SemCor 2026: Brown Corpus text sense-tagged against Open English WordNet.

dataset1 (mentions) carries context=(sentence_text, start, end) for every sense-tagged content word across every document in the corpus (352 documents, ~190K sense-tagged tokens as of this corpus's 2026 state) -- not a toy subset. dataset2 is a WnEntitySource querying the lexicon this corpus's oewn_key layer is aligned to.

Parameters:

Name	Type	Description	Default
source	str	A "owner/repo" GitHub id, or a local directory containing the same data/<category>/<file>.yaml layout -- used by tests to point at a tiny local fixture instead of the real (46MB, 352-file) GitHub repository.	_SOURCE
ref	str	The git ref to fetch from, when source names a GitHub repo.	_REF
categories	list[str] | None	Restrict to these data/ subdirectory names (e.g. ["press_reportage"]) instead of the full corpus.	None
cache_dir	Path | None	Override for the download cache directory.	None
lexicon	str	The wn lexicon dataset2 queries -- must match the WordNet release this corpus's oewn_key layer is aligned to. Defaults to "oewn:2025+" (the rolling, most-current Open English WordNet release) since the corpus tracks the current release, not the one WnEntitySource itself defaults to standalone -- bump this if a newer dated release ships and the corpus's oewn_key layer is regenerated against it instead of 2025+.	'oewn:2025+'

Source code in src/linkingtk/datasets/semcor.py

def __init__(
    self,
    source: str = _SOURCE,
    ref: str = _REF,
    categories: list[str] | None = None,
    cache_dir: Path | None = None,
    lexicon: str = "oewn:2025+",
) -> None:
    self.source = source
    self.ref = ref
    self.categories = categories
    self.cache_dir = cache_dir
    self.lexicon = lexicon

UFSAC

ufsac

UFSAC sense-annotated corpus loader.

UFSAC (Vial et al., 2018) unifies 16+ Word Sense Disambiguation corpora (SemCor, WordNet Gloss Tagged, MASC, OMSTI, the SensEval/SemEval all-words and lexical-sample tasks, ...) into one shared XML schema, all tagged with WordNet 3.0 sense keys.

Unlike this package's other real-world dataset loaders, UFSAC has no stable, programmatically fetchable per-corpus URL: the whole collection is distributed as a single Google Drive archive bundling every corpus's XML together (see the repository's README), not the kind of per-file HTTPS link fetch_cached (or any other loader in this package) can point at. So unlike e.g. AidaConllDataset, this loader doesn't fetch UFSAC's data itself -- source must be a path or URL to a single corpus's XML (or .xml.xz, UFSAC's own distributed compression) file that you've already extracted from that archive, e.g. UfsacDataset("~/Downloads/ufsac-public-2.1/semcor.xml"). The parser itself handles UFSAC's one shared schema, so this loader works for any of the 16+ corpora, not just SemCor.

UFSAC's sense keys are WordNet 3.0 sense keys (e.g. "group%1:03:00::"), not wn synset ids, so dataset2 (a WnEntitySource) needs each one translated via sensekey_to_synset_id before it can serve as a ground-truth id dataset2.get()/.search() can resolve -- unlike SemCorDataset, whose oewn_key layer is already in wn's own id format.

UfsacDataset

UfsacDataset(
    source: str,
    cache_dir: Path | None = None,
    lexicon: str = "omw-en:1.4",
)

Bases: DatasetLoader

A UFSAC-format sense-annotated corpus (SemCor, OMSTI, SensEval/SemEval, ...).

See the module docstring for why source must be a corpus file you've already downloaded, and why dataset2's ids need sense-key translation unlike SemCorDataset.

Parameters:

Name	Type	Description	Default
source	str	A local path or http(s):///file:// URL to a single UFSAC corpus's .xml or .xml.xz file (e.g. semcor.xml, omsti.xml -- see UFSAC's README for the full list of per-corpus filenames).	required
cache_dir	Path | None	Override for the download cache directory. Ignored for a local source.	None
lexicon	str	The wn lexicon dataset2 queries and sense keys are resolved against. Defaults to "omw-en:1.4" (OMW's English WordNet, built directly from WordNet 3.0 -- the release UFSAC's sense keys are aligned to).	'omw-en:1.4'

Mentions whose sense key doesn't resolve to a synset in lexicon are dropped entirely (from both dataset1 and ground_truth), same as AIDA-CoNLL's NIL-mention convention -- rare in practice, but WordNet 3.0 sense keys and lexicon's own sense inventory can disagree at the margins.

Source code in src/linkingtk/datasets/ufsac.py

def __init__(
    self,
    source: str,
    cache_dir: Path | None = None,
    lexicon: str = "omw-en:1.4",
) -> None:
    self.source = source
    self.cache_dir = cache_dir
    self.lexicon = lexicon

Hugging Face Hub republishing tooling

hub_publish

Tooling to package and publish a linkingtk-maintained dataset copy to the Hugging Face Hub.

For datasets that are hard to load directly from their original source (no fetchable per-file URL, an unreliable host, ...) but whose content is clearly redistributable, this republishes it under a linkingtk-controlled Hugging Face Hub dataset repo so a DatasetLoader can point source at a stable https://huggingface.co/datasets/... URL the way fetch_cached already accepts -- see UfsacDataset and examples/publish_ufsac.py for the one dataset this has actually been applied to.

This is not a license clearance tool. Republishing under a linkingtk-controlled repo is only appropriate for content that's actually redistributable (a clear open license, or -- like AidaConllDataset's and ZeshelDataset's community mirrors -- content someone else has already cleared). Callers are responsible for choosing which files to package; this module doesn't inspect content or licenses. Genuinely non-redistributable data (e.g. LDC-licensed corpora, per TAC KBP's precedent -- see zeshel.py's docstring) should never be passed to publish_dataset_files, not even to a private repo.

HubApi

Bases: Protocol

The subset of huggingface_hub.HfApi's interface this module needs.

A real HfApi() instance already satisfies this directly (its create_repo/upload_file methods accept these same keyword arguments) -- this exists only so tests can inject a fake and never touch the network, the same dependency-injection convention as fetch_wikipedia_extracts's description_fetcher.

PublishedFile dataclass

PublishedFile(path_in_repo: str, url: str)

One file's location within a published Hugging Face Hub dataset repo.

package_files

package_files(
    source_dir: Path,
    filenames: Iterable[str],
    *,
    compress: bool = True,
) -> dict[str, bytes]

Read named files out of source_dir, optionally xz-compressing each.

Parameters:

Name	Type	Description	Default
source_dir	Path	Directory each of filenames is read from directly (no recursion -- callers pass an explicit allowlist, not a directory sweep, so a new file dropped into source_dir never gets silently swept up).	required
filenames	Iterable[str]	The files to package, relative to source_dir.	required
compress	bool	When true (the default), each file's bytes are xz-compressed and the output key gets a .xz suffix -- matching UFSAC's own distributed compression, already handled by UfsacDataset and any other loader that reads a .xz-suffixed source.	True

Returns:

Type	Description
dict[str, bytes]	{output_filename: bytes} for each of filenames, ready to
dict[str, bytes]	pass to publish_dataset_files.

Source code in src/linkingtk/datasets/hub_publish.py

def package_files(
    source_dir: Path, filenames: Iterable[str], *, compress: bool = True
) -> dict[str, bytes]:
    """Read named files out of `source_dir`, optionally xz-compressing each.

    Args:
        source_dir: Directory each of `filenames` is read from directly
            (no recursion -- callers pass an explicit allowlist, not a
            directory sweep, so a new file dropped into `source_dir`
            never gets silently swept up).
        filenames: The files to package, relative to `source_dir`.
        compress: When true (the default), each file's bytes are
            xz-compressed and the output key gets a ``.xz`` suffix --
            matching UFSAC's own distributed compression, already
            handled by [UfsacDataset][linkingtk.datasets.ufsac.UfsacDataset]
            and any other loader that reads a ``.xz``-suffixed `source`.

    Returns:
        ``{output_filename: bytes}`` for each of `filenames`, ready to
        pass to [publish_dataset_files][linkingtk.datasets.hub_publish.publish_dataset_files].
    """
    packaged: dict[str, bytes] = {}
    for name in filenames:
        data = (source_dir / name).read_bytes()
        if compress:
            packaged[f"{name}.xz"] = lzma.compress(data)
        else:
            packaged[name] = data
    return packaged

publish_dataset_files

publish_dataset_files(
    repo_id: str,
    files: dict[str, bytes],
    *,
    token: str | None = None,
    private: bool = False,
    api: HubApi | None = None,
) -> list[PublishedFile]

Create (or reuse) a Hugging Face Hub dataset repo and upload files to it.

This always performs the real create-repo/upload-file calls -- there is no dry_run here. Callers that want a preview without touching the network should compute resolve_url for each file themselves instead of calling this function (see examples/publish_ufsac.py's --publish gate).

Parameters:

Name	Type	Description	Default
repo_id	str	The target Hugging Face Hub dataset repo, e.g. "linkingtk/ufsac".	required
files	dict[str, bytes]	{path_in_repo: content_bytes} -- typically package_files's return value.	required
token	str | None	A Hugging Face Hub write token. Defaults to None, which lets HfApi fall back to its own HF_TOKEN environment variable / cached CLI login.	None
private	bool	Whether to create the repo as private, if it doesn't already exist.	False
api	HubApi | None	Overrides the Hugging Face Hub client. Defaults to a real huggingface_hub.HfApi(), lazily imported (this package's other Hub-backed loaders import datasets the same way -- see e.g. aida_conll.py's _build) -- pass a fake here in tests.	None

Returns:

Type	Description
list[PublishedFile]	One PublishedFile
list[PublishedFile]	per input file, in files' iteration order.

Source code in src/linkingtk/datasets/hub_publish.py

def publish_dataset_files(
    repo_id: str,
    files: dict[str, bytes],
    *,
    token: str | None = None,
    private: bool = False,
    api: HubApi | None = None,
) -> list[PublishedFile]:
    """Create (or reuse) a Hugging Face Hub dataset repo and upload `files` to it.

    This always performs the real create-repo/upload-file calls -- there
    is no `dry_run` here. Callers that want a preview without touching
    the network should compute [resolve_url][linkingtk.datasets.hub_publish.resolve_url]
    for each file themselves instead of calling this function (see
    ``examples/publish_ufsac.py``'s ``--publish`` gate).

    Args:
        repo_id: The target Hugging Face Hub dataset repo, e.g.
            ``"linkingtk/ufsac"``.
        files: ``{path_in_repo: content_bytes}`` -- typically
            [package_files][linkingtk.datasets.hub_publish.package_files]'s
            return value.
        token: A Hugging Face Hub write token. Defaults to `None`, which
            lets `HfApi` fall back to its own ``HF_TOKEN`` environment
            variable / cached CLI login.
        private: Whether to create the repo as private, if it doesn't
            already exist.
        api: Overrides the Hugging Face Hub client. Defaults to a real
            ``huggingface_hub.HfApi()``, lazily imported (this package's
            other Hub-backed loaders import ``datasets`` the same way --
            see e.g. ``aida_conll.py``'s ``_build``) -- pass a fake here
            in tests.

    Returns:
        One [PublishedFile][linkingtk.datasets.hub_publish.PublishedFile]
        per input file, in `files`' iteration order.
    """
    if api is None:
        from huggingface_hub import HfApi

        api = HfApi()

    api.create_repo(repo_id, repo_type="dataset", private=private, token=token, exist_ok=True)

    published = []
    for path_in_repo, content in files.items():
        api.upload_file(
            path_or_fileobj=content,
            path_in_repo=path_in_repo,
            repo_id=repo_id,
            repo_type="dataset",
            token=token,
        )
        published.append(PublishedFile(path_in_repo, resolve_url(repo_id, path_in_repo)))
    return published

resolve_url

resolve_url(repo_id: str, path_in_repo: str) -> str

The stable download URL for a file in a Hugging Face Hub dataset repo.

Same .../resolve/main/... shape fetch_cached already accepts as a DatasetLoader source.

Source code in src/linkingtk/datasets/hub_publish.py

def resolve_url(repo_id: str, path_in_repo: str) -> str:
    """The stable download URL for a file in a Hugging Face Hub dataset repo.

    Same ``.../resolve/main/...`` shape
    [fetch_cached][linkingtk.datasets._util.fetch_cached] already accepts
    as a `DatasetLoader` ``source``.
    """
    return f"https://huggingface.co/datasets/{repo_id}/resolve/main/{path_in_repo}"