Skip to content

Teanga utils

TokenizationMismatch

Bases: Exception

Exception raised for tokenization mismatches.

Source code in teanga/utils.py 
63
64
65
66
67
68
69
70
71
72
class TokenizationMismatch(Exception):
    """Exception raised for tokenization mismatches.

    Attributes:
        message -- explanation of the error
    """

    def __init__(self, tokens, text):
        self.message = f"Tokens do not match text: {tokens} != {text}"
        super().__init__(self.message)

find_spans(tokens, text)

Find the spans of tokens in the text.

Parameters:

Name Type Description Default
tokens

list A list of tokens to find in the text.

 required
text

str The text to search for the tokens.

 required

Raises:

Type Description
ValueError

If the tokens do not match the text.
Source code in teanga/utils.py 
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
def find_spans(tokens, text):
    """Find the spans of tokens in the text.

    Parameters:
        tokens: list
            A list of tokens to find in the text.
        text: str
            The text to search for the tokens.

    Exceptions:
        ValueError: If the tokens do not match the text.
    """
    i = 0
    tk_idx = 0
    spans = []
    while i < len(text) and tk_idx < len(text):
        if text[i:].startswith(tokens[tk_idx]):
            spans.append([i, i+len(tokens[tk_idx])])
            i += len(tokens[tk_idx])
            tk_idx += 1
        else:
            i += 1
    if tk_idx < len(tokens):
        raise TokenizationMismatch(tokens, text)
    return spans

teanga_id_for_doc(ids, *args, **kwargs)

Return the Teanga ID for a document.

Parameters:

Name Type Description Default
ids

str The IDs already generated and not to be repeated

 required

Examples:

>>> teanga_id_for_doc(set(), text="This is a document.")
'Kjco'
>>> teanga_id_for_doc(set(), en="This is a document.", nl="Dit is een document.")
'Nnrd'
Source code in teanga/utils.py 
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
def teanga_id_for_doc(ids, *args, **kwargs):
    """Return the Teanga ID for a document.

    Parameters:
        ids: str
            The IDs already generated and not to be repeated

        This works as the add_doc method, but returns the Teanga ID for the document.
        It is not necessary to call this method directly but instead you can use it
        via the Corpus class.

    Examples:
        >>> teanga_id_for_doc(set(), text="This is a document.")
        'Kjco'
        >>> teanga_id_for_doc(set(), en="This is a document.", nl="Dit is een document.")
        'Nnrd'
    """
    text = ""
    if len(kwargs) == 0:
        raise Exception("No arguments given.")
    for key in sorted(kwargs.keys()):
        text += key
        text += "\x00"
        text += kwargs[key]
        text += "\x00"
    code = b64encode(sha256(text.encode("utf-8")).digest()).decode("utf-8")
    n = 4
    while code[:n] in ids and n < len(code):
        n += 1
    return code[:n]