Tokens

tokenize() returns a flat, lossless list of Token objects; concatenating every token’s value reproduces the original source exactly.

Token model shared by the tokenizer and the match/extract APIs.

class PyReprism.tokens.Token(type: TokenType, value: str, start: int, end: int, line: int)

Bases: object

A single lexical token located within a source string.

Parameters:

type – the TokenType of the token.
value – the exact substring the token spans.
start – 0-based start offset in the source.
end – 0-based end offset (exclusive) in the source.
line – 1-based line number of start.

end: int

line: int

start: int

type: TokenType

value: str

class PyReprism.tokens.TokenType(value)

Bases: str, Enum

The kind of source-code construct a Token represents.

COMMENT = 'comment'

IDENTIFIER = 'identifier'

KEYWORD = 'keyword'

NUMBER = 'number'

OPERATOR = 'operator'

OTHER = 'other'

STRING = 'string'

WHITESPACE = 'whitespace'