Engines

PyReprism can tokenize with different backends. The default regex engine is zero-dependency; the optional pygments engine (pip install pyreprism[accurate]) is more accurate — for example it will not treat a # inside a string as a comment. Select one per call with engine=.

Tokenization backends.

regex (default) is zero-dependency and uses each language’s own regexes. pygments is more accurate but needs the optional Pygments dependency. auto uses pygments when it is importable, otherwise falls back to regex.

class PyReprism.engines.Engine

Bases: object

Base class for tokenization backends.

An engine only has to implement tokenize(); every higher-level operation (remove/extract/count/normalize/stats) is derived from the token stream by PyReprism._tokenops.

name = 'base'

tokenize(source: str, language_cls) → List[Token]

class PyReprism.engines.PygmentsEngine

Bases: Engine

Tokenize using Pygments lexers, mapped onto PyReprism’s token model.

name = 'pygments'

tokenize(source: str, language_cls) → List[Token]

class PyReprism.engines.RegexEngine

Bases: Engine

Tokenize using the language class’s regex-based tokenize().

name = 'regex'

tokenize(source: str, language_cls) → List[Token]

PyReprism.engines.get_engine(name: str = 'regex') → Engine

Return an Engine instance for name (regex/pygments/auto).

Backend abstraction: an engine turns source into a typed token stream.

class PyReprism.engines.base.Engine

Bases: object

Base class for tokenization backends.

An engine only has to implement tokenize(); every higher-level operation (remove/extract/count/normalize/stats) is derived from the token stream by PyReprism._tokenops.

name = 'base'

tokenize(source: str, language_cls) → List[Token]