High-level API

The top-level PyReprism package exposes a convenience API. Import it as pr and pass lang as a language name ("python"), a file extension (".py"), or a language class:

import PyReprism as pr

pr.remove_comments(source, lang="python")
pr.extract_comments(source, lang="python")
pr.count_comments(source, lang="python")
pr.tokenize(source, lang="python")
pr.normalize(source, lang="python")
pr.stats(source, lang="python")
pr.preprocess(source, lang="java", steps=["comments", "strings", "whitespace"])

Every operation accepts engine="regex" (default), engine="pygments" (requires pip install pyreprism[accurate]), or engine="auto".

The re-exported Token, TokenType, CodeStats and Normalizer are documented on their own pages (Tokens, Metrics, Normalizer).

PyReprism: a framework for source-code preprocessing.

High-level convenience API:

import PyReprism as pr

pr.remove_comments(src, lang="python")
pr.extract_comments(src, lang="python")
pr.count_comments(src, lang="python")
pr.tokenize(src, lang="python")
pr.preprocess(src, lang="java", steps=["comments", "strings", "whitespace"])

lang may be a language name ("python"), a file extension (".py"), or a language class. Use detect_language() to infer it from a filename.

PyReprism.blank_comments(source: str, lang: str | type, replacement: str = ' ', engine: str = 'regex') → str: Remove comment content while preserving line numbers.

PyReprism.count_comments(source: str, lang: str | type, engine: str = 'regex') → int

PyReprism.count_keywords(source: str, lang: str | type, engine: str = 'regex') → int

PyReprism.count_numbers(source: str, lang: str | type, engine: str = 'regex') → int

PyReprism.detect_language(filename: str | None = None, source: str | None = None) → Type | None

Infer a language class from a filename and/or a source string.

Resolution order: filename extension, then (if source is given) a #! shebang line, then a best-effort Pygments content guess when Pygments is installed. Returns None when nothing matches.

PyReprism.extract_comments(source: str, lang: str | type, engine: str = 'regex') → List[str]

PyReprism.extract_identifiers(source: str, lang: str | type, engine: str = 'regex') → List[str]

PyReprism.extract_keywords(source: str, lang: str | type, engine: str = 'regex') → List[str]

PyReprism.extract_numbers(source: str, lang: str | type, engine: str = 'regex') → List[str]

PyReprism.extract_operators(source: str, lang: str | type, engine: str = 'regex') → List[str]

PyReprism.extract_strings(source: str, lang: str | type, engine: str = 'regex') → List[str]

PyReprism.get_language(lang: str | type) → Type

Resolve lang to a registered language class.

Accepts a language class, a registry name ("Python"/"python"), or a file extension (".py"). Raises ValueError if it cannot be resolved.

PyReprism.match_comments(source: str, lang: str | type, engine: str = 'regex') → List[Token]

PyReprism.normalize(source: str, lang: str | type, engine: str = 'regex', **options) → str

Return a canonicalized form of source for ML / clone detection.

See PyReprism.languages.base.BaseLanguage.normalize() for options.

PyReprism.preprocess(source: str, lang: str | type, steps: Sequence[str] = ('comments',), engine: str = 'regex') → str

Apply a sequence of removal steps in order and return the result.

Valid steps: comments, strings, numbers, operators, keywords, whitespace.

PyReprism.remove_comments(source: str, lang: str | type, engine: str = 'regex', isList: bool = False): Remove comments from source for the given language.

PyReprism.remove_keywords(source: str, lang: str | type, engine: str = 'regex') → str

PyReprism.remove_numbers(source: str, lang: str | type, engine: str = 'regex') → str

PyReprism.remove_operators(source: str, lang: str | type, engine: str = 'regex') → str

PyReprism.remove_strings(source: str, lang: str | type, engine: str = 'regex') → str

PyReprism.remove_whitespaces(source: str) → str: Collapse insignificant whitespace (language-independent).

PyReprism.stats(source: str, lang: str | type, engine: str = 'regex') → CodeStats: Return line- and token-level CodeStats for source.

PyReprism.tokenize(source: str, lang: str | type, engine: str = 'regex') → List[Token]: Return the flat list of typed tokens for source.