N-grams & code naturalness

The PyReprism.ngrams module extracts token n-grams and provides an n-gram language model for measuring code naturalness (cross-entropy / perplexity), following Hindle et al., “On the Naturalness of Software”.

n-grams can be taken over token text or over token types (structural, AST-free), which is useful for clone detection and style analysis:

from PyReprism import ngrams

ngrams.ngram_counts(source, "python", n=3).most_common(10)
ngrams.ngrams(source, "python", n=2, types=True)

model = ngrams.train("corpus/", n=3)
seq = ngrams.token_sequence(source, "python")
model.perplexity(seq)      # lower = more predictable / "natural"
model.save("model.json")

Token n-gram analysis and an n-gram language model for code “naturalness”.

Built on tokenize(). Supports value n-grams (over token text) and type n-grams (over token kinds — structural and AST-free). NgramModel estimates a smoothed n-gram distribution over a corpus so a file’s cross-entropy / perplexity can be measured, following Hindle et al., “On the Naturalness of Software”.

class PyReprism.ngrams.NgramModel(n: int = 3, k: float = 1.0, types: bool = False)

Bases: object

A smoothed n-gram language model over token sequences.

Train with fit()/update() (sequences of token strings), then score a sequence’s cross_entropy() or perplexity(). Uses add-k (Laplace) smoothing with an implicit unknown-token slot.

cross_entropy(seq: Sequence[str]) → float: Average bits per token (lower = more predictable / “natural”).

fit(sequences: Iterable[Sequence[str]]) → NgramModel

classmethod from_dict(data: dict) → NgramModel

classmethod load(path: str) → NgramModel

logprob(seq: Sequence[str]) → float: Total log2-probability of seq under the model.

perplexity(seq: Sequence[str]) → float: 2 ** cross_entropy — lower means more natural.

save(path: str) → None

to_dict() → dict

update(seq: Sequence[str]) → NgramModel

property vocab_size: int

PyReprism.ngrams.ngram_counts(source: str, lang, n: int = 3, **kwargs) → Counter: Return a collections.Counter of n-grams in source.

PyReprism.ngrams.ngrams(source: str, lang, n: int = 3, *, types: bool = False, include_comments: bool = False, pad: bool = False) → List[Tuple[str, ...]]: Return the list of token n-grams in source.

PyReprism.ngrams.ngrams_of(seq: Sequence[str], n: int, *, pad: bool = False) → List[Tuple[str, ...]]: Return the n-grams of a token sequence.

PyReprism.ngrams.token_sequence(source: str, lang, *, types: bool = False, include_comments: bool = False) → List[str]

Reduce source to a flat list of token strings.

With types the token kind is emitted instead of its text (structural n-grams). Whitespace is always skipped; comments are skipped unless include_comments.

PyReprism.ngrams.train(paths, *, n: int = 3, types: bool = False, include_comments: bool = False, recursive: bool = True, include=None, exclude=None, lang: str | None = None) → NgramModel: Train an NgramModel over a corpus (files or directories).