Fingerprinting & similarity

The PyReprism.fingerprints module detects clones and plagiarism using winnowing k-gram fingerprints (Schleimer, Wilkerson & Aiken) over the normalized token stream, so matches are robust to variable renaming and literal changes (Type-2 clones):

from PyReprism import fingerprints as fp

fp.similarity(code_a, code_b, "python")     # Jaccard of fingerprints, 0-1
fp.containment(code_a, code_b, "python")    # fraction of A found in B

index = fp.FingerprintIndex()
index.add_paths("submissions/")
index.similar_pairs(threshold=0.7)

Fingerprints are deterministic (CRC32-hashed) and comparable across runs.

Code fingerprinting and similarity for clone / plagiarism detection.

Uses winnowing (Schleimer, Wilkerson & Aiken) over k-grams of a normalized token stream, so matches are robust to renaming and literal changes (Type-2 clones). Hashes are deterministic (CRC32) so fingerprints are stable and persistable across runs.

class PyReprism.fingerprints.FingerprintIndex(*, k: int = 5, w: int = 4, normalize: bool = True, types: bool = False)

Bases: object

An inverted index of fingerprints for many-to-many clone detection.

add(name: str, source: str, lang) → Set[int]

add_paths(paths, *, recursive: bool = True, include=None, exclude=None) → None: Fingerprint every supported file under paths.

matches(name: str, threshold: float = 0.0) → List[Tuple[str, float]]: Return (other, score) for entries similar to name, best first.

similar_pairs(threshold: float = 0.6) → List[Tuple[str, str, float]]: Return unique (a, b, score) pairs at or above threshold.

PyReprism.fingerprints.containment(a: str, b: str, lang, **options) → float: Fraction of a’s fingerprints also present in b (asymmetric).

PyReprism.fingerprints.fingerprint(source: str, lang, *, k: int = 5, w: int = 4, normalize: bool = True, types: bool = False) → Set[int]

Return the winnowing fingerprint (set of hashes) of source.

k is the k-gram (shingle) size, w the winnowing window.

PyReprism.fingerprints.jaccard(a: Set[int], b: Set[int]) → float: Jaccard similarity of two fingerprint sets, in [0, 1].

PyReprism.fingerprints.normalized_tokens(source: str, lang, *, normalize: bool = True, types: bool = False) → List[str]

Reduce source to a token sequence for fingerprinting.

With types the token kind is emitted (fully structural). With normalize (default) identifiers/numbers/strings collapse to ID/ NUM/STR while keywords, operators and punctuation stay literal — so renaming variables or changing literals does not change the fingerprint.

PyReprism.fingerprints.similarity(a: str, b: str, lang, **options) → float: Similarity (Jaccard of fingerprints) between two sources, in [0, 1].