LeakLens

Data leakage audit · Python CLI

Your model isn't smart.
It's leaking.

A leaked column can make any model look perfect. LeakLens reads your code and your data, finds the leaks that inflate your metrics, and tells you what to fix — before the numbers reach production.

Open the interactive demo Live scan API (backend required)
  • v0.13.0 on PyPI
  • Python 3.10–3.12, 3.14
  • MIT licensed
  • Offline by default

Recorded audit example

Recorded audit leaky_transactions.csv

Leaky table

1 high · 1 medium · 0 low

  • hightarget_leakagechargeback_flag
  • mediumid_liketxn_id

Prerecorded output: leaky_transactions.csv, target is_fraud, generated by leaklens.leakage.scan and score().

109labeled scanner regression cases (synthetic)
32advanced tuning & encoding cases (synthetic)
12synthetic dataset-detector cases
0network calls in a default audit

The problem

Three ways a pipeline lies to you

Leakage is not a bug in the model. It is information arriving from the future, from the test set, or from the label itself. The score goes up and the model gets worse.

high

The answer is a feature

A chargeback flag copied from the label predicts fraud perfectly. The model learns the copy, not the behaviour.

target_leakage
chargeback_flag
roc_auc = 1.000
leaklens.leakage.single_feature_leaks
high

The test set was seen

Duplicated transactions land on both sides of the split. The held-out score measures memorisation, not generalisation.

split_overlap
20 test rows (10.0%)
also appear in train
leaklens.leakage.split_overlap
high

The order is wrong

Scaling, SMOTE or feature selection runs before the split. Every learned statistic has already looked at the held-out rows.

preprocess_before_split
StandardScaler.fit_transform()
feeds the split
leaklens.scanner.scan_source

Real rows, not an illustration

See the overlap

This field plots rows from the checked-in split fixture. Amber points are test rows, blue points are train rows, and red points exist on both sides.

20 duplicated test rows highlighted
Open the split demo

Drag to rotate, or focus the canvas and use the arrow keys. Coordinates are log amount, hour and distance — the same features the exact-overlap check hashes. This is a 2-D projection of a 3-D scene, drawn locally with canvas; no WebGL or external library.

drag to rotate · arrow keys when focused

Coverage

What LeakLens looks for

More than thirty detector kinds across code and data. Each finding names the exact rule, the line or column, and a concrete fix. Severity reflects how confident the check is — not how scary the label sounds.

14 shown

target_leakage

A single feature alone predicts the label above the threshold (cross-validated ROC AUC, R² or balanced accuracy).

highdataset

categorical_target_leakage

A high-cardinality category is nearly pure with respect to the target, leaking the label through conditional entropy.

highdataset

id_like

A column is almost entirely unique values. It looks like an identifier and usually memorises rows instead of generalising.

mediumdataset

split_overlap

Exact row hashes appear in both train and test. The held-out score is partly a memory test.

highdataset

feature_overlap

Feature vectors match across the split while the label is ignored. Conflicting labels are reported as high, otherwise medium.

high / mediumdataset

near_duplicate_overlap

Rows match after lower-casing text, stripping whitespace and rounding numbers to four decimals. No fuzzy matching is claimed.

mediumdataset

preprocess_before_split

A scaler, encoder, imputer or transformer is fitted before train_test_split, so held-out rows shape the transform.

highcode

target_encode_before_split

Target means or aggregates are mapped into a feature before the split. The encoding has already seen the held-out labels.

highcode

feature_selection_before_split

SelectKBest, RFE or SelectFromModel chooses features using the full label before the split.

highcode

tuning_on_test_data

A search or manual loop fits on the test partition. The test set stops being held out.

highcode

cv_on_test_data

Cross-validation runs over the test partition, reusing it for training and validation.

mediumcode

temporal_split_shuffle

train_test_split shuffles a frame sorted by a time column, so future rows can enter training.

high / mediumcode

no_random_state

A split or sampler omits random_state, so every run produces different partitions and results are not reproducible.

lowcode

notebook_out_of_order_execution

Notebook cells have non-monotonic execution counts, so the saved output may not match the visible source order.

lownotebook

Also covered: group and entity overlap (--group), near-duplicate images across folders (leaklens images), notebook cell ordering, fit-on-eval-data, preprocessing outside CV, frequency encoding, fill/one-hot/temporal leakage, and repeated test-set reuse. The full list with severities lives in the documentation.

Interactive demo

Every finding, with its evidence

The demo replays real output from the Python scanners: code snippets with the offending line highlighted, dataset previews with the trust-score arithmetic, and split checks with the duplicated rows called out.

  • Filter by severity and open a finding for the exact evidence and remediation.
  • Switch between leaky and corrected fixtures to see the gate change.
  • Copy the exact CLI command that produced each result.

The browser does not run Python. The page is labelled prerecorded wherever results appear; run the CLI for a live audit.

Explore the demo Read the quickstart
leaklens --scan examples/advanced_leaks.py --explain Exit 1
highfeature_selection_before_splitline 18

SelectKBest.fit_transform() runs before the split at advanced_leaks.py:26

Fix: fit feature selectors inside a Pipeline on the training split only.
hightarget_encode_before_splitline 16

df['city'].map(city_target_mean) turns a target-derived statistic into a feature before the split.

Fix: compute the encoding from the training fold only.
mediumtemporal_split_shuffleline 26

train_test_split() shuffles data that was sorted by a time column.

Fix: split chronologically with shuffle=False.

Regression, not marketing

Measured on labeled fixtures

Every case is deterministic and checked into the repository. The benchmark exists to catch regressions; it is not an accuracy claim for arbitrary real-world code.

synthetic fixtures These counts are authored regression cases, not a population accuracy estimate. A perfect score here proves the rules still fire on the cases we wrote.

109 labeled scanner cases 58 leaky / 51 clean · precision 1.00 · recall 1.00 · authored fixtures only
32 advanced tuning and encoding cases Reported separately so historical counts stay stable; authored fixtures only.
12 synthetic dataset-detector cases 9 leaky / 3 clean · precision 1.00 · recall 1.00 · median case 3 ms

Real-world evidence is thinner. The evaluation in docs/REALWORLD.md has only 8 labels: 5 confirmed leaks and 3 legitimate notebooks, drawn from 60 public notebooks. The other 52 are unlabelled and excluded, so treat its precision and recall as a smoke check, not a population estimate.

Run them yourself: python -m leaklens.bench, python -m leaklens.bench_advanced, python -m leaklens.bench_data. Generated reports live in docs/BENCHMARK.md and docs/DATA_BENCHMARK.md.

Get started

Three commands to a trust score

The base install handles CSV, JSON, dataset audits, code scans and every report format. Optional extras add Parquet, Excel, images, the model arena, and the API/UI.

Install

Python 3.10 or newer. No account, API key or network call required for a standard audit.

python -m pip install --upgrade leaklens-ml

Check the install

doctor verifies Python, the entry point, core dependencies and optional extras without reading secrets.

leaklens doctor

Audit something

Scan a dataset with its label, compare a split, or scan a whole Python project for pipeline leaks.

leaklens --scan . --explain

Dataset audit

Find target leakage, identifiers and overlap, then write a shareable report or a CI gate.

leaklens train.csv --target label --test test.csv
leaklens train.csv --target label --fail-under 80
leaklens train.csv --target label --html report.html

Code scan

Parse Python and notebooks for ordering, encoding, tuning and group-CV mistakes — no execution, no data upload.

leaklens --scan src/ --explain
leaklens --scan . --sarif leaklens.sarif
leaklens --scan notebooks/ --json

Honest limits

What LeakLens does not do

A clean audit is evidence, not proof. These boundaries are deliberate and documented so you can judge a result instead of trusting a badge.

It does not execute your code

The code scanner parses source and notebooks. Data-dependent leaks that only appear at runtime, through dynamic dispatch or external state, can be missed.

Statistical checks use thresholds

Target leakage is a cross-validated score above a threshold (default 0.95). A strong but legitimate feature can be flagged for review; a weak leak can fall below it.

Duplicate checks are bounded and exact

Hashing is bounded to 50k rows for feature vectors and 20k for near-duplicates. Near-duplicate matching only normalises case, whitespace and four decimal places; it is not fuzzy matching.

It is not a privacy or PII scanner

LeakLens looks for label and split leakage in ML workflows. It does not classify personal data, redact columns, or replace a security review.

No automatic fixes

Every finding ships with a remediation hint, but the change is yours. Moving a transform into a pipeline can alter results and must be re-validated.

The browser demo is prerecorded

GitHub Pages serves static files, so the demo replays fixtures produced by the real scanners at build time. It never pretends to execute Python, and no backend is required.