Documentation

Install

The PyPI package is leaklens-ml; the installed command is leaklens. Python 3.10 or newer is required, and the base install needs no account, key, or network access.

python -m pip install --upgrade leaklens-ml
leaklens --version
leaklens doctor

leaklens doctor checks the interpreter, install location, entry point, core dependencies, optional extras and TOML validity without a network call. A missing optional extra is reported but does not fail the check.

Optional extras

Quickstart

Run a dataset audit with a label, compare a held-out split, and scan a codebase. The same audit can write JSON, HTML, SARIF, or a shields.io badge.

# Single-table audit
leaklens train.csv --target label

# Compare train and test, including temporal ordering
leaklens train.csv --test test.csv --target label --time date

# Statically scan Python files and notebooks
leaklens --scan src/ --explain

# Gate CI on the trust score
leaklens train.csv --target label --fail-under 80

The CLI exits 1 when any high-severity finding is present, or when --fail-under is set and the trust score is lower. That single rule works in GitHub Actions, pre-commit, Makefiles, and any shell.

From Python

import pandas as pd
from leaklens.leakage import scan, score

df = pd.read_csv("train.csv")
findings = scan(df, "label")
print(f"trust score: {score(findings)}/100")
for finding in findings:
    print(finding.severity, finding.kind, finding.column, finding.detail)

Scanning source uses leaklens.scanner.scan_source(source, name="snippet.py"). It parses the code; it never imports or executes it.

New to the problem? The walkthrough notebook plants three leaks, shows the inflated score, then fixes them. The interactive demo replays real scanner output for the same kinds of case.

Detections

Severity is about confidence in the observed pattern. High means the code or data satisfies a confirmed leakage rule; medium means the pattern is suspicious and context-dependent; low means a reproducibility or hygiene issue that rarely changes a score on its own.

Dataset checks

Code checks: pipeline order

Code checks: tuning, time and reproducibility

Every kind has a remediation string in leaklens/fixes.py, printed by --explain and shown in the demo. The scanner never executes the code it reads.

CLI reference

Dataset audit

leaklens train.csv --target label \
  --test test.csv --group patient_id --time date \
  --fail-under 80 --html report.html

Code scan

leaklens --scan src/ tests/ --explain
leaklens --scan . --sarif leaklens.sarif

--scan accepts any number of .py, .ipynb files or directories, ignores common environment folders, and reports parse failures instead of guessing. It works on Windows, Linux and macOS.

Other commands

Configuration

leaklens init writes a documented .leaklens.toml. The file is a reference today: the CLI does not read it yet, and explicit flags always win. Treat the generated file as documentation for the options you may want to pass.

[leaklens]
# threshold = 0.95          # --threshold
# explain = false           # --explain
# fail_under = 80           # --fail-under
# sarif = "leaklens.sarif"  # --sarif
# html = "report.html"      # --html
# badge = "badge.json"      # --badge
# scan = ["notebooks"]      # --scan

This is an honest limitation, not a hidden feature. If you need configuration in CI today, put the flags in the workflow or a shell script.

CI and pre-commit

LeakLens is designed to fail a build on a confirmed leak. Add the scan to any workflow; SARIF uploads to GitHub code scanning when you also pass --sarif.

- uses: actions/checkout@v7
- uses: actions/setup-python@v6
  with:
    python-version: "3.12"
- run: pip install leaklens-ml
- run: leaklens --scan src/ --explain
- run: leaklens train.csv --target label --fail-under 80

repos:
  - repo: https://github.com/eyadstein/leaklens
    rev: v0.13.0
    hooks:
      - id: leaklens

The hook scans staged Python and Jupyter files. It is fast because it only parses source; no dataset is read and no model is trained.

Reports and exports

The trust score is max(0, 100 − 35·high − 15·medium − 5·low). It is computed centrally by leaklens.leakage.score so the CLI, API, badge, and HTML report cannot disagree.

Experimental AI intelligence

An optional, offline layer adds evidence-bound hypotheses and a root-cause remediation plan on top of the deterministic detectors. It is off by default, needs no GPU, network, paid service or credentials, and is available in the base install.

leaklens --scan notebooks/ --ai
leaklens train.csv --target label --ai
leaklens --scan src --ai-llm

It combines deterministic semantic rules (pipeline order, future information, target-proxy names, identifier names, shared keys) with a small in-memory CPU model trained on grouped, labelled code patterns. Every hypothesis carries inspectable evidence and an explicit limitation. Code is never executed.

POST /audit and POST /scan accept include_ai=true; the response adds an ai object and leaves the deterministic score and findings unchanged. The live page has an explicit opt-in checkbox and renders AI output in a separate panel.

Measured precision, recall, F1, false-positive rate, false positives, misses and limitations are in AI_BENCHMARK.md, with the design and safety model in AI_INTELLIGENCE.md. The independent real-world check is AI_REALWORLD.md: 48 reviewed units from 12 public repositories, measured at accuracy 0.833, precision 0.767, recall 0.958 and FPR 0.292, with every mistake listed. It is a real measurement, not a production-accuracy claim, and is lower than the authored locked test. The layer is not claimed as an improvement over the deterministic scanner.

API and UI

Install the app extra, then run the FastAPI service or the Streamlit UI. Both are optional; the CLI has no server dependency.

pip install "leaklens-ml[app]"
uvicorn leaklens.api:app --port 8000
streamlit run leaklens/ui.py

Public live-scan API

The static website can connect to a separate, hardened API for real scans in the browser. It intentionally exposes no /predict endpoint and no model deserialization:

pip install "leaklens-ml[api]"
uvicorn leaklens.public_api:app --host 127.0.0.1 --port 8000

GET /health and GET /capabilities describe the deployment. POST /inspect lists columns, POST /audit runs the dataset detectors (optionally with a held-out file plus group and time columns), and POST /scan runs the static Python scanner. Uploads default to 4 MiB, 5,000 rows and 200 columns, are parsed in memory, and are never retained.

Docker users can run docker compose up --build for the API on port 8000 and the UI on 8501.

Benchmarks

The scanner benchmark is a regression guard built from labeled fixtures. It is not a claim of accuracy on your codebase.

Generated results are committed to docs/BENCHMARK.md and docs/DATA_BENCHMARK.md; CI runs all three suites.

Limitations

• Static code analysis. Source is parsed, never executed. Runtime data flow, dynamic dispatch and information from external state can be missed.
• Thresholds. Target leakage is a statistical test. A strong legitimate feature can be flagged; a weak leak can fall below the threshold.
• Bounded duplicate checks. Feature overlap samples up to 50,000 rows and near-duplicates up to 20,000. Matching is exact after normalisation, not fuzzy.
• No auto-fix. Every finding has a remediation hint, but changes must be reviewed and re-validated.
• Not a privacy scanner. LeakLens does not detect or redact personal data.
• LLM explanations are opt-in. Only finding text is sent; dataset rows and full source are not. The default provider is a local Ollama endpoint.
• AI hypotheses are experimental. They are evidence-bound and offline, never change deterministic findings or the trust score, and can be wrong. A clean AI pass does not prove the pipeline is risk-free.
• The website demo is prerecorded. GitHub Pages is static, so the demo replays committed fixture output and never simulates a backend.

Troubleshooting

A format raises ImportError

The message names the extra that provides the reader. Install it, for example pip install "leaklens-ml[parquet]" or pip install "leaklens-ml[excel]".

A file with no extension is read as CSV

Unknown suffixes fall back to CSV by design. Pass a filename with the correct suffix or convert the file first.

The command exits 1 but no high finding is printed

Check whether --fail-under is set: the gate also fails when the trust score is below the floor. Remove the flag or raise the score.

My .leaklens.toml is ignored

This is expected in 0.13.0. The file is documentation-only; pass flags explicitly. The limitation is recorded in the changelog.

Doctor reports a missing optional extra

Optional extras are listed but do not fail the check. Only a missing core dependency, Python below 3.10, or invalid TOML returns exit 1.

I need help or found a bug

Open an issue on GitHub. Include the LeakLens version, Python version, the command, and the finding output with any sensitive data removed.