Before a skill, plugin, or bundle leaves a private working environment, this runs the check that is easy to skip under deadline pressure: does anything in what's about to ship carry a private leak.
Installing a Claude Code plugin copies the plugin directory onto the installer's machine, and relative-path plugins pull in the whole marketplace repo. A public repo, gist, or downloadable bundle works the same way. Once it's out, there's no "delete it later."
What you get
/sanitize-public: scan a target and walk the manual-review checklist before you publish it.
The scan reads the exact files that will ship (key material, credentials, internal hostnames, LAN IPs, personal home paths, secrets-management references) with a deterministic scanner, then walks a short manual-review checklist for the leaks a regex can't see: unreleased plans, a client named in prose, infra topology described in words, git history baked into a bundle that still has its .git folder.
Out of the box it ships generic patterns only. Your own private identifiers, an internal product name, a client codename, a work email domain, a named infra host, live in a local override file on your machine and are never part of what the plugin ships to anyone who installs it.
What's in the bundle
7 files
.claude-plugin/plugin.json
{
"name": "sanitize-public",
"description": "Pre-publish leak scanner. Scans a skill, plugin, repo, or bundle for secrets, internal hostnames, private paths, and other identifiers before it ships publicly, then walks a manual-review checklist for the contextual leaks a regex can't see.",
"version": "1.1.2",
"author": {
"name": "Ivan Misic",
"url": "https://ivanmisic.net"
},
"homepage": "https://ivanmisic.net/toolshed/plugins/sanitize-public",
"repository": "https://github.com/imisic/claude-marketplace",
"license": "MIT",
"keywords": [
"security",
"secrets",
"privacy",
"publishing",
"leak-detection"
]
}
CHANGELOG.md
# sanitize-public changelog ## 1.1.2 The public guidance now makes the artifact-first gate and the history sweep clearer. ## 1.1.1 The README now says which command you get after installing. ## 1.1.0 The skills in this plugin now carry an `agents/openai.yaml` sidecar, so they show up in Codex's skill picker with a proper name and one-line description instead of a raw folder name. It stays implicitly invocable, so Codex can reach for it when a publish is near. ## Earlier Versions before this predate this file. See the git history at https://github.com/imisic/claude-marketplace.
README.md
# sanitize-public Pre-publish leak scanner. Scans a skill, plugin, repo, or bundle for the lexical shape of a private leak (key material, credentials, internal hostnames, LAN IPs, personal home paths, secrets-management references) before it ships to a public surface: a Claude Code plugin marketplace, a public repo or gist, a downloadable bundle, a shared skill or command. Installing a Claude Code plugin **copies the plugin directory onto the installer's machine**, and relative-path plugins pull in the whole marketplace repo. There is no "delete it later" once something ships. This skill turns the pre-publish check into a deterministic scan plus a short manual-review checklist, instead of something you have to remember to do by hand every time. ## Quickstart ```bash python3 skills/sanitize-public/scripts/leak_scan.py <artifact> --severity medium ``` Point it at the artifact that will actually ship (a plugin directory, a bundle folder, a repo root), not the wider tree you authored it in. Exit code 0 means clean at that floor; exit code 1 means findings to review. `high` findings should never ship; `medium` findings almost never should. ```bash python3 skills/sanitize-public/scripts/leak_scan.py <artifact> --severity high # hard blocks only python3 skills/sanitize-public/scripts/leak_scan.py <artifact> --json # for a hook or CI ``` A clean scan means the lexical layer is clean. It is not the same as "safe to publish": run the manual-review section in `skills/sanitize-public/references/patterns.md` for the contextual leaks a regex cannot see (unreleased plans, a client named in prose, infra topology described in words). ## Local override (your own private identifiers) Out of the box the scanner only knows generic shapes. Your own private identifiers, an internal product name, a client codename, a work email domain, a named infra host, never belong in a script that ships. Add them to a local file instead: - `$LEAK_SCAN_PRIVATE` if set, else `~/.config/leak-scan/private-terms` if it exists. - One rule per line: `<low|medium|high> <regex>` (first whitespace splits severity from the regex). `#`-comments and blank lines are ignored. That file lives only on your machine. It is never read into, or copied by, this plugin. ## Install ``` /plugin marketplace add imisic/claude-marketplace /plugin install sanitize-public@imisic ``` Then the command is `/sanitize-public`. The longer write-up lives on the storefront: [ivanmisic.net/toolshed/plugins/sanitize-public](https://ivanmisic.net/toolshed/plugins/sanitize-public).
skills/sanitize-public/SKILL.md
---
name: sanitize-public
effort: xhigh
description: >-
Pre-publish privacy gate. Runs before anything crosses from a private working environment to a
public one: a plugin marketplace repo, a downloadable bundle, a public GitHub repo or gist, a
shared skill or command. Scans the exact files that will ship for key material, credentials,
internal hostnames, LAN IPs, personal home paths, work-email domains, secrets plumbing and
internal or client project names with a deterministic scanner, then does a human pass for the
contextual leaks a regex cannot see. Fires on packaging, bundling, publishing, exporting or
"shipping" content for other people, and on "is this safe to publish", even if the word
"private" is never said, as long as the destination is public.
---
# sanitize-public
The gate between a private working environment and anything other people can
see. Most setups carry some private infrastructure worth keeping private (a
LAN host, machine names, encrypted secrets, work that cannot be named
publicly). Most of what gets authored is clean, but the cost of one leak
crossing into a marketplace others clone onto their own machines is high and
irreversible. This skill makes the check mechanical and repeatable instead of
a thing you have to remember.
## Why this exists (the marketplace fact that raises the stakes)
When someone installs a Claude Code plugin, the plugin directory is **copied
onto their machine**, and relative-path plugins clone the whole marketplace
repo. Whatever sits in that repo ships. The same is true of a downloadable
content bundle (a tar.gz someone fetches) and any public repo. There is no
"delete it later" once it is on other people's disks and in git history. So
the scan runs on the **artifact that will actually ship**, not the source
tree it came from.
## When this fires (and when it does not)
**Fires on:**
- Building or updating a plugin marketplace repo, or adding a plugin to one
- Packaging a downloadable bundle or exporting content for publish
- Pushing a new public repo or gist, or sharing a skill/command with someone
- Explicit ask: "is this safe to publish," "scan before shipping," "make sure
nothing private leaks," "check this bundle"
**Does not fire on:**
- Work that stays inside a private surface (private repos, local-only files,
a private Claude config repo). Those are allowed to contain private data;
that is the point of them.
- Routine editing that is not about to be published. Do not nag mid-draft.
If the destination is ambiguous, ask one question: "is this going somewhere
the public can reach?" If yes, gate it.
## Rebuild, do not redact (the rule that actually prevents leaks)
**Never copy a private file into a public artifact and then remove the private parts.** Redaction can only remove what you thought to look for, so it fails on the thing you did not think of, every time. The scanner is a backstop for when this rule is broken, not a substitute for it. A scanner finding means the process already failed.
The failure is mechanical rather than careless. You build a substitution map from the identifiers you can name, run it over the copied file, and every identifier outside that map survives untouched. The map covers the people and terms you remembered; the file carries the ones you did not. A bare first name, an internal acronym, a market or product codename, a former employer: all of those read as ordinary words to you and to the scanner's generic layer alike.
**Port in two phases, with a hard break between them.**
1. **Read the private artifact and write a behaviour spec.** For a test, that is the function name plus what it asserts, in words. For a script, the mechanism, keyed to function and phase names. No fixture strings, no names, no subjects, no example data. If the spec contains something you would not publish, it does not belong in the spec.
2. **Close the private source. Write the new file from the spec alone,** inventing every fixture from the destination's own roster and vocabulary.
The assertions still match, because the code under test is identical. The content is authored rather than laundered, so there is nothing private left to find and remove.
**Fixtures and test data are the high-risk surface**, not prose. Nobody reads a test fixture the way they read a paragraph, so real names survive there. Treat any ported fixture as unsanitized until proven otherwise.
**Redaction is acceptable only for a file that is already public**, where you are editing in place rather than deriving a new artifact.
**A remediation commit must not quote what it removed.** Naming the value in the commit message, the changelog or the issue writes it straight back into history, in a place the working-tree scan will never look. Describe the shape and the location ("a first name in a fixture table"), never the value. The same applies to the report you hand someone: say what class of thing was found and where, and quote the string itself only when they need it to make a decision.
## Method
1. **Identify the shipping artifact.** Not the repo you author in, the thing
that leaves. For a marketplace, that is the plugin directory (and the repo
root if plugins use relative paths). For a content bundle, the folder that
gets packaged. Point the scanner at that.
2. **Seed your own private terms first (once per machine).** The scanner
ships knowing only generic shapes (key material, LAN IPs, home paths);
it cannot know your employer domain, your client codenames, or your
internal product names, and a scan blind to those returns a false
"clean" that is worse than no scan. Check for the override file
(`$LEAK_SCAN_PRIVATE`, else `~/.config/leak-scan/private-terms`); if it
is absent, offer to create it now, seeded with the private identifiers
already visible in this session or environment, before running the
scanner. If you proceed without one, say plainly the run only catches
generic patterns. Format is under "Running the scanner" below.
3. **Run the scanner at the gate floor.** `python3 scripts/leak_scan.py
<artifact> --severity medium`. Exit code 1 means stop and review. `high`
findings never ship. `medium` (internal host, LAN IP, home path, a private
email domain) almost never ship; each needs a reason. Re-run until clean.
4. **Do the human pass.** The scanner is blind to contextual leaks. Walk
`references/patterns.md` "manual review" section against the content:
unreleased plans, a client or vertical named in prose, infra topology
described in words, a private opinion about a named person, a screenshot
or attachment with private data, git history baked into a bundle.
5. **Report, do not silently pass.** State what shipped clean, what was
flagged, and what was allow-listed and why. A clean scan is "the lexical
layer is clean," not "safe to publish." Say that distinction out loud.
6. **Sweep git history when the history itself will ship.** The scanner above
only reads the working tree. If the shipping artifact is a git repo whose
`.git` directory goes public too (a marketplace repo pushed as-is, a
tarball that still has `.git` inside), a term can be clean in the current
tree and still sit in an old commit. For every rule at severity medium or
higher, run `git log --all -S"<term>" --oneline | head -5` from the repo
root. Any hit means the term was introduced or removed at some point in
history and is still reachable by anyone who clones or fetches. State that
explicitly, and say what it means: either a history rewrite (`git
filter-repo` or equivalent) or, usually cheaper, a fresh-history repo (new
`git init`, one clean commit, old repo stays private).
## Running the scanner
```bash
python3 scripts/leak_scan.py <path> # scan, medium floor (the gate)
python3 scripts/leak_scan.py <path> --severity high # hard blocks only
python3 scripts/leak_scan.py <path> --json # for a hook or CI
python3 scripts/leak_scan.py <path> --require-private # fail if there is no private layer
```
Exit codes: `0` clean, `1` findings at or above the floor, `2` bad invocation, `3` no private layer under `--require-private`.
Every run reports which layers it used, how many files it read, and which it skipped as binary. **A run with no private layer says so on its own output and sets `"trustworthy": false` in `--json`.** It still exits 0, because a fresh install has no private layer yet. Once you have one, add `--require-private` in CI so a layer that goes missing fails the build instead of quietly narrowing the scan.
The pattern set in `scripts/leak_scan.py` is the source of truth;
`references/patterns.md` mirrors it in prose and carries the manual-review
layer. Out of the box the scanner only knows generic shapes (key material,
credential assignments, internal-host and LAN-IP patterns, home paths,
secrets-plumbing references). Your own private identifiers, an internal
product name, a client codename, a work email domain, a named infra host,
never belong in the shipped script. Add them instead to a local override file
that stays on your machine and is never part of what ships:
- `$LEAK_SCAN_PRIVATE` if set, else `~/.config/leak-scan/private-terms` if it
exists.
- One rule per line: `<low|medium|high> <regex>` (the first whitespace
splits severity from the regex). `#`-comments and blank lines are ignored.
- Present -> the scanner catches your private terms too. Absent -> generic
layer only (this is what installing this skill gives you by default).
## The private layer is generated, not hand-written
The section above says the file exists. This says how to keep it honest. A hand-written `private-terms` holds the two dozen terms you thought of on the day you wrote it. Whatever system already holds your identities (a contact directory, an org chart, a CRM export, a notes vault) holds every name that actually exists. The gap between those two counts is where a leak lives, and it widens every time the real system grows and the hand list does not.
So generate the file rather than typing it. A small script that reads your source of truth, batches the names into a few compiled alternations and emits the `<severity> <regex>` format above is enough. Run it before a publish gate, and make it idempotent so regenerating is never a decision.
Two inputs stay manual, and they are the only two you edit by hand:
- **A vocabulary file** for internal terms no directory holds: employers, org units, internal product and project names, market names, infra hostnames. Add a term the first time you catch yourself typing it into public content.
- **An allow file** for names that must never be flagged: your own public identity, and the fictional roster of each public destination. Every line here is a deliberate blind spot, so keep it short and know why each one is there.
Keep those two wherever your config syncs between machines. Do **not** track the generated output: it expands to every real name you know, and it is one regenerate away on any machine.
**A scan is only as good as the machine's private layer.** The generated file does not travel with the repo, so a fresh machine scans with the generic rules alone and will happily call an artifact clean while it is full of colleagues' names. Regenerate there before trusting a clean result.
## Escape hatch
A line containing `ship-allow` is skipped, for a value that is public on
purpose (a published contact address, a documented public repo URL). Use it
only after looking at the line. An unexamined allow-list defeats the gate.
## Not a secret scanner for code security
This is about **publication**, not vulnerability. It does not audit for
injection, weak crypto, or exposed endpoints in running code. It answers one
question: is there anything in this artifact that should not leave your
private environment.
skills/sanitize-public/agents/openai.yaml
interface: display_name: "Sanitize Public" short_description: "Leak scan before anything ships publicly"
skills/sanitize-public/references/patterns.md
# Leak patterns and the manual-review layer Two halves. The **scanned patterns** are what `scripts/leak_scan.py` catches deterministically; this section mirrors the script in prose so a human can read the intent. The **manual review** is what a regex is blind to and a person has to look for. A clean scan covers only the first half. ## Scanned patterns (the script is the source of truth) The shipped `RULES` are **generic**: they contain no one's private identifiers, so the script is safe to publish (it ships inside this plugin). Your own private identifiers load at runtime from a local override file that never ships (see "Local override" below). ### high (never ships, hard block) - **Key material**: PEM / OpenSSH / PGP private key blocks, age secret keys. - **Provider keys**: AWS access key ids, Google API keys, OpenAI-style `sk-` keys, GitHub `ghp_`/`gho_` tokens, Slack `xox*` tokens. - **Bearer / Authorization** headers carrying a real token value. - **Credential assignments**: `password=`, `secret=`, `api_key=`, `access_token=`, `client_secret=` with a concrete value (placeholders like `changeme`, `<...>`, `$VAR`, `env`, `your-...` are ignored). ### medium (almost never ships, needs a reason each time) - **Internal hosts**: any hostname ending in `.lan`, `.local`, `.internal`, or `.home` (one generic rule covers every internal host on your network; no per-host pattern needed). - **RFC1918 private IPs**: `10.x`, `192.168.x`, `172.16-31.x`. - **Personal home paths**: `/home/<user>/`, `/Users/<user>/`. ### low (context-dependent, manual call) - **Secrets plumbing**: mentions of a secrets manager, key file, or bootstrap script name. Fine in a private runbook, a leak in a public README. ## Local override (your private identifiers, never shipped) Internal product names, project codenames, a work email domain, named infra hosts: these are private to you and must not be baked into a script that ships. They load from a local file instead: - `$LEAK_SCAN_PRIVATE` if set, else `~/.config/leak-scan/private-terms` if it exists. - One rule per line: `<low|medium|high> <regex>` (the first whitespace splits severity from the regex). `#`-comments and blank lines are ignored. - Present -> full coverage locally. Absent (the shipped default) -> generic layer only. When a new identifier enters your private set (a new machine name, a new client, a new internal host), add it to the **local override file**, not to `RULES` in the shipped script. The prose here follows the script's generic layer, never leads it. ## Git history (when the artifact's `.git` ships too) The scanner only reads the current working tree. If the shipping artifact still has its `.git` directory (a repo pushed as-is, a tarball that forgot to strip it), a term can be gone from the tree today and still sit in an old commit or diff. For every medium-or-higher rule above, run `git log --all -S"<term>" --oneline | head -5` from the repo root. Any hit means the term is reachable by anyone who clones the repo, even if `HEAD` is clean. The fix is a history rewrite or, usually cheaper, a fresh-history repo: new `git init`, one clean commit, the old repo stays private. ## Manual review (the scanner cannot see these) Run these by eye against the shipping artifact. Every one has leaked from someone at some point with zero keyword a regex could catch. - **Unreleased or internal plans.** Roadmap detail, an unshipped feature, a dated internal decision. Public copy is work-focused and does not preview private plans. - **Named people or clients.** A colleague, a client, a vertical named in prose. If you have a standing rule about which of your own details never appear in public copy, this is where it applies. No opinion about a named individual. - **Infra topology in words.** Describing which machine talks to which over the LAN leaks the same map as a hostname would, with no string to match. - **Absolute paths and usernames in examples.** Command examples that hardcode a real home path or machine instead of `~` or a placeholder. - **Attachments and media.** A screenshot with a private tab, terminal, or path visible. A PDF/image the scanner skipped by extension. - **Git history baked into a bundle.** A tar of a folder that still contains a `.git` with private commit messages or author emails. Ship the files, not the history. - **Commented-out real values.** A disabled credential line the credential rule might miss if malformed, or a "TODO: remove before publish" that never got removed. - **The over-share in a docstring or comment.** A comment that explains *why* using a private incident, a client's constraint, or an internal name. ## Reporting shape After the gate, say plainly: - what was scanned (the artifact, not the source tree), - scan result at the `medium` floor, - anything allow-listed and the reason, - the manual-review outcome, - and the honest caveat: clean scan = lexical layer clean, not "safe to publish." The human pass is what makes it safe.
skills/sanitize-public/scripts/leak_scan.py
#!/usr/bin/env python3
"""
leak_scan.py - scan a file or directory for private data before it ships to a
public surface (a plugin marketplace repo, a toolshed bundle, a public gist or
GitHub repo). Deterministic source of truth for the leak patterns; the
human-readable view is references/patterns.md. If the two disagree, this wins.
What ships vs what stays local. The RULES below are GENERIC: key material,
credential assignments, internal-host and LAN-IP shapes, personal home paths,
secrets-management plumbing. They contain no one's private identifiers, so this
script is safe to publish (it is itself shipped inside the sanitize-public
plugin). Your OWN private identifiers (internal product names, project
codenames, a work email domain, named infra hosts) load at runtime from a local
override file that never ships:
$LEAK_SCAN_PRIVATE if set, else
~/.config/leak-scan/private-terms if it exists
That file is one rule per line: `<severity> <regex>` (first whitespace splits
severity from the regex; blank lines and #-comments ignored). With it present
you get full coverage locally; without it you get the generic layer only. Point
the scanner at the shipping artifact and it catches your private terms via the
local file; the copy someone else installs carries only the generic rules.
A MISSING local file does not make a scan pass, it makes it narrow. This layer
carries every real name, address and internal term, so without it the generic
layer alone will happily call an artifact clean while it is full of colleagues'
names. That is an observed failure, not a hypothetical. So a run without the
local file says so, loudly, on every line of output that reports a result, and
sets "trustworthy": false in --json. It still exits 0, because a fresh install
has no local file yet and a tool that refuses on first run reads as broken. Pass
--require-private to turn that warning into exit 3, which is what you want once
you have a local file and want CI to notice if it goes missing.
What it can and cannot see. The mechanical tells live here. The high-value
leaks are CONTEXTUAL and a regex cannot see them: unreleased plans, a client's
name, infra topology described in prose, a private opinion about a named
person. Those are in references/patterns.md for a human pass. A clean scan
means the lexical layer is clean, not that the content is safe to publish.
Plain Python, standard library only.
Escape hatch. A line containing ship-allow is skipped, for a value that is
public on purpose (the site's own public contact address, a documented public
repo). Use it sparingly and only after you have looked at the line.
Usage:
python3 leak_scan.py <path> # scan a file or dir
python3 leak_scan.py <path> --severity high # only the hard blocks (gate tier)
python3 leak_scan.py <path> --json # machine-readable (for CI / hooks)
Exit codes: 0 = clean at the requested severity floor, 1 = findings at or above
the floor, 2 = bad invocation, 3 = no local private layer (scan not trustworthy).
Wire it as a gate: default floor is 'medium'.
"""
import argparse
import json
import os
import re
import sys
SEV_RANK = {"low": 0, "medium": 1, "high": 2}
# Each rule: (id, severity, regex, human note). Severity order:
# high = near-certain secret or credential. Never ships. Hard block.
# medium = internal infra / personal identifier shape. Almost never ships; review.
# low = a token that is often private in this context; manual call.
# These are GENERIC by design. Private identifiers live in the local override
# file (see load_local_rules), never here, so this script can ship as-is.
RULES = [
("private-key-block", "high",
r"-----BEGIN (?:RSA |EC |OPENSSH |DSA |PGP )?PRIVATE KEY-----",
"PEM/OpenSSH/PGP private key material"),
("age-secret-key", "high",
r"AGE-SECRET-KEY-1[0-9A-Z]+",
"age private key"),
("aws-access-key", "high",
r"\bAKIA[0-9A-Z]{16}\b",
"AWS access key id"),
("google-api-key", "high",
r"\bAIza[0-9A-Za-z_\-]{35}\b",
"Google API key"),
("openai-key", "high",
r"\bsk-[A-Za-z0-9]{20,}\b",
"OpenAI-style secret key"),
("github-token", "high",
r"\bgh[pousr]_[A-Za-z0-9]{36,}\b",
"GitHub personal access / OAuth token"),
("slack-token", "high",
r"\bxox[baprs]-[A-Za-z0-9-]{10,}\b",
"Slack token"),
("bearer-token", "high",
r"(?i)\b(?:authorization|bearer)\b\s*[:=]?\s*['\"]?[A-Za-z0-9._\-]{20,}",
"Authorization / bearer token with a value"),
# `pass` on its own is ordinary English ("- Pass: document file list", "a
# second pass") and fired far more often on prose than on credentials. A
# high-severity rule that cries wolf is worse than no rule, because it is the
# tier people are told never to ship past. password/passwd only.
("credential-assignment", "high",
r"(?i)\b(pass(?:word|wd)|secret|api[_-]?key|access[_-]?token|"
r"client[_-]?secret|private[_-]?token|db[_-]?pass\w*)\b\s*[:=]\s*"
r"['\"]?(?!(?:null|none|true|false|changeme|<|\$|\{|env|your[_-])\b)[^\s'\"]{6,}",
"credential assigned a concrete value"),
# The trailing (?!\.\w) keeps ordinary filenames out: `settings.local.json`,
# `config.local.yml` and friends are a widespread convention, not hosts, and
# a rule that cries wolf on them teaches you to skim past its real findings.
("lan-host", "medium",
r"\b[a-z0-9][a-z0-9\-]*\.(?:lan|local|internal|home)\b(?!\.\w)",
"internal-only hostname"),
("private-ipv4", "medium",
r"\b(?:10\.\d{1,3}\.\d{1,3}\.\d{1,3}|192\.168\.\d{1,3}\.\d{1,3}|"
r"172\.(?:1[6-9]|2\d|3[01])\.\d{1,3}\.\d{1,3})\b",
"RFC1918 private IP"),
("home-path", "medium",
r"/home/[a-z_][a-z0-9_\-]*/|/Users/[a-z_][a-z0-9_\-]*/",
"absolute personal home path"),
("sops-age-ref", "low",
r"(?i)\b(?:sops|age|keys\.txt|known_hosts|dev-bootstrap)\b",
"secrets-management plumbing reference"),
]
# Populated by load_local_rules() so the report can state which layers actually
# ran. A scan that cannot name its layers is a scan you cannot act on.
LOCAL_LAYER = {"path": None, "rules": 0}
# What the scan actually looked at. A skipped file must be visible in the report:
# silent skips are how a gate reports "clean" on content it never opened.
SCAN_STATS = {"scanned": 0, "binary": 0, "unreadable": 0,
"skipped_ext": 0, "skipped_names": []}
def load_local_rules():
"""Load private, machine-local rules that must never ship inside this
script. Path: $LEAK_SCAN_PRIVATE, else ~/.config/leak-scan/private-terms.
Format: one rule per line, `<severity> <regex>` (first whitespace splits).
Missing file is normal (the shipped default) and yields no extra rules."""
path = os.environ.get("LEAK_SCAN_PRIVATE") or os.path.expanduser(
"~/.config/leak-scan/private-terms")
# Recorded before the existence check: the "no private layer" warning has to
# name the path it looked for, which is exactly the case where it is absent.
LOCAL_LAYER["path"] = path
if not path or not os.path.isfile(path):
return []
extra = []
try:
with open(path, "r", encoding="utf-8") as fh:
for lineno, raw in enumerate(fh, 1):
line = raw.strip()
if not line or line.startswith("#"):
continue
parts = line.split(None, 1)
if len(parts) != 2 or parts[0] not in SEV_RANK:
sys.stderr.write(
f"leak-scan: {path}:{lineno} skipped (want "
f"'<low|medium|high> <regex>')\n")
continue
sev, rx = parts
try:
extra.append(("private-local", sev, re.compile(rx),
"private identifier (local rule)"))
except re.error as exc:
sys.stderr.write(
f"leak-scan: {path}:{lineno} bad regex: {exc}\n")
except OSError as exc:
sys.stderr.write(f"leak-scan: cannot read {path}: {exc}\n")
return extra
COMPILED = [(rid, sev, re.compile(rx), note) for rid, sev, rx, note in RULES]
_LOCAL = load_local_rules()
LOCAL_LAYER["rules"] = len(_LOCAL)
COMPILED.extend(_LOCAL)
# Never descend into these; they are not the shipped artifact.
SKIP_DIRS = {".git", "node_modules", "vendor", ".claude", "storage",
"__pycache__", ".idea", ".vscode"}
# Binary / non-text extensions we do not read.
SKIP_EXT = {".png", ".jpg", ".jpeg", ".gif", ".webp", ".ico", ".pdf", ".zip",
".gz", ".tar", ".tgz", ".woff", ".woff2", ".ttf", ".mp4", ".mp3",
".wav", ".phar", ".lock"}
def iter_files(path):
if os.path.isfile(path):
yield path
return
for root, dirs, files in os.walk(path):
dirs[:] = [d for d in dirs if d not in SKIP_DIRS]
for name in files:
if os.path.splitext(name)[1].lower() in SKIP_EXT:
# Counted, not silent: these are exactly the files that need a
# human to open them (a screenshot with a real terminal in frame
# leaks more than any string, and no regex will ever see it).
SCAN_STATS["skipped_ext"] += 1
SCAN_STATS["skipped_names"].append(os.path.join(root, name))
continue
yield os.path.join(root, name)
def scan_file(fpath, floor):
"""Scan one file. Decodes leniently on purpose.
This used to open with errors="strict" and swallow UnicodeDecodeError, so a
single stray byte anywhere in a file made the WHOLE file scan as clean with
no message. That is the worst possible failure for a gate, and it was
observed: a 4MB git-history extract containing two embedded PNG blobs
reported zero findings while holding a real name in its text. Binary is now
decided by a NUL byte in the first block (the standard heuristic), counted,
and reported; everything else is decoded with errors="replace" and scanned.
"""
findings = []
try:
with open(fpath, "rb") as fh:
raw = fh.read()
except OSError:
SCAN_STATS["unreadable"] += 1
return findings
if b"\x00" in raw[:8192]:
SCAN_STATS["binary"] += 1
return findings
SCAN_STATS["scanned"] += 1
lines = raw.decode("utf-8", errors="replace").splitlines()
for lineno, line in enumerate(lines, 1):
if "ship-allow" in line:
continue
for rid, sev, rx, note in COMPILED:
if SEV_RANK[sev] < SEV_RANK[floor]:
continue
m = rx.search(line)
if m:
findings.append({
"file": fpath, "line": lineno, "rule": rid,
"severity": sev, "note": note,
"match": m.group(0)[:80],
})
return findings
def main():
ap = argparse.ArgumentParser(description="Pre-publish leak scan.")
ap.add_argument("path")
ap.add_argument("--severity", choices=("low", "medium", "high"),
default="medium", help="minimum severity to report (gate floor)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--require-private", action="store_true",
help="exit 3 when there is no local private layer (for CI)")
args = ap.parse_args()
if not os.path.exists(args.path):
sys.stderr.write(f"path not found: {args.path}\n")
sys.exit(2)
# The local layer carries every real name, address and internal term. Without
# it a "clean" result only means no key material and no LAN hostnames, which
# is not what anyone reads it as. Say so up front rather than mislead.
blind = LOCAL_LAYER["rules"] == 0
if blind and not args.json:
sys.stderr.write(
"leak-scan: NO LOCAL PRIVATE LAYER. Generic rules only, so this scan\n"
"cannot see names, employers, internal products or work addresses.\n"
" add one: " + LOCAL_LAYER["path"] + "\n"
" format: one rule per line, `<low|medium|high> <regex>`\n"
" in CI: --require-private to make this an error instead\n")
all_findings = []
for f in iter_files(args.path):
all_findings.extend(scan_file(f, args.severity))
all_findings.sort(key=lambda x: (-SEV_RANK[x["severity"]], x["file"], x["line"]))
# Always state the layers. "Clean" is only meaningful next to what was checked.
layers = (f"generic({len(COMPILED) - LOCAL_LAYER['rules']})"
f" + private({LOCAL_LAYER['rules']} from {LOCAL_LAYER['path']})"
if LOCAL_LAYER["rules"] else
f"generic({len(COMPILED)}) ONLY - no private layer")
if args.json:
print(json.dumps({"findings": all_findings, "count": len(all_findings),
"layers": {"generic": len(COMPILED) - LOCAL_LAYER["rules"],
"private": LOCAL_LAYER["rules"],
"private_source": LOCAL_LAYER["path"]},
"read": dict(SCAN_STATS),
"trustworthy": not blind}, indent=2))
else:
print(f"leak-scan: layers = {layers}")
skipped = SCAN_STATS["binary"] + SCAN_STATS["skipped_ext"]
print(f"leak-scan: read {SCAN_STATS['scanned']} text file(s); "
f"skipped {skipped} binary/media, "
f"{SCAN_STATS['unreadable']} unreadable.")
if skipped:
print("leak-scan: OPEN THESE YOURSELF, a regex cannot read a "
"screenshot:")
for name in SCAN_STATS["skipped_names"][:20]:
print(f" {name}")
if len(SCAN_STATS["skipped_names"]) > 20:
print(f" ... and {len(SCAN_STATS['skipped_names']) - 20} more")
if blind:
print("leak-scan: WARNING, generic layer only. This scan cannot see "
"names, employers, internal products or work addresses.")
if not all_findings:
print(f"leak-scan: no findings at severity>={args.severity} ({args.path})")
print("leak-scan: this is the LEXICAL layer only. Contextual leaks "
"(unreleased plans, a named client, infra described in prose) "
"still need the human pass in references/patterns.md.")
else:
for f in all_findings:
print(f"[{f['severity']:>6}] {f['file']}:{f['line']} "
f"{f['rule']} -- {f['note']}\n {f['match']!r}")
print(f"\nleak-scan: {len(all_findings)} finding(s) "
f"at severity>={args.severity}. Review before shipping.")
if all_findings:
sys.exit(1)
# Only the caller who asked for strictness gets it. Everyone else gets a
# clean exit with the narrowness stated in the output and in `trustworthy`.
if blind and args.require_private:
sys.exit(3)
sys.exit(0)
if __name__ == "__main__":
main()
Everything mine on this shelf comes from my own working setup, shared for learning. Test it and adapt it to your project before relying on it; you run it at your own risk.