diff options
| author | Mohamed Hegazy <mhegazy@anthropic.com> | 2026-05-30 11:29:07 -0700 |
|---|---|---|
| committer | Mohamed Hegazy <mhegazy@anthropic.com> | 2026-05-30 11:29:07 -0700 |
| commit | 1ecf3d1bac6ca2be1fbedef9500ca1457da8d42d (patch) | |
| tree | c68441cfe3ca7221df02c9373dd35835e097e16c /plugins/security-guidance/hooks/gitutil.py | |
| parent | c40770ae5a1d6855fce096c5b747e709ec946b2b (diff) | |
security-guidance: purge text=True from subprocess.run + bake PYTHONUTF8=1 (#2099)fix-2099-text-true-pythonutf8
URGENT WINDOWS FIX. Sibling of #2056 / PR #2075 but covering 14 more
sites that PR #2075 missed.
The bug class: on Windows with cp1252 default encoding (typical
en-US locale), `subprocess.run(..., text=True)` decodes child stdout
AND stderr via `locale.getpreferredencoding()`. When git emits a
UTF-8 byte that's undefined in cp1252 (e.g. `0x81` from ف, present
in any path/filename/branch ref/commit message containing
Arabic/Hebrew/CJK), Python's internal `_readerthread` raises
UnicodeDecodeError. The thread crash is silent in Python 3.13+ (only
printed to stderr), but `subprocess.run` returns `stdout=None` and
the caller AttributeErrors on `.strip()`. The user sees a misleading
"WinError 267" or similar catch-all message instead of the real
decode failure.
PR #2075 fixed 6 specific helpers in `diffstate.py` / `gitutil.py`.
This commit covers the 14 survivors. Plus a defense-in-depth belt:
`PYTHONUTF8=1` exported by sg-python.sh.
This commit:
1. sg-python.sh: `export PYTHONUTF8=1` (PEP 540). No-op on
macOS/Linux (already UTF-8). On Windows, makes Python's
`locale.getpreferredencoding()` return UTF-8 instead of cp1252 —
so even if a future regression slips in text=True, the decode
succeeds. Must be set BEFORE Python starts; changing it from
inside the interpreter has no effect.
2. gitutil.py: convert 8 subprocess.run sites from
`capture_output=True, text=True` to `capture_output=True` +
manual `r.stdout.decode("utf-8", errors="replace")`:
- _git_rev_parse_head (stdout = SHA, stderr risk)
- _find_git_index (stdout = PATH, primary bug site)
- _temp_index git add (returncode only, stderr risk)
- _git_toplevel (stdout = PATH, primary bug site)
- _git_dir (stdout = PATH, primary bug site)
- _git_rev_list_range (stdout = SHAs, stderr risk)
- _detect_main_branch (stdout = ref, stderr risk)
- merge-base --is-ancestor (returncode only, stderr risk)
3. security_reminder_hook.py: convert 6 subprocess.run sites
(rev-parse @{u}/@{u}@{1}/local_ref, merge-base, HEAD lookup,
reflog SHA resolution) — same pattern.
4. security_reminder_hook.py: fix the misleading log line in
handle_user_prompt_submit. Was:
debug_log("Failed to capture git baseline (not a git repo?)")
Now includes the cwd in the message so the next reporter doesn't
waste an hour grepping for the real WinError, per reporter's
secondary finding.
Verified locally on macOS Python 3.13:
- py_compile clean on all modified files.
- bash -n sg-python.sh clean.
- sg-python.sh actually propagates PYTHONUTF8=1 to child Python
(verified via probe — sys.flags.utf8_mode=1).
- Existing 353 tests still pass — 0 regression.
- 25 new tests in test_2099_subprocess_text_true.py:
* 10 static-shape catchers (one per hooks/*.py file). Any
future PR that reintroduces text=True OR encoding= in
subprocess.run fails this check at PR time. Single source
of truth for the regression class.
* 2 sg-python.sh verifiers (literal export + actual
propagation to child Python).
* 5 macOS end-to-end against a real git repo containing
non-cp1252 content (`ف.py` filename): _git_toplevel,
_git_dir, _find_git_index, _git_rev_parse_head,
_git_rev_list_range all return clean values without
AttributeError / UnicodeDecodeError.
* 7 round-trip bytes-decode pattern verifiers (parametrized
over Arabic ف, Hebrew א, Japanese 案, raw 0x81, multiple
cp1252-undefined bytes, real-world git diff headers).
* 1 sanity check that cp1252 strict DOES raise on 0x81
(proves the test environment can catch the bug class).
- Full suite: 378/378 pass in 5.56s.
- End-to-end tmux smoke test driving real claude 2.1.145 CLI:
Made a git commit via Bash tool call. All 4 hooks fired through
the fixed plugin path:
11:28:16.730 Hook called with args: …/plugin/hooks/security_reminder_hook.py
11:28:16.734 Processing: hook_event=UserPromptSubmit
11:28:16.825 Captured git baseline: 445f7f213256
11:28:19.923 Hook called with args: …
11:28:19.923 Processing: hook_event=PostToolUse, tool=Bash
11:28:19.971 Commit review: detected git commit in command
11:28:20.020 Commit review: 1/1 sha(s) resolved, 1 files
11:28:26.415 Hook called with args: …
11:28:26.416 Processing: hook_event=Stop
11:28:26.550 Stop hook: empty review set
Confirms: PYTHONUTF8=1 export doesn't break anything; converted
helpers (_git_rev_parse_head, _git_toplevel, _git_dir,
_find_git_index) run end-to-end without issue on the happy path.
NOT verified end-to-end on Windows with actual non-cp1252 content
in path/filename/stderr. The static-shape catcher pins the
regression class permanently. Reporter's PYTHONUTF8=1 workaround
empirically proves the encoding-mode fix works for the affected
scenario; this commit just bakes it in.
Closes #2099.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Diffstat (limited to 'plugins/security-guidance/hooks/gitutil.py')
| -rw-r--r-- | plugins/security-guidance/hooks/gitutil.py | 54 |
1 files changed, 41 insertions, 13 deletions
diff --git a/plugins/security-guidance/hooks/gitutil.py b/plugins/security-guidance/hooks/gitutil.py index 5495d18f..0144ef08 100644 --- a/plugins/security-guidance/hooks/gitutil.py +++ b/plugins/security-guidance/hooks/gitutil.py @@ -32,12 +32,17 @@ GIT_CMD = [ def _git_rev_parse_head(cwd): """Return the current HEAD SHA, or None if not a git repo / no commits.""" try: + # See #2099: text=True on Windows cp1252 crashes the reader thread on + # any UTF-8 byte undefined in cp1252 (e.g. via a git error message + # referencing a non-ASCII filename in stderr). stdout is a SHA so it + # IS safe; stderr is not. capture_output=True with bytes-by-default + # never decodes, so the reader thread can't crash. result = subprocess.run( [*GIT_CMD, "rev-parse", "HEAD"], - cwd=cwd, capture_output=True, text=True, timeout=5 + cwd=cwd, capture_output=True, timeout=5 ) if result.returncode == 0 and result.stdout.strip(): - return result.stdout.strip() + return result.stdout.decode("utf-8", errors="replace").strip() return None except (subprocess.TimeoutExpired, FileNotFoundError, OSError): return None @@ -52,13 +57,17 @@ def _find_git_index(cwd): Returns the absolute path to the index file, or None. """ try: + # See #2099: stdout here is a PATH which can contain non-ASCII bytes + # (e.g. C:\אבטחה\repo\.git). text=True decodes via cp1252 strict on + # Windows → crashes the reader thread → returns stdout=None → + # caller does .strip() on None → AttributeError. Decode manually. result = subprocess.run( [*GIT_CMD, "rev-parse", "--git-dir"], - cwd=cwd, capture_output=True, text=True, timeout=5 + cwd=cwd, capture_output=True, timeout=5 ) if result.returncode != 0: return None - git_dir = result.stdout.strip() + git_dir = result.stdout.decode("utf-8", errors="replace").strip() if not os.path.isabs(git_dir): git_dir = os.path.join(cwd, git_dir) index_path = os.path.join(git_dir, "index") @@ -128,9 +137,13 @@ def _temp_index(cwd, untracked_paths=None): else: add_args = None if add_args: + # No stdout used here (only returncode matters), but text=True + # still spawns reader threads that decode stderr — git error + # messages can reference non-ASCII filenames and crash on + # cp1252. See #2099. Drop text=True so bytes stay raw. subprocess.run( [*GIT_CMD, "add", "--intent-to-add"] + add_args, - cwd=cwd, capture_output=True, text=True, timeout=10, + cwd=cwd, capture_output=True, timeout=10, env=env, ) yield env @@ -144,11 +157,17 @@ def _temp_index(cwd, untracked_paths=None): def _git_toplevel(cwd): """Absolute repo root for `cwd`, or None if not in a work tree.""" try: + # See #2099: stdout is a PATH — `C:\אבטחה\repo` returned as UTF-8 + # bytes by git. text=True would decode via cp1252 strict on Windows + # → reader-thread crash. Decode manually with errors="replace". r = subprocess.run( [*GIT_CMD, "rev-parse", "--show-toplevel"], - cwd=cwd, capture_output=True, text=True, timeout=5, + cwd=cwd, capture_output=True, timeout=5, ) - return r.stdout.strip() if r.returncode == 0 and r.stdout.strip() else None + if r.returncode != 0: + return None + path = r.stdout.decode("utf-8", errors="replace").strip() + return path if path else None except (subprocess.TimeoutExpired, FileNotFoundError, OSError): return None @@ -164,13 +183,15 @@ def _git_dir(repo_root): callers can degrade (push-sweep state is best-effort). """ try: + # See #2099: stdout is a PATH (shared gitdir), may be non-ASCII. + # Decode bytes manually to avoid cp1252 reader-thread crash. r = subprocess.run( [*GIT_CMD, "rev-parse", "--git-common-dir"], - cwd=repo_root, capture_output=True, text=True, timeout=5, + cwd=repo_root, capture_output=True, timeout=5, ) if r.returncode != 0: return None - d = r.stdout.strip() + d = r.stdout.decode("utf-8", errors="replace").strip() return d if os.path.isabs(d) else os.path.join(repo_root, d) except (subprocess.TimeoutExpired, FileNotFoundError, OSError): return None @@ -179,13 +200,15 @@ def _git_dir(repo_root): def _git_rev_list_range(repo_root, base, head="HEAD"): """Shas in `base..head`, oldest→newest. Empty list on error.""" try: + # See #2099: stdout is ASCII SHAs, but stderr can carry git error + # messages referencing non-ASCII filenames — keep bytes raw. r = subprocess.run( [*GIT_CMD, "rev-list", "--reverse", f"{base}..{head}"], - cwd=repo_root, capture_output=True, text=True, timeout=10, + cwd=repo_root, capture_output=True, timeout=10, ) if r.returncode != 0: return [] - return [s for s in r.stdout.strip().split("\n") if s] + return [s for s in r.stdout.decode("utf-8", errors="replace").strip().split("\n") if s] except (subprocess.TimeoutExpired, FileNotFoundError, OSError): return [] @@ -220,9 +243,11 @@ def _git_diff_range(repo_root, base, head="HEAD"): def _detect_main_branch(repo_root): for ref in ("origin/HEAD", "origin/main", "origin/master", "main", "master"): try: + # See #2099: stdout is a SHA but stderr can carry non-ASCII git + # warnings — keep bytes raw to avoid cp1252 reader-thread crash. r = subprocess.run( [*GIT_CMD, "rev-parse", "--verify", "-q", ref], - cwd=repo_root, capture_output=True, text=True, timeout=5, + cwd=repo_root, capture_output=True, timeout=5, ) if r.returncode == 0 and r.stdout.strip(): return ref @@ -410,9 +435,12 @@ def _is_ancestor(cwd, maybe_ancestor, descendant): """True if `maybe_ancestor` is reachable from `descendant` (i.e. HEAD moved forward via commit/merge, not sideways via checkout).""" try: + # See #2099: only returncode matters, but text=True spawns reader + # threads that decode stderr — git error messages can carry non-ASCII + # filenames. Drop text=True to keep bytes raw, avoid cp1252 crash. result = subprocess.run( [*GIT_CMD, "merge-base", "--is-ancestor", maybe_ancestor, descendant], - cwd=cwd, capture_output=True, text=True, timeout=5, + cwd=cwd, capture_output=True, timeout=5, ) return result.returncode == 0 except (subprocess.TimeoutExpired, FileNotFoundError, OSError): |
