diff options
| author | Mohamed Hegazy <mhegazy@anthropic.com> | 2026-06-11 23:31:55 -0700 |
|---|---|---|
| committer | Mohamed Hegazy <mhegazy@anthropic.com> | 2026-06-11 23:31:55 -0700 |
| commit | e7fe15d9ba8bc02a40cdccdb184f6d61a58b33cc (patch) | |
| tree | 1c0ad01cf56353d630cf1e5a9fb8612a911307ff | |
| parent | e97f9a40b9c1a6b9aeb0cfab429121163f2197c6 (diff) | |
security-guidance: pip --target fallback when venv can't bootstrap pip (2.0.4 → 2.0.5)venv-target-fallback
Option A, the data-gated fix for venv_ensurepip_fail (#2154 follow-up).
v2.0.4 telemetry made the call: of the venv_ensurepip_fail cohort, ~95%
HAVE pip (sdk_has_pip=true) and run Python 3.11–3.14 — so it's not the
Apple-3.9 problem; it's modern interpreters where `python -m venv` can't
bootstrap pip (Debian python3-venv absent, or python.org/pyenv builds
without ensurepip) but pip itself works. `pip install --target` needs only
pip, so it recovers the agentic reviewer for them instead of degrading to
pattern + single-shot review.
Producer (ensure_agent_sdk.py):
- New outcomes BUILT_TARGET=7, NOOP_TARGET=8; new phase pip_target=5.
- _build_via_target(): `pip install --target <state>/agent-sdk-libs
--upgrade --prefer-binary claude-agent-sdk`. Failures categorized via
_pip_err_from_stderr (sibling of main()'s pip chain — kept separate to
avoid disturbing the working venv categorizer); errno embedded for
OSError-family exceptions.
- _target_sdk_importable(): probes a prior target install → NOOP_TARGET.
Dir-check short-circuits before any subprocess, and it's only reached
when there's no working venv, so the 81% NOOP_VENV cohort never pays.
- main() falls through to the target build ONLY on venv_ensurepip_fail;
every other venv/pip failure stays terminal BUILD_FAILED. The sentinel
is released before the target build so a retry isn't seen as SKIP_SENTINEL.
Consumer (llm.py):
- _inject_agent_sdk_venv_into_syspath() adds the flat agent-sdk-libs dir
(packages sit directly in it, not under site-packages). The existing
pywin32 .pth bootstrap applies (target installs don't run .pth either).
No change to the happy path — the new branch is taken only on the
ensurepip failure, and the extra candidate dir is a no-op when absent.
Verified locally on macOS Python 3.13:
- py_compile clean.
- 30 new tests (test_venv_target_fallback.py): outcome/phase codes
(append-only, 4 stays retired), _pip_err_from_stderr categories,
_build_via_target success/CalledProcessError/timeout/exc+errno (mocked
subprocess), _target_sdk_importable dir-short-circuit, main() wiring
(ensurepip→target fallthrough + NOOP_TARGET probe + sentinel release),
consumer adds the flat dir. Full suite 533/533 pass + 2 skipped.
- END-TO-END harness (real install, simulated ensurepip failure):
main() → BUILT_TARGET, target dir has claude_agent_sdk; 2nd run →
NOOP_TARGET; consumer _inject → `import claude_agent_sdk` resolves
FROM the --target dir. Full chain proven without needing a
broken-ensurepip box.
- Real `pip install --target` + import confirmed independently (exit 0,
SDK imports from the flat layout).
NOT validated in tmux: the ensurepip failure can't be reproduced on macOS
(working ensurepip), so the fallback was proven via the real-install
harness above instead. The happy path (NOOP_VENV / normal agentic review)
is unchanged and covered by the existing hook-smoke suite.
Version 2.0.4 -> 2.0.5 per the per-PR-bump policy (#2114).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
| -rw-r--r-- | .claude-plugin/marketplace.json | 2 | ||||
| -rw-r--r-- | plugins/security-guidance/.claude-plugin/plugin.json | 2 | ||||
| -rw-r--r-- | plugins/security-guidance/hooks/ensure_agent_sdk.py | 116 | ||||
| -rw-r--r-- | plugins/security-guidance/hooks/llm.py | 6 |
4 files changed, 124 insertions, 2 deletions
diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index de524a6e..1e33ce36 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -2387,7 +2387,7 @@ { "name": "security-guidance", "description": "Security review for Claude-generated code. Pattern-based warnings on edits, LLM-powered diff review on Stop, and an agentic commit reviewer that catches injection, XSS, SSRF, hardcoded secrets, and 25+ other vulnerability classes.", - "version": "2.0.4", + "version": "2.0.5", "author": { "name": "Anthropic", "email": "support@anthropic.com" diff --git a/plugins/security-guidance/.claude-plugin/plugin.json b/plugins/security-guidance/.claude-plugin/plugin.json index fa96245d..83813a4a 100644 --- a/plugins/security-guidance/.claude-plugin/plugin.json +++ b/plugins/security-guidance/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "security-guidance", - "version": "2.0.4", + "version": "2.0.5", "description": "Security review for Claude-generated code. Pattern-based warnings on edits, LLM-powered diff review on Stop, and an agentic commit reviewer that catches injection, XSS, SSRF, hardcoded secrets, and 25+ other vulnerability classes.", "author": { "name": "David Dworken", diff --git a/plugins/security-guidance/hooks/ensure_agent_sdk.py b/plugins/security-guidance/hooks/ensure_agent_sdk.py index 13ca4650..69df6a22 100644 --- a/plugins/security-guidance/hooks/ensure_agent_sdk.py +++ b/plugins/security-guidance/hooks/ensure_agent_sdk.py @@ -40,6 +40,15 @@ BUILD_FAILED = 3 # venv create or pip install raised/timed out SKIP_SENTINEL = 5 # another SessionStart is currently building HOOK_PY_INCOMPATIBLE = 6 # hook interpreter is <3.10 — SDK syntax can't load # here no matter how the venv was built. See #2071. +# --target fallback: when `python -m venv` can't bootstrap pip (ensurepip +# missing — Debian python3-venv not installed, or a python.org/pyenv build +# without ensurepip), fall back to `pip install --target <dir>` which needs +# only the system pip, not venv/ensurepip. Telemetry (v2.0.4 sdk_has_pip +# probe) confirmed ~95% of venv_ensurepip_fail users HAVE pip, so this +# recovers the agentic reviewer for them instead of degrading to pattern + +# single-shot review. See #2154 follow-up. +BUILT_TARGET = 7 # venv ensurepip failed → SDK pip-installed via --target +NOOP_TARGET = 8 # --target libs already present and importable # Phase + err-kind integer encoding for sdk_bootstrap_phase / sdk_bootstrap_err. @@ -63,6 +72,7 @@ SDK_BOOTSTRAP_PHASE_CODES = { "venv": 2, # python -m venv --clear "pip": 3, # pip install "main": 4, # uncaught exception above main() + "pip_target": 5, # `pip install --target` fallback (venv ensurepip failed) } SDK_BOOTSTRAP_ERR_CODES = { "pip_no_match": 1, @@ -242,6 +252,96 @@ def _probe_has_pip() -> bool: return False +def _pip_err_from_stderr(stderr_b): + """Categorize a pip-install stderr into a known err_kind (the pip subset + of SDK_BOOTSTRAP_ERR_CODES). Used by the --target fallback; mirrors the + pip branches of main()'s inline categorizer. Kept as a sibling rather + than extracting main()'s chain (which also has venv-phase branches) to + avoid disturbing the working venv categorization.""" + if isinstance(stderr_b, bytes): + s = stderr_b.decode("utf-8", errors="replace") + else: + s = str(stderr_b or "") + low = s.lower() + if "no matching distribution" in low or "could not find a version" in low: + return "pip_no_match" + if ("name or service not known" in low or "name resolution" in low + or "nodename nor servname" in low or "temporary failure in name" in low): + return "dns_fail" + if "connection refused" in low or "connection reset" in low: + return "conn_refused" + if "ssl" in low and ("verify" in low or "certificate" in low): + return "ssl_verify" + if "permission denied" in low or "read-only file system" in low: + return "perm_denied" + if "no module named pip" in low or "no module named ensurepip" in low: + return "no_pip" + if "no space left" in low or "disk quota" in low: + return "disk_full" + if "proxy" in low and ("authent" in low or "tunnel" in low or "407" in low): + return "proxy_auth" + if "timeout" in low or "timed out" in low: + return "stderr_timeout" + tail = next((ln.strip() for ln in reversed(s.splitlines()) if ln.strip()), "")[:60] + return f"other:{tail}" if tail else "other" + + +def _target_dir(state_dir) -> Path: + return Path(state_dir) / "agent-sdk-libs" + + +def _target_sdk_importable(state_dir) -> bool: + """True iff the --target libs dir has an importable claude_agent_sdk, + probed with THIS interpreter (the one llm.py will import it from) and the + target dir prepended to sys.path. Cheap dir-check first to avoid a + subprocess on the common no-target path.""" + target = _target_dir(state_dir) + if not (target / "claude_agent_sdk").is_dir(): + return False + try: + r = subprocess.run( + [sys.executable, "-c", + "import sys; sys.path.insert(0, sys.argv[1]); import claude_agent_sdk", + str(target)], + capture_output=True, timeout=10, + ) + return r.returncode == 0 + except Exception: + return False + + +def _build_via_target(state_dir) -> tuple[int, str, str]: + """Fallback install when `python -m venv` can't bootstrap pip (ensurepip + missing — Debian python3-venv absent, or a python.org/pyenv build without + ensurepip). `pip install --target <dir>` needs only the system pip, not + venv/ensurepip. v2.0.4 telemetry (sdk_has_pip) confirmed ~95% of + venv_ensurepip_fail users have pip. The consumer (llm.py) adds this flat + dir to sys.path. Returns (outcome, err_phase, err_kind). + + --upgrade so a stale/partial target dir from a prior failed attempt + doesn't make pip refuse; --prefer-binary mirrors the venv path's wheel + preference (ARM64 Windows cryptography).""" + target = _target_dir(state_dir) + try: + subprocess.run( + [sys.executable, "-m", "pip", "install", + "--target", str(target), "--upgrade", + "--disable-pip-version-check", "--prefer-binary", + "claude-agent-sdk"], + capture_output=True, timeout=120, check=True, + ) + return BUILT_TARGET, "", "" + except subprocess.CalledProcessError as e: + return BUILD_FAILED, "pip_target", _pip_err_from_stderr(e.stderr) + except subprocess.TimeoutExpired: + return BUILD_FAILED, "pip_target", "subprocess_timeout" + except Exception as e: + errno = getattr(e, "errno", None) + if isinstance(errno, int): + return BUILD_FAILED, "pip_target", f"exc:{type(e).__name__}:{errno}" + return BUILD_FAILED, "pip_target", f"exc:{type(e).__name__}" + + def _sdk_on_syspath() -> bool: # find_spec is ~10ms; actually importing the SDK pulls in # transitive deps and costs ~800ms — too heavy for a @@ -330,6 +430,12 @@ def main() -> tuple[int, str, str]: except Exception: pass # broken venv; rebuild below + # If a prior run installed the SDK via the --target fallback (ensurepip + # path), reuse it. Only reached when there's no working venv, so healthy + # NOOP_VENV users never pay for this probe. + if _target_sdk_importable(state_dir): + return NOOP_TARGET, "", "" + err_phase = "" err_kind = "" we_own_sentinel = False @@ -444,6 +550,16 @@ def main() -> tuple[int, str, str]: "", )[:60] err_kind = f"other:{tail}" if tail else "other" + # venv couldn't bootstrap pip (ensurepip missing) but pip itself may + # work — fall back to a flat `pip install --target`. Only this one + # category falls through; every other venv/pip failure is terminal. + # The finally block unlinks our sentinel first (so the target build + # isn't blocked by it); _build_via_target does the target install. + if err_kind == "venv_ensurepip_fail": + if we_own_sentinel: + sentinel.unlink(missing_ok=True) + we_own_sentinel = False + return _build_via_target(state_dir) return BUILD_FAILED, err_phase, err_kind except subprocess.TimeoutExpired: return BUILD_FAILED, err_phase, "subprocess_timeout" diff --git a/plugins/security-guidance/hooks/llm.py b/plugins/security-guidance/hooks/llm.py index e2ac37db..8b7feef8 100644 --- a/plugins/security-guidance/hooks/llm.py +++ b/plugins/security-guidance/hooks/llm.py @@ -55,6 +55,12 @@ def _inject_agent_sdk_venv_into_syspath(state_dir): candidates = ( glob.glob(os.path.join(venv_root, "lib", "python*", "site-packages")) + glob.glob(os.path.join(venv_root, "Lib", "site-packages")) + # `pip install --target` fallback (ensure_agent_sdk BUILT_TARGET, used + # when venv can't bootstrap pip): a FLAT layout — packages sit directly + # in agent-sdk-libs/, not under a site-packages subdir. See #2154 + # follow-up. The pywin32 .pth bootstrap below applies here too (target + # installs don't process .pth at runtime, same as a manual venv insert). + + [os.path.join(state_dir, "agent-sdk-libs")] ) added = False for sp in candidates: |
