diff options
| author | Jesse Vincent <jesse@fsck.com> | 2026-08-13 00:32:49 +0000 |
|---|---|---|
| committer | Jesse Vincent <jesse@fsck.com> | 2026-08-13 00:32:49 +0000 |
| commit | c7b456dd0ecd0eceb111396190c882607f0e974a (patch) | |
| tree | 4ca6615e2345230f3ef92a396eeb6921b654240f /tests/claude-code/test-subagent-driven-development.sh | |
| parent | 72ee5bbc5e489dc9c76fa64ba0fc31ab85185238 (diff) | |
tests: update SDD assertions and prompts to current skill behaviorfix/claude-code-test-suite
The SDD skill tests still asserted on pre-rename skill text, so correct
model answers failed the suite:
- "Read at beginning" asserted "Step 1|beginning|start|Load Plan";
the current skill has no numbered steps or Load Plan phase (setup
covers it: "Read the plan once" during Setup). A live run today
failed this assertion when the model correctly answered "during
setup". Pattern now accepts setup/before-dispatch paraphrases while
still requiring an at-the-start answer.
- "Provides text directly" asserted the removed provide-full-task-text
behavior; SDD now routes task requirements through brief files
(scripts/task-brief). The test now asks brief-file-vs-whole-plan and
asserts the brief-based flow.
- The integration test's prompt and summary told the agent to
"provide full task text to subagents (don't make them read files)",
contradicting the skill it verifies; reworded to the task-brief flow.
Also gave its direct `timeout 1800 claude -p` the same </dev/null
stdin guard as run_claude.
Live-LLM tests; verified with bash -n on every touched file.
Part of #2130; defects documented in PR #2071 by @ericyen97903-lab.
Diffstat (limited to 'tests/claude-code/test-subagent-driven-development.sh')
| -rwxr-xr-x | tests/claude-code/test-subagent-driven-development.sh | 12 |
1 files changed, 6 insertions, 6 deletions
diff --git a/tests/claude-code/test-subagent-driven-development.sh b/tests/claude-code/test-subagent-driven-development.sh index 151fc64..5bc1f66 100755 --- a/tests/claude-code/test-subagent-driven-development.sh +++ b/tests/claude-code/test-subagent-driven-development.sh @@ -4,7 +4,7 @@ # # No drill coverage: this test asks the agent to *describe* SDD (string- # matches its verbal explanation against expected keywords like -# "self-review", "skeptical", "worktree", "Step 1", "loop"). Drill scenarios +# "self-review", "skeptical", "worktree", "setup", "loop"). Drill scenarios # test behavior (real subagent dispatch, plan-following, review loops), # not description-recall. Kept by design. set -euo pipefail @@ -83,7 +83,7 @@ else exit 1 fi -if assert_contains "$output" "Step 1\|beginning\|start\|Load Plan" "Read at beginning"; then +if assert_contains "$output" "beginning\|start\|setup\|before.*dispatch\|before.*task" "Read at beginning"; then : # pass else exit 1 @@ -133,16 +133,16 @@ echo "" echo "Test 7: Task context provision..." output=$(run_claude "In subagent-driven-development, how does the controller provide task information to the implementer subagent? Answer using exactly this structure: -Controller provides: <directly or by file> -Implementer must read plan file: <yes or no>" "$CLAUDE_PROMPT_TIMEOUT") +Controller provides: <brief file or whole plan file> +Implementer must read whole plan file: <yes or no>" "$CLAUDE_PROMPT_TIMEOUT") -if assert_contains "$output" "provide.*directly\|full.*text\|paste\|include.*prompt" "Provides text directly"; then +if assert_contains "$output" "task-brief\|brief file\|brief.*path\|Controller provides:.*brief" "Provides task brief file"; then : # pass else exit 1 fi -if assert_contains "$output" "Implementer must read plan file:.*no" "Doesn't make subagent read file"; then +if assert_contains "$output" "Implementer must read whole plan file:.*no" "Doesn't make subagent read whole plan"; then : # pass else exit 1 |
