summaryrefslogtreecommitdiff
path: root/docs/testing.md
diff options
context:
space:
mode:
authorJesse Vincent <jesse@fsck.com>2026-09-18 18:38:59 +0000
committerJesse Vincent <jesse@fsck.com>2026-09-18 18:38:59 +0000
commit8412aaf0385cbfd850675093f01b16c88052f268 (patch)
tree88ae48d06c4d104d65e35d345e65f7c210189564 /docs/testing.md
parentb36e0829c6d0140e93cfef2ca599b1b07d4a7797 (diff)
Release v6.4.0: diagnosing-superpowers and movie skills, Native execution, OpenCode 2.0, Muse, and Qwen Code supportrelease/v6.4.0
Everything on dev since the v6.3.0 merge-back (777ceb5), as one commit. Full notes: RELEASE-NOTES.md, v6.4.0 section. Claude-Session: https://claude.ai/code/session_01PRUnVUm4g4EcP9eNjAiT2B
Diffstat (limited to 'docs/testing.md')
-rw-r--r--docs/testing.md20
1 files changed, 11 insertions, 9 deletions
diff --git a/docs/testing.md b/docs/testing.md
index 414d697..19f8aed 100644
--- a/docs/testing.md
+++ b/docs/testing.md
@@ -14,22 +14,24 @@ Live in `tests/`. Currently:
- `tests/codex-plugin-sync/` — bash sync verification.
- `tests/kimi/` — bash/Python checks for Kimi plugin manifest wiring.
- `tests/claude-code/test-helpers.sh`, `analyze-token-usage.py` — utilities used by remaining bash tests.
-- `tests/claude-code/test-subagent-driven-development.sh` — agent-can-describe-SDD test (no drill counterpart; tests description-recall, not behavior).
-- `tests/claude-code/test-subagent-driven-development-integration.sh` — extended SDD integration with token analysis (drill covers the YAGNI subset; bash adds commit-count, Claude Code task-tracking, and token telemetry assertions).
-- `tests/claude-code/test-worktree-native-preference.sh` — RED-GREEN-REFACTOR validation for worktree skill (drill covers the PRESSURE phase; bash also covers RED/GREEN baselines).
-- `tests/explicit-skill-requests/` — Haiku-specific, multi-turn, and skill-name-prompted tests not covered by drill.
+- `tests/claude-code/test-subagent-driven-development.sh` — agent-can-describe-SDD test (no quorum counterpart; tests description-recall, not behavior).
+- `tests/claude-code/test-subagent-driven-development-integration.sh` — extended SDD integration with token analysis (quorum covers the YAGNI subset; bash adds commit-count, Claude Code task-tracking, and token telemetry assertions).
+- `tests/claude-code/test-worktree-native-preference.sh` — RED-GREEN-REFACTOR validation for worktree skill (quorum covers the PRESSURE phase; bash also covers RED/GREEN baselines).
+- `tests/explicit-skill-requests/` — Haiku-specific, multi-turn, and skill-name-prompted tests not covered by quorum.
+- `tests/diagnosing-superpowers/test-skill-structure.sh` — structural checks for the diagnosing-superpowers skill (frontmatter, referenced files, leak scan, word budget); behavior-scenario eval records are kept by the maintainer outside the repo.
Run plugin tests via the relevant directory's `run-*.sh` or `npm test`.
## Skill behavior evals
-Live in `evals/`. Drill is the harness; scenarios live at `evals/scenarios/*.yaml`. See `evals/README.md` for setup. Quick start:
+Live in `evals/` (the [superpowers-evals](https://github.com/prime-radiant-inc/superpowers-evals/) eval lab, since renamed from Drill). Quorum is the harness CLI — one part of the system: it drives real coding-agent CLIs through a Gauntlet QA agent and grades them against each scenario's acceptance criteria plus deterministic post-checks. Scenarios live at `evals/scenarios/<name>/`. See `evals/README.md` for setup, the container runtime, and the safety model. Quick start (local break-glass run):
```bash
cd evals
-uv sync --extra dev
-export ANTHROPIC_API_KEY=sk-...
-uv run drill run triggering-test-driven-development -b claude
+bun install
+export SUPERPOWERS_ROOT=/path/to/superpowers
+bun run quorum run scenarios/triggering-test-driven-development --coding-agent claude
+bun run quorum show <run-dir>
```
-Drill scenarios are slow (3-30+ minutes each) and run real LLM sessions. They are not part of CI today; the natural follow-up is a tiered model (fast subset on PR, full sweep nightly + on-demand).
+Quorum scenarios are slow (3-30+ minutes each) and run real LLM sessions in permissive modes — read `evals/README.md`'s Live Eval Risk section first. Only the static gates (`bun run check`, `bun run quorum check`) are safe for public CI; the natural follow-up remains a tiered model (static gates on PR, live sweep nightly + on-demand).