diff options
| author | Jesse Vincent <jesse@fsck.com> | 2026-08-13 00:29:43 +0000 |
|---|---|---|
| committer | Drew Ritter <drew@primeradiant.com> | 2026-09-17 17:06:12 -0700 |
| commit | aa2d24178dd06afb1bc779b8852ad419d060afe1 (patch) | |
| tree | 29302b9b2bfc64bb3e9b4dcb3f2b7930a82c1c60 /docs/testing.md | |
| parent | 942005eeb233f48cfe4df7e9be57986440a58585 (diff) | |
docs(testing): describe the Quorum eval lab accurately, replacing stale Drill referencesdocs/drill-to-quorum
The evals harness was renamed Drill -> Quorum and rewritten from
Python/uv to Bun/TypeScript; docs/testing.md and CLAUDE.md still
described the old tool. Beyond the rename, the old text also
misdescribed the system: quorum is the harness CLI, one part of the
eval lab — it drives real coding-agent CLIs through a Gauntlet QA
agent and grades against scenario acceptance criteria plus
deterministic post-checks. The quick start now matches the eval
repo's actual commands (bun install / bun run quorum run
scenarios/<name> --coding-agent claude; scenarios are directories,
not *.yaml) and points at the Live Eval Risk section before anyone
runs a permissive-mode session.
Drift reported in closed PR #2121 (@JFWaskin); that PR's replacement
quick start kept the uv commands, so this rewrite goes from the eval
repo's README instead.
Diffstat (limited to 'docs/testing.md')
| -rw-r--r-- | docs/testing.md | 19 |
1 files changed, 10 insertions, 9 deletions
diff --git a/docs/testing.md b/docs/testing.md index d8ff016..19f8aed 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -14,23 +14,24 @@ Live in `tests/`. Currently: - `tests/codex-plugin-sync/` — bash sync verification. - `tests/kimi/` — bash/Python checks for Kimi plugin manifest wiring. - `tests/claude-code/test-helpers.sh`, `analyze-token-usage.py` — utilities used by remaining bash tests. -- `tests/claude-code/test-subagent-driven-development.sh` — agent-can-describe-SDD test (no drill counterpart; tests description-recall, not behavior). -- `tests/claude-code/test-subagent-driven-development-integration.sh` — extended SDD integration with token analysis (drill covers the YAGNI subset; bash adds commit-count, Claude Code task-tracking, and token telemetry assertions). -- `tests/claude-code/test-worktree-native-preference.sh` — RED-GREEN-REFACTOR validation for worktree skill (drill covers the PRESSURE phase; bash also covers RED/GREEN baselines). -- `tests/explicit-skill-requests/` — Haiku-specific, multi-turn, and skill-name-prompted tests not covered by drill. +- `tests/claude-code/test-subagent-driven-development.sh` — agent-can-describe-SDD test (no quorum counterpart; tests description-recall, not behavior). +- `tests/claude-code/test-subagent-driven-development-integration.sh` — extended SDD integration with token analysis (quorum covers the YAGNI subset; bash adds commit-count, Claude Code task-tracking, and token telemetry assertions). +- `tests/claude-code/test-worktree-native-preference.sh` — RED-GREEN-REFACTOR validation for worktree skill (quorum covers the PRESSURE phase; bash also covers RED/GREEN baselines). +- `tests/explicit-skill-requests/` — Haiku-specific, multi-turn, and skill-name-prompted tests not covered by quorum. - `tests/diagnosing-superpowers/test-skill-structure.sh` — structural checks for the diagnosing-superpowers skill (frontmatter, referenced files, leak scan, word budget); behavior-scenario eval records are kept by the maintainer outside the repo. Run plugin tests via the relevant directory's `run-*.sh` or `npm test`. ## Skill behavior evals -Live in `evals/`. Drill is the harness; scenarios live at `evals/scenarios/*.yaml`. See `evals/README.md` for setup. Quick start: +Live in `evals/` (the [superpowers-evals](https://github.com/prime-radiant-inc/superpowers-evals/) eval lab, since renamed from Drill). Quorum is the harness CLI — one part of the system: it drives real coding-agent CLIs through a Gauntlet QA agent and grades them against each scenario's acceptance criteria plus deterministic post-checks. Scenarios live at `evals/scenarios/<name>/`. See `evals/README.md` for setup, the container runtime, and the safety model. Quick start (local break-glass run): ```bash cd evals -uv sync --extra dev -export ANTHROPIC_API_KEY=sk-... -uv run drill run triggering-test-driven-development -b claude +bun install +export SUPERPOWERS_ROOT=/path/to/superpowers +bun run quorum run scenarios/triggering-test-driven-development --coding-agent claude +bun run quorum show <run-dir> ``` -Drill scenarios are slow (3-30+ minutes each) and run real LLM sessions. They are not part of CI today; the natural follow-up is a tiered model (fast subset on PR, full sweep nightly + on-demand). +Quorum scenarios are slow (3-30+ minutes each) and run real LLM sessions in permissive modes — read `evals/README.md`'s Live Eval Risk section first. Only the static gates (`bun run check`, `bun run quorum check`) are safe for public CI; the natural follow-up remains a tiered model (static gates on PR, live sweep nightly + on-demand). |
