johba/harb
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
harb/scripts/harb-evaluator/AGENTS.md
johba b276392e7a chore: gardener housekeeping 2026-03-23 
- Update all AGENTS.md watermarks to HEAD (224edcc)
- onchain/AGENTS.md: document VWAPTracker _hasRecenterTick guard (#609),
  overflow guard for slots 0-7 (#997), Floor Ratchet defeated (#1067),
  fee-income delta_bps audit trail (#1084)
- landing/AGENTS.md: document SecurityInfo.vue component (#147)

Issues actioned via API:
- Unblocked #1099, #1100, #1101 (all prereqs 1031/997/1067/1054 closed)
- Created #1134 bundled backlog: onchain source quality cleanup (4 dust items)
- Closed dust #601, #627, #739, #741 → bundled into #1134
2026-03-23 07:04:19 +00:00

3 KiB
Raw Blame History

Agent Brief: harb-evaluator

The evaluator runtime executes formula-defined pipelines. Scripts in this directory handle stack lifecycle, scenario execution, evidence collection, and the adversarial agent harness.

Directory Layout

File Purpose
evaluate.sh Holdout gate: worktree checkout → Docker stack → Playwright scenarios → teardown
red-team.sh Adversarial agent runner: Anvil bootstrap → attack suite → Claude agent → evidence
run-protocol.sh On-chain health snapshot (TVL, fees, positions, rebalances) via cast/forge
run-resources.sh Infrastructure snapshot (disk, RAM, API budget, CI queue) via shell commands
bootstrap-light.sh Lightweight Anvil bootstrap with contract deployment (used by red-team.sh)
promote-attacks.sh Deduplicate and PR novel attack vectors discovered by the red-team agent
export-attacks.py Extract cast send commands from agent stream log into .jsonl attack files
red-team-program.md System prompt for the adversarial Claude agent
holdout.config.ts Playwright config for holdout scenario execution
helpers/ TypeScript helpers: RPC, assertions, swap, stake, floor, market, reporting
scenarios/ Holdout scenario scripts and the passive-confidence suite

Exit Code Convention

All evaluator scripts follow the same three-code contract:

Code Meaning
0 Success / gate passed
1 Gate failed (scenario or attack found a problem)
2 Infrastructure error (stack down, missing dependency, RPC unreachable)

Formulas and the orchestrator rely on these codes for routing — do not introduce additional exit codes without updating the formula TOML.

Stack Lifecycle

Heavy formulas (run-holdout, run-red-team, run-evolution) need a running Anvil or full Docker stack. Port 8545 is shared — these formulas are mutually exclusive and must not run concurrently.

  • evaluate.sh manages Docker compose (harb-eval-{pr} project) with full teardown via shell trap.
  • red-team.sh uses bootstrap-light.sh for a lightweight Anvil-only stack (no Docker). Cleanup is also trap-registered.
  • run-protocol.sh and run-resources.sh are lightweight — no Anvil, no Docker.

Evidence Output

Every script writes its evidence file to evidence/{category}/{date}.json conforming to the schema in evidence/README.md. The deliver step in each formula handles committing and posting an issue comment.

Adding a New Evaluator Script

  1. Place the script in this directory. Use #!/usr/bin/env bash and set -euo pipefail.
  2. Follow the exit code convention (0 / 1 / 2).
  3. Accept configuration via environment variables, not positional args (except evaluate.sh which takes a PR number).
  4. Write evidence to evidence/{category}/{date}.json.
  5. Wire it into a formula TOML in formulas/ — see formulas/AGENTS.md for the full walkthrough.