fix: run-attack-suite is spec-only — no implementation in red-team.sh (#1000)

Implement the attack catalogue loop (step 5a) in red-team.sh that was
previously a forward spec in the formula. The loop replays every *.jsonl
attack file through AttackRunner.s.sol with snapshot revert between files,
records LM total ETH before/after each attack, and injects results into
the adversarial agent prompt so it knows which strategies are already
catalogued.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
johba 2026-03-22 15:30:46 +00:00
parent 311b8192f6
commit 52ba6b2f38
3 changed files with 86 additions and 11 deletions

View file

@ -38,9 +38,8 @@ required = false
default = "onchain/script/backtesting/attacks"
description = """
Directory containing existing .jsonl attack patterns for the structured
attack suite. Not yet forwarded to red-team.sh see run-attack-suite step.
attack suite. Forwarded to red-team.sh as ATTACK_DIR.
"""
status = "planned" # consumed only when run-attack-suite is implemented
[inputs.claude_timeout]
type = "integer"
@ -104,7 +103,6 @@ already understood.
attack_source = "{attack_dir}/*.jsonl"
forge_script = "onchain/script/backtesting/AttackRunner.s.sol"
snapshot_mode = "revert-between-attacks"
status = "planned" # not yet implemented in red-team.sh; tracked for future PR
[[steps]]
id = "run-adversarial-agent"
@ -186,8 +184,7 @@ to the originating issue.
Comment includes:
- Verdict (floor_held / floor_broken).
- ETH extracted (formatted in ETH) and delta in basis points.
- Total attacks tried (agent-discovered count; structured suite count
added once run-attack-suite is implemented).
- Total attacks tried (agent-discovered count + structured suite count).
- Link to committed evidence file.
- If novel vectors were promoted: link to the attack-vector PR.
On floor_broken: also include the highest-yield attack strategy name and
@ -253,9 +250,8 @@ records, attack filenames, PR titles).
"""
run_attack_suite_gap = """
The run-attack-suite step (structured loop over attacks/*.jsonl via
AttackRunner.s.sol with snapshot revert between files) is not yet implemented
in red-team.sh. The current script runs only the adversarial Claude agent.
The step is documented here as a forward spec; implementation is tracked as a
follow-up issue.
The run-attack-suite step is implemented in red-team.sh (step 5a). It loops
through every *.jsonl file in the attack directory, replays each through
AttackRunner.s.sol, records LM total ETH before/after with snapshot revert
between files, and injects results into the agent prompt.
"""