fix: run-attack-suite is spec-only — no implementation in red-team.sh (#1000)
Implement the attack catalogue loop (step 5a) in red-team.sh that was previously a forward spec in the formula. The loop replays every *.jsonl attack file through AttackRunner.s.sol with snapshot revert between files, records LM total ETH before/after each attack, and injects results into the adversarial agent prompt so it knows which strategies are already catalogued. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
311b8192f6
commit
52ba6b2f38
3 changed files with 86 additions and 11 deletions
|
|
@ -38,9 +38,8 @@ required = false
|
|||
default = "onchain/script/backtesting/attacks"
|
||||
description = """
|
||||
Directory containing existing .jsonl attack patterns for the structured
|
||||
attack suite. Not yet forwarded to red-team.sh — see run-attack-suite step.
|
||||
attack suite. Forwarded to red-team.sh as ATTACK_DIR.
|
||||
"""
|
||||
status = "planned" # consumed only when run-attack-suite is implemented
|
||||
|
||||
[inputs.claude_timeout]
|
||||
type = "integer"
|
||||
|
|
@ -104,7 +103,6 @@ already understood.
|
|||
attack_source = "{attack_dir}/*.jsonl"
|
||||
forge_script = "onchain/script/backtesting/AttackRunner.s.sol"
|
||||
snapshot_mode = "revert-between-attacks"
|
||||
status = "planned" # not yet implemented in red-team.sh; tracked for future PR
|
||||
|
||||
[[steps]]
|
||||
id = "run-adversarial-agent"
|
||||
|
|
@ -186,8 +184,7 @@ to the originating issue.
|
|||
Comment includes:
|
||||
- Verdict (floor_held / floor_broken).
|
||||
- ETH extracted (formatted in ETH) and delta in basis points.
|
||||
- Total attacks tried (agent-discovered count; structured suite count
|
||||
added once run-attack-suite is implemented).
|
||||
- Total attacks tried (agent-discovered count + structured suite count).
|
||||
- Link to committed evidence file.
|
||||
- If novel vectors were promoted: link to the attack-vector PR.
|
||||
On floor_broken: also include the highest-yield attack strategy name and
|
||||
|
|
@ -253,9 +250,8 @@ records, attack filenames, PR titles).
|
|||
"""
|
||||
|
||||
run_attack_suite_gap = """
|
||||
The run-attack-suite step (structured loop over attacks/*.jsonl via
|
||||
AttackRunner.s.sol with snapshot revert between files) is not yet implemented
|
||||
in red-team.sh. The current script runs only the adversarial Claude agent.
|
||||
The step is documented here as a forward spec; implementation is tracked as a
|
||||
follow-up issue.
|
||||
The run-attack-suite step is implemented in red-team.sh (step 5a). It loops
|
||||
through every *.jsonl file in the attack directory, replays each through
|
||||
AttackRunner.s.sol, records LM total ETH before/after with snapshot revert
|
||||
between files, and injects results into the agent prompt.
|
||||
"""
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue