From 7d58490dcd33438373ba0390e607a80a0486ec7a Mon Sep 17 00:00:00 2001 From: johba Date: Tue, 24 Mar 2026 20:17:20 +0000 Subject: [PATCH 1/2] fix: Red-team schema should document snapshot-isolation methodology for lm_eth fields (#1083) Co-Authored-By: Claude Opus 4.6 (1M context) --- evidence/README.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/evidence/README.md b/evidence/README.md index db4e530..24a5ff0 100644 --- a/evidence/README.md +++ b/evidence/README.md @@ -265,6 +265,38 @@ Records one adversarial red-team run against a candidate optimizer. | `attacks[].delta_bps` | integer | LM ETH change in basis points | | `attacks[].insight` | string | Key finding from this strategy | +### Snapshot-Isolation Methodology + +All red-team runs use **snapshot isolation** as the standard methodology. This +ensures that each attack is evaluated independently against the same initial +state, rather than against a cumulative balance modified by prior attacks. + +**How it works:** + +1. Before the first attack, the test runner records the initial `lm_eth_before` + value and takes an Anvil snapshot via `vm.snapshot()`. +2. Each attack executes against this snapshot: run the attack, measure + `lm_eth_after`, compute `delta_bps`, then revert to the snapshot via + `vm.revertTo()`. +3. The next attack begins from the exact same chain state as the previous one. + +**Field semantics under snapshot isolation:** + +| Field | Semantics | +|-------|-----------| +| `lm_eth_before` | LM total ETH at the shared initial snapshot — identical for every attack in the run | +| `lm_eth_after` | LM total ETH measured after this specific attack, before reverting | +| `attacks[].delta_bps` | Change relative to the shared `lm_eth_before`, not relative to any prior attack | + +**Key implications:** + +- `lm_eth_before` and `lm_eth_after` reflect **per-attack state**, not + cumulative historical balance. Each attack sees the same starting ETH. +- Attack results are independent and order-insensitive — reordering attacks does + not change any individual `delta_bps` value. +- The top-level `lm_eth_after` and `eth_extracted` fields reflect the + worst-case single attack, not a sum of all attacks. + --- ## Schema: `holdout/YYYY-MM-DD-prNNN.json` From 6f2b202b86993ece7889dedec54be5dbcd9ac914 Mon Sep 17 00:00:00 2001 From: johba Date: Tue, 24 Mar 2026 20:41:39 +0000 Subject: [PATCH 2/2] fix: address review feedback on snapshot-isolation docs (#1083) - Use anvil_snapshot/anvil_revert RPC methods instead of vm.snapshot()/vm.revertTo() - Remove incorrect claim about top-level lm_eth_after reflecting worst-case attack Co-Authored-By: Claude Opus 4.6 (1M context) --- evidence/README.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/evidence/README.md b/evidence/README.md index 24a5ff0..4574b2e 100644 --- a/evidence/README.md +++ b/evidence/README.md @@ -274,10 +274,10 @@ state, rather than against a cumulative balance modified by prior attacks. **How it works:** 1. Before the first attack, the test runner records the initial `lm_eth_before` - value and takes an Anvil snapshot via `vm.snapshot()`. + value and takes an Anvil snapshot via the `anvil_snapshot` RPC method. 2. Each attack executes against this snapshot: run the attack, measure `lm_eth_after`, compute `delta_bps`, then revert to the snapshot via - `vm.revertTo()`. + the `anvil_revert` RPC method. 3. The next attack begins from the exact same chain state as the previous one. **Field semantics under snapshot isolation:** @@ -294,8 +294,6 @@ state, rather than against a cumulative balance modified by prior attacks. cumulative historical balance. Each attack sees the same starting ETH. - Attack results are independent and order-insensitive — reordering attacks does not change any individual `delta_bps` value. -- The top-level `lm_eth_after` and `eth_extracted` fields reflect the - worst-case single attack, not a sum of all attacks. ---