Merge pull request 'fix: Red-team schema should document snapshot-isolation methodology for lm_eth fields (#1083)' (#1145) from fix/issue-1083 into master
This commit is contained in:
commit
e1cd283f6a
1 changed files with 30 additions and 0 deletions
|
|
@ -265,6 +265,36 @@ Records one adversarial red-team run against a candidate optimizer.
|
||||||
| `attacks[].delta_bps` | integer | LM ETH change in basis points |
|
| `attacks[].delta_bps` | integer | LM ETH change in basis points |
|
||||||
| `attacks[].insight` | string | Key finding from this strategy |
|
| `attacks[].insight` | string | Key finding from this strategy |
|
||||||
|
|
||||||
|
### Snapshot-Isolation Methodology
|
||||||
|
|
||||||
|
All red-team runs use **snapshot isolation** as the standard methodology. This
|
||||||
|
ensures that each attack is evaluated independently against the same initial
|
||||||
|
state, rather than against a cumulative balance modified by prior attacks.
|
||||||
|
|
||||||
|
**How it works:**
|
||||||
|
|
||||||
|
1. Before the first attack, the test runner records the initial `lm_eth_before`
|
||||||
|
value and takes an Anvil snapshot via the `anvil_snapshot` RPC method.
|
||||||
|
2. Each attack executes against this snapshot: run the attack, measure
|
||||||
|
`lm_eth_after`, compute `delta_bps`, then revert to the snapshot via
|
||||||
|
the `anvil_revert` RPC method.
|
||||||
|
3. The next attack begins from the exact same chain state as the previous one.
|
||||||
|
|
||||||
|
**Field semantics under snapshot isolation:**
|
||||||
|
|
||||||
|
| Field | Semantics |
|
||||||
|
|-------|-----------|
|
||||||
|
| `lm_eth_before` | LM total ETH at the shared initial snapshot — identical for every attack in the run |
|
||||||
|
| `lm_eth_after` | LM total ETH measured after this specific attack, before reverting |
|
||||||
|
| `attacks[].delta_bps` | Change relative to the shared `lm_eth_before`, not relative to any prior attack |
|
||||||
|
|
||||||
|
**Key implications:**
|
||||||
|
|
||||||
|
- `lm_eth_before` and `lm_eth_after` reflect **per-attack state**, not
|
||||||
|
cumulative historical balance. Each attack sees the same starting ETH.
|
||||||
|
- Attack results are independent and order-insensitive — reordering attacks does
|
||||||
|
not change any individual `delta_bps` value.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Schema: `holdout/YYYY-MM-DD-prNNN.json`
|
## Schema: `holdout/YYYY-MM-DD-prNNN.json`
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue