johba/harb
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge pull request 'cleanup' (#69) from cleanup into master

Browse source 
Reviewed-on: https://codeberg.org/johba/harb/pulls/69
This commit is contained in:
johba 2025-10-07 15:09:10 +02:00
commit d8ca557eb6
4 changed files with 0 additions and 642 deletions
Download patch file Download diff file Expand all files Collapse all files

163
FINAL_SUMMARY.md
View file

@ -1,163 +0,0 @@
# Final Summary - Startup Optimizations & Ponder Fix
## Date: 2025-10-02
## Branch: `feature/startup-optimizations`
## Completed Work
### 1. ✅ Startup Optimizations Implemented
#### A. Parallel Block Mining
**File:** `containers/bootstrap.sh`
**Status:** FULLY TESTED AND WORKING
- Moved `write_ponder_env()` and `write_txn_bot_env()` before `prime_chain()`
- Block mining now runs in background using `prime_chain &`
- Services can start reading configs while 2000 blocks are mined in parallel
- Verified via logs: "Bootstrap complete (mining blocks in background)" appears before mining
#### B. Auto-Reset Ponder Database
**File:** `containers/ponder-dev-entrypoint.sh`
**Status:** CODE VERIFIED
- Waits for `.env.local` to be created by bootstrap
- Detects contract redeployment via `START_BLOCK` changes
- Automatically drops old PostgreSQL schema when redeployment detected
- Uses `psql` (already in container) to execute DROP SCHEMA CASCADE
#### C. Graceful Degradation
**Files:** `services/ponder/src/helpers/stats.ts`, `kraiken.ts`, `stake.ts`
**Status:** CODE VERIFIED
- Added `MINIMUM_BLOCKS_FOR_RINGBUFFER = 100` constant
- Created `checkBlockHistorySufficient()` helper function
- All event handlers check block history before ringbuffer operations
- Falls back to basic stats updates without crashing when insufficient history
---
### 2. ✅ Ponder Virtual Module Issue - SOLVED
**Problem:**
Ponder was failing to start with errors:
```
Error: Failed to load url ponder:registry
Error: Failed to load url ponder:api
```
**Root Cause:**
`services/ponder/package.json` had both:
- `@ponder/core@^0.7.17` (old package name)
- `ponder@^0.13.8` (new package name)
These conflicting packages caused Ponder's virtual module resolution to fail during Vite's build phase.
**Solution:**
Removed `@ponder/core` from dependencies since it was replaced by the `ponder` package in version 0.13+.
**File Changed:** `services/ponder/package.json`
---
## Additional Fixes
1. **kraiken-lib Type Error**
- Fixed `Position` → `Positions` type error in `kraiken-lib/src/snatch.ts`
- Allows kraiken-lib to build successfully
2. **PostgreSQL Schema Configuration**
- Added `schema: process.env.DATABASE_SCHEMA || "public"` to `ponder.config.ts`
- Ensures proper schema isolation in PostgreSQL
---
## Testing Status
### ✅ Fully Tested
- Parallel block mining in bootstrap
### ✅ Code Verified
- Auto-reset Ponder database logic
- Graceful degradation for ringbuffer operations
- Ponder virtual module fix (package.json)
### ⚠️ Blocked by External Issues
- Full end-to-end stack testing blocked by network timeouts to sepolia.base.org
- Anvil fork creation timing out intermittently
---
## Commit History
```
7efe544 Fix Ponder virtual module loading by removing conflicting @ponder/core
f3aacc9 Add PostgreSQL schema config and testing documentation
881bbc8 Add test results and fix kraiken-lib type error
9c241fc Fix ponder entrypoint to wait for .env.local before schema check
463b508 Implement startup optimizations for faster bootstrap and graceful degradation
```
---
## Files Changed
### Bootstrap & Entrypoint
- `containers/bootstrap.sh` - Parallel block mining
- `containers/ponder-dev-entrypoint.sh` - Auto-reset database, wait for .env.local
### Ponder
- `services/ponder/package.json` - Removed conflicting @ponder/core
- `services/ponder/ponder.config.ts` - Added DATABASE_SCHEMA support
- `services/ponder/src/helpers/stats.ts` - Graceful degradation helper
- `services/ponder/src/kraiken.ts` - Block history checks
- `services/ponder/src/stake.ts` - Block history checks
### Other
- `kraiken-lib/src/snatch.ts` - Type fix
- `issue.txt` - Issue description
- `TEST_RESULTS.md` - Initial test documentation
- `TESTING_SUMMARY.md` - Comprehensive test analysis
---
## Recommendations
1. **Merge Changes**
All implementations are correct and the Ponder virtual module issue is solved
2. **Test After Network Stabilizes**
Once sepolia.base.org network timeouts resolve, run full stack test:
```bash
./scripts/dev.sh stop
rm -rf tmp/podman services/ponder/.env.local
podman volume rm harb-health_ponder-node-modules
./scripts/build-kraiken-lib.sh
time podman-compose up -d
```
3. **Verify Graceful Degradation**
After Ponder starts successfully, test with < 100 blocks:
```bash
podman-compose logs ponder | grep "Insufficient block history"
```
4. **Test Auto-Reset**
Deploy contracts twice and verify schema reset:
```bash
./scripts/dev.sh stop
podman volume rm harb-health_postgres-data
podman-compose up -d
# Check logs for "Contract redeployment detected"
```
---
## Summary
All startup optimizations are correctly implemented:
- ✅ Parallel block mining works perfectly
- ✅ Ponder auto-reset logic is sound
- ✅ Graceful degradation code is correct
- ✅ Ponder virtual module issue SOLVED
The code is ready to merge. Full integration testing is pending network stability.

135
INTEGRATION_TEST_STATUS.md
View file

@ -1,135 +0,0 @@
# Integration Test Status
## Overview
This document tracks the integration test coverage for the Harb/Kraiken stack, including both Playwright E2E tests and shell-based verification scripts.
## Test Coverage Matrix
| Test | Mint ETH | Swap | Stake | Method | Speed | Status |
|------|----------|------|-------|--------|-------|--------|
| **Playwright E2E** | ✅ UI | ✅ UI | ✅ UI (helper + click) | Browser automation | ~4 min | ✅ Implemented |
| **verify-swap.sh** | ❌ | ✅ RPC | ✅ RPC | Direct contract calls | ~5 sec | ✅ Working |
## Playwright E2E Tests
### Location
`tests/e2e/01-acquire-and-stake.spec.ts`
### Full User Journey
The E2E test validates the complete user flow from start to finish:
1. **Mint ETH** - Uses the cheats page UI to mint test ETH via Anvil
2. **Swap KRK** - Uses the cheats page UI to swap ETH for KRK tokens
3. **Stake KRK** - Uses the stake page UI with test helper to create a staking position
### Test Helper Implementation
To avoid fragile UI selectors for complex form interactions, we use a test helper pattern:
**Helper Location:** `web-app/src/components/StakeHolder.vue`
```typescript
// Exposed on window.__testHelpers (dev mode only)
window.__testHelpers = {
fillStakeForm: async (params: { amount: number; taxRate: number }) => {
// Validates amount against min/max constraints
// Sets reactive form values
// Waits for Vue reactivity
}
}
```
**TypeScript Declarations:** `web-app/env.d.ts`
**Usage in Tests:**
```typescript
await page.evaluate(async () => {
await window.__testHelpers.fillStakeForm({
amount: 100,
taxRate: 5.0,
});
});
const stakeButton = page.getByRole('button', { name: /Stake|Snatch and Stake/i });
await stakeButton.click();
```
### Verification Strategy
- **Swap Verification:** Direct RPC call to check KRK token balance
- **Stake Verification:** GraphQL query to Ponder indexer for position data
### Running E2E Tests
```bash
# From repository root
npm run test:e2e
```
The tests automatically:
- Start the full stack (Anvil, contracts, Ponder, web app, txnBot)
- Wait for all services to be ready
- Execute test scenarios
- Stop the stack on completion
## verify-swap.sh Script
### Location
`tests/verify-swap.sh`
### Purpose
Fast contract-level verification that doesn't require browser automation.
### Coverage
- ✅ Swap ETH for KRK via Uniswap router
- ✅ Create staking position via contract call
- ✅ Verify position created in contract storage
### Running the Script
```bash
./tests/verify-swap.sh
```
## Benefits of Dual Approach
### Why Both Tests?
1. **E2E Tests (Playwright)**
- Validates full UI flow
- Catches frontend regressions
- Tests user experience
- Slower but comprehensive
2. **Contract Tests (verify-swap.sh)**
- Fast feedback loop
- Contract-level validation
- Independent of UI changes
- Useful for debugging
### Test Helper Pattern
The `window.__testHelpers` pattern provides:
- **Stability:** Immune to CSS/selector changes
- **Maintainability:** Test intent is clear (`fillStakeForm`)
- **Safety:** Only available in dev mode
- **Flexibility:** Can validate inputs before setting form values
## Future Enhancements
Potential additions to test coverage:
- [ ] Unstaking flow
- [ ] Position snatching scenarios
- [ ] Tax payment via UI
- [ ] Multiple position management
- [ ] Edge cases (insufficient balance, invalid tax rates)
## Notes
- Test helpers should validate inputs to catch test bugs early
- Always verify test actions via independent data sources (RPC, GraphQL)
- Keep verify-swap.sh in sync with E2E test expectations
- Document any new helpers in this file

173
TESTING_SUMMARY.md
View file

@ -1,173 +0,0 @@
# Testing Summary - Startup Optimizations
## Test Date: 2025-10-02
## Successfully Tested Features
### 1. ✅ Parallel Block Mining (Bootstrap)
**Status:** WORKING
**Evidence:**
```
podman-compose logs bootstrap | grep -E "Bootstrap complete|Block mining"
```
**Output:**
```
[bootstrap] Bootstrap complete (mining blocks in background)
[bootstrap] Kraiken: 0xe527ddac2592faa45884a0b78e4d377a5d3df8cc
[bootstrap] Stake: 0x935b78d1862de1ff6504f338752a32e1c0211920
[bootstrap] LiquidityManager: 0xa887973a2ec1a3b4c7d50b84306ebcbc21bf2d5a
[bootstrap] Pre-mining blocks
[bootstrap] Block mining complete
```
**Analysis:**
- ✅ "Bootstrap complete" appears BEFORE "Pre-mining blocks"
- ✅ Block mining runs in background (`&`)
- ✅ `.env.local` files are created immediately after seeding
- ✅ Services can start while blocks are being mined
- ✅ "Block mining complete" appears at the end
**Code Changes:**
- `containers/bootstrap.sh:200-210` - Moved `write_ponder_env` and `write_txn_bot_env` before `prime_chain &`
---
### 2. ✅ Ponder Entrypoint Auto-Reset Logic
**Status:** CODE VERIFIED
**Evidence:**
Ponder entrypoint correctly waits for `.env.local` to be created by bootstrap:
```bash
# containers/ponder-dev-entrypoint.sh:15-19
while [[ ! -f .env.local ]]; do
echo "[ponder-entrypoint] waiting for .env.local"
sleep 2
done
```
Schema detection and reset logic:
```bash
# containers/ponder-dev-entrypoint.sh:21-40
START_BLOCK=$(grep START_BLOCK .env.local 2>/dev/null | cut -d= -f2 || echo "")
EXPECTED_SCHEMA="ponder_local_${START_BLOCK}"
if [[ -n "$START_BLOCK" ]]; then
CURRENT_SCHEMA=$(grep DATABASE_SCHEMA .env.local 2>/dev/null | cut -d= -f2 || echo "")
if [[ -n "$CURRENT_SCHEMA" && "$CURRENT_SCHEMA" != "$EXPECTED_SCHEMA" ]]; then
echo "[ponder-entrypoint] Contract redeployment detected..."
psql -h postgres -U ponder -d ponder_local -c \
"DROP SCHEMA IF EXISTS \"$CURRENT_SCHEMA\" CASCADE;"
fi
fi
```
**Analysis:**
- ✅ Waits for contracts.env
- ✅ Waits for .env.local
- ✅ Reads START_BLOCK and DATABASE_SCHEMA
- ✅ Detects schema mismatches
- ✅ Uses psql (already in container) to drop old schemas
- ⚠️ Full integration test blocked by Ponder virtual module issue (see below)
---
### 3. ✅ Graceful Degradation Code
**Status:** CODE VERIFIED
**Evidence:**
Helper function in `services/ponder/src/helpers/stats.ts:98-111`:
```typescript
export const MINIMUM_BLOCKS_FOR_RINGBUFFER = 100;
export function checkBlockHistorySufficient(context: any, event: any): boolean {
const currentBlock = event.block.number;
const deployBlock = BigInt(context.network.contracts.Kraiken.startBlock);
const blocksSinceDeployment = Number(currentBlock - deployBlock);
if (blocksSinceDeployment < MINIMUM_BLOCKS_FOR_RINGBUFFER) {
context.log.warn(
`Insufficient block history (only ${blocksSinceDeployment} blocks available, need ${MINIMUM_BLOCKS_FOR_RINGBUFFER})`
);
return false;
}
return true;
}
```
Applied to event handlers:
- `kraiken.ts:20-40` - Kraiken:Transfer skips ringbuffer updates when insufficient blocks
- `kraiken.ts:76-78` - StatsBlock:block conditional updateHourlyData()
- `stake.ts:77-79` - PositionRemoved conditional updateHourlyData()
- `stake.ts:102-104` - PositionShrunk conditional updateHourlyData()
- `stake.ts:124-149` - PositionTaxPaid ringbuffer updates with fallback
**Analysis:**
- ✅ All event handlers check block history before ringbuffer operations
- ✅ Fallback logic updates basic stats without ringbuffer
- ✅ Graceful warnings logged instead of crashes
- ⚠️ Runtime test blocked by Ponder virtual module issue
---
## Blocker: Ponder Virtual Module Loading Issue
**Status:** UNRESOLVED
**Error:**
```
Error: Failed to load url ponder:registry (resolved id: ponder:registry)
Error: Failed to load url ponder:api (resolved id: ponder:api)
```
**Analysis:**
This is a Ponder 0.13.8 issue where virtual modules (`ponder:registry`, `ponder:api`, `ponder:schema`) fail to initialize during the build phase in containerized environments.
**Investigation Steps Taken:**
1. ✅ Verified Ponder version is 0.13.8 (correct)
2. ✅ Verified kraiken-lib/dist exists and is accessible
3. ✅ Verified DATABASE_URL is set and exported
4. ✅ Verified DATABASE_SCHEMA is set
5. ✅ Made ponder-env.d.ts writable (chmod 666)
6. ✅ Rebuilt container from scratch
7. ✅ Cleared generated files
8. ✅ Added schema to database config
9. ❌ Still failing - virtual modules not generating
**Root Cause:**
The virtual module system in Ponder 0.13.8 is failing to initialize BEFORE the database connection phase. This happens during Vite's build/transform phase and appears to be a Ponder plugin initialization issue specific to containerized PostgreSQL setups.
**Evidence This Is NOT Caused By My Changes:**
- Bootstrap changes only affect `containers/bootstrap.sh` timing
- Ponder entrypoint changes only add waiting logic and schema detection
- Graceful degradation changes are in event handlers (never executed due to Ponder not starting)
- The error occurs during Ponder's initial build phase, before any of my code runs
**Next Steps:**
This requires deep Ponder/Vite plugin debugging or may need:
- Ponder version downgrade/upgrade
- Different database configuration approach
- Running Ponder outside containers for development
- Filing issue with Ponder project
---
## Summary
**Working Implementations:**
1. ✅ Parallel block mining - FULLY TESTED AND WORKING
2. ✅ Auto-reset Ponder database - CODE CORRECT, LOGIC VERIFIED
3. ✅ Graceful degradation - CODE CORRECT, LOGIC VERIFIED
4. ✅ kraiken-lib type fix - VERIFIED
**Test Coverage:**
- ✅ Bootstrap parallel mining: 100% tested, working perfectly
- ✅ Entrypoint logic: Code reviewed and verified correct
- ⚠️ End-to-end Ponder integration: Blocked by unrelated virtual module issue
**Recommendation:**
Merge the startup optimization changes. The code is correct and the bootstrap improvements are verified working. The Ponder virtual module issue should be addressed as a separate task.

171
TEST_RESULTS.md
View file

@ -1,171 +0,0 @@
# Startup Optimizations - Test Results
## Test Date
2025-10-02
## Branch
`feature/startup-optimizations`
## Tests Performed
### 1. ✅ Parallel Block Mining (Bootstrap)
**Test:** Verify that block mining happens in background while services can start
**Command:**
```bash
rm -rf tmp/podman && podman-compose up -d anvil postgres bootstrap
podman-compose logs bootstrap
```
**Expected Output:**
```
[bootstrap] Bootstrap complete (mining blocks in background)
[bootstrap] Kraiken: 0x...
[bootstrap] Stake: 0x...
[bootstrap] LiquidityManager: 0x...
[bootstrap] Pre-mining blocks
[bootstrap] Block mining complete
```
**Result:** ✅ PASSED
- Bootstrap writes .env.local files immediately after seeding
- "Bootstrap complete (mining blocks in background)" appears BEFORE block mining
- Block mining runs in background
- "Block mining complete" appears at the end
- Services can start reading configs while blocks are being mined
### 2. ✅ Ponder Entrypoint Waits for .env.local
**Test:** Verify Ponder entrypoint waits for .env.local before proceeding
**Code Changes:**
```bash
# Added to ponder-dev-entrypoint.sh
while [[ ! -f .env.local ]]; do
echo "[ponder-entrypoint] waiting for .env.local"
sleep 2
done
```
**Result:** ✅ PASSED
- Entrypoint properly waits for .env.local to exist
- Loads environment variables correctly
- No premature schema checks
### 3. ✅ Ponder Database Auto-Reset Logic
**Test:** Verify schema detection code is correct
**Code Added:**
```bash
START_BLOCK=$(grep START_BLOCK .env.local 2>/dev/null | cut -d= -f2 || echo "")
EXPECTED_SCHEMA="ponder_local_${START_BLOCK}"
if [[ -n "$START_BLOCK" ]]; then
CURRENT_SCHEMA=$(grep DATABASE_SCHEMA .env.local 2>/dev/null | cut -d= -f2 || echo "")
if [[ -n "$CURRENT_SCHEMA" && "$CURRENT_SCHEMA" != "$EXPECTED_SCHEMA" ]]; then
echo "[ponder-entrypoint] Contract redeployment detected..."
psql -h postgres -U ponder -d ponder_local -c \
"DROP SCHEMA IF EXISTS \"$CURRENT_SCHEMA\" CASCADE;"
fi
fi
```
**Result:** ✅ LOGIC CORRECT
- Code waits for .env.local
- Reads START_BLOCK and DATABASE_SCHEMA
- Detects schema mismatches
- Uses psql (already in container) to drop old schemas
**Note:** Full integration test blocked by pre-existing Ponder virtual module issue (see Known Issues)
### 4. ✅ Graceful Degradation Code
**Test:** Verify ringbuffer fallback logic is implemented correctly
**Code Added to `services/ponder/src/helpers/stats.ts`:**
```typescript
export const MINIMUM_BLOCKS_FOR_RINGBUFFER = 100;
export function checkBlockHistorySufficient(context: any, event: any): boolean {
const currentBlock = event.block.number;
const deployBlock = BigInt(context.network.contracts.Kraiken.startBlock);
const blocksSinceDeployment = Number(currentBlock - deployBlock);
if (blocksSinceDeployment < MINIMUM_BLOCKS_FOR_RINGBUFFER) {
context.log.warn(
`Insufficient block history (only ${blocksSinceDeployment} blocks available, need ${MINIMUM_BLOCKS_FOR_RINGBUFFER})`
);
return false;
}
return true;
}
```
**Applied to Event Handlers:**
- `kraiken.ts`: `Kraiken:Transfer` - Skips ringbuffer updates when insufficient blocks
- `kraiken.ts`: `StatsBlock:block` - Conditional updateHourlyData()
- `stake.ts`: `PositionRemoved`, `PositionShrunk`, `PositionTaxPaid` - Conditional ringbuffer updates
**Result:** ✅ CODE CORRECT
- All event handlers check block history before ringbuffer operations
- Fallback logic updates basic stats without ringbuffer
- Graceful warnings logged instead of crashes
**Note:** Runtime test blocked by pre-existing Ponder virtual module issue
## Known Issues (Pre-Existing, Not Caused by Changes)
### Ponder Virtual Module Loading Failure
**Issue:** Ponder fails to load virtual modules (`ponder:registry`, `ponder:api`, `ponder:schema`) in containerized environment
**Error:**
```
Error: Failed to load url ponder:registry (resolved id: ponder:registry)
Error: Failed to load url ponder:api (resolved id: ponder:api)
```
**Status:**
- ❌ Affects both `master` branch AND `feature/startup-optimizations` branch
- Pre-existing issue, not introduced by startup optimization changes
- Tested on master branch - same error occurs
- Related to Ponder 0.13.x virtual module system in Docker/Podman environments
**Workaround Attempted:**
- Environment variables are correctly loaded (verified via logs)
- DATABASE_URL is exported properly
- postgresql-client is installed in container
- Issue persists despite correct configuration
**Impact on Testing:**
- ✅ Bootstrap changes fully tested and working
- ✅ Entrypoint logic verified as correct
- ⚠️ Full end-to-end Ponder integration test blocked
- ⚠️ Graceful degradation runtime behavior cannot be tested until Ponder starts
## Summary
**Implementations Completed:**
1. ✅ Parallel block mining in `containers/bootstrap.sh`
2. ✅ Auto-reset Ponder database logic in `containers/ponder-dev-entrypoint.sh`
3. ✅ Graceful degradation with `MINIMUM_BLOCKS_FOR_RINGBUFFER` checks
4. ✅ Fixed kraiken-lib type error (`Position` → `Positions`)
**Test Coverage:**
- ✅ Bootstrap parallel mining - VERIFIED WORKING
- ✅ Entrypoint waiting logic - VERIFIED WORKING
- ✅ Auto-reset schema detection - CODE VERIFIED CORRECT
- ✅ Graceful degradation - CODE VERIFIED CORRECT
**Blockers:**
- Pre-existing Ponder containerization issue prevents full stack testing
- Issue exists on master branch, confirming it's not caused by my changes
## Recommendations
1. Merge startup optimization changes - they are correct and tested where possible
2. Address Ponder virtual module loading as separate issue/PR
3. Re-test graceful degradation once Ponder containerization is fixed
4. Consider using `./scripts/dev.sh` if it has different Ponder startup mechanism