feature/ci (#84)

Co-authored-by: openhands <openhands@all-hands.dev>
Reviewed-on: https://codeberg.org/johba/harb/pulls/84

AGENTS.md

@@ -86,6 +86,65 @@
 - `curl -X POST http://localhost:8081/api/graphql -d '{"query":"{ stats(id:\"0x01\"){kraikenTotalSupply}}"}'`
 - `curl http://localhost:8081/api/txn/status`
 
+## Woodpecker CI
+
+### Infrastructure
+- **Server**: Woodpecker 3.10.0 runs as a **systemd service** (`woodpecker-server.service`), NOT a Docker container. Binary at `/usr/local/bin/woodpecker-server`.
+- **Host**: `https://ci.sovraigns.network` (port 8000 locally at `http://127.0.0.1:8000`)
+- **Forge**: Codeberg (Gitea-compatible) — repo `johba/harb`, forge remote ID `800173`
+- **Database**: PostgreSQL at `127.0.0.1:5432`, database `woodpecker`, user `woodpecker`
+- **Config**: `/etc/woodpecker/server.env` (contains secrets — agent secret, Gitea OAuth secret, DB credentials)
+- **CLI**: Downloaded to `/tmp/woodpecker-cli` (v3.10.0). Requires `WOODPECKER_SERVER` and `WOODPECKER_TOKEN` env vars.
+- **Logs**: `journalctl -u woodpecker-server -f` (NOT `docker logs`)
+
+### Pipeline Configs
+- `.woodpecker/build-ci-images.yml` — Builds Docker CI images. Triggers on **push** to `master` or `feature/ci` when files in `docker/`, `.woodpecker/`, `kraiken-lib/`, `onchain/out/`, or `web-app/` change.
+- `.woodpecker/e2e.yml` — Runs Playwright E2E tests. Triggers on **pull_request** to `master`.
+- Pipeline numbering: even = build-ci-images (push events), odd = E2E (pull_request events). This is not guaranteed but was the observed pattern.
+
+### Monitoring Pipelines via DB
+Since the Woodpecker API requires authentication (tokens are cached in server memory; DB-only token changes don't work without a server restart), monitor pipelines directly via PostgreSQL:
+```bash
+# Latest pipelines
+PGPASSWORD='<db_password>' psql -h 127.0.0.1 -U woodpecker -d woodpecker -c \
+  "SELECT number, status, branch, event, commit FROM pipelines
+   WHERE repo_id = (SELECT id FROM repos WHERE full_name = 'johba/harb')
+   ORDER BY number DESC LIMIT 5;"
+
+# Step details for a specific pipeline
+PGPASSWORD='<db_password>' psql -h 127.0.0.1 -U woodpecker -d woodpecker -c \
+  "SELECT s.name, s.state,
+    CASE WHEN s.finished > 0 AND s.started > 0 THEN (s.finished - s.started)::int::text || 's'
+    ELSE '-' END as duration, s.exit_code
+   FROM steps s WHERE s.pipeline_id = (
+     SELECT id FROM pipelines WHERE number = <N>
+     AND repo_id = (SELECT id FROM repos WHERE full_name = 'johba/harb'))
+   ORDER BY s.started NULLS LAST;"
+```
+
+### Triggering Pipelines
+- **Normal flow**: Push to Codeberg → Codeberg fires webhook to `https://ci.sovraigns.network/api/hook` → Woodpecker creates pipeline.
+- **Known issue**: Codeberg webhooks can stop firing if `ci.sovraigns.network` becomes unreachable (DNS/connectivity). Check Codeberg repo settings → Webhooks to verify delivery history and re-trigger.
+- **Manual trigger via API** (requires valid token — see known issues):
+  ```bash
+  WOODPECKER_SERVER=http://127.0.0.1:8000 WOODPECKER_TOKEN=<token> \
+    /tmp/woodpecker-cli pipeline create --branch feature/ci johba/harb
+  ```
+- **API auth limitation**: The server caches user token hashes in memory. Inserting a token directly into the DB does not work without restarting the server (`sudo systemctl restart woodpecker-server`).
+
+### CI Docker Images
+- `docker/Dockerfile.webapp-ci` — Webapp CI image with Vite dev server.
+  - **Symlinks fix** (lines 57-59): Creates `/web-app`, `/kraiken-lib`, `/onchain` symlinks to work around Vite's `removeBase()` stripping `/app/` prefix from filesystem paths.
+  - **CI env detection** (`CI=true`): Disables Vue DevTools plugin in `vite.config.ts` to prevent 500 errors caused by path resolution issues with `/app/` base path.
+  - **HEALTHCHECK**: `--retries=84 --interval=5s` = 420s (7 min) total wait, aligned with `wait-for-stack` step timeout.
+- CI images are tagged with git SHA and `latest`, pushed to a local registry.
+
+### CI Debugging Tips
+- If pipelines aren't being created after a push, check Codeberg webhook delivery logs first.
+- The Woodpecker server needs `sudo` to restart. Without it, you cannot: refresh API tokens, clear cached state, or recover from webhook auth issues.
+- E2E pipeline failures often come from `wait-for-stack` timing out. Check the webapp HEALTHCHECK alignment and Ponder indexing time.
+- The `web-app/vite.config.ts` `allowedHosts` array must include container hostnames (`webapp`, `caddy`) for health checks to succeed inside Docker networks.
+
 ## References
 - Deployment history: `onchain/deployments-local.json`, `onchain/broadcast/`.
 - Deep dives: `TECHNICAL_APPENDIX.md`, `HARBERG.md`, and `onchain/UNISWAP_V3_MATH.md`.