harb/docs/ci-pipeline.md

# Woodpecker CI

## Infrastructure
- **Server**: Woodpecker 3.10.0 runs as a **systemd service** (`woodpecker-server.service`), NOT a Docker container. Binary at `/usr/local/bin/woodpecker-server`.
- **Host**: `https://ci.sovraigns.network` (port 8000 locally at `http://127.0.0.1:8000`)
- **Forge**: Codeberg (Gitea-compatible) — repo `johba/harb`, forge remote ID `800173`
- **Database**: PostgreSQL at `127.0.0.1:5432`, database `woodpecker`, user `woodpecker`
- **Config**: `/etc/woodpecker/server.env` (contains secrets — agent secret, Gitea OAuth secret, DB credentials)
- **CLI**: Downloaded to `/tmp/woodpecker-cli` (v3.10.0). Requires `WOODPECKER_SERVER` and `WOODPECKER_TOKEN` env vars.
- **Logs**: `journalctl -u woodpecker-server -f` (NOT `docker logs`)

## Pipeline Configs
- `.woodpecker/build-ci-images.yml` — Builds Docker CI images using unified `docker/Dockerfile.service-ci`. Triggers on **push** to `master` or `feature/ci` when files in `docker/`, `.woodpecker/`, `containers/`, `kraiken-lib/`, `onchain/`, `services/`, `web-app/`, or `landing/` change.
- `.woodpecker/e2e.yml` — Runs Playwright E2E tests. Bootstrap step sources `scripts/bootstrap-common.sh` for shared deploy/seed logic. Health checks use `scripts/wait-for-service.sh`. Triggers on **pull_request** to `master`.
- Pipeline numbering: even = build-ci-images (push events), odd = E2E (pull_request events). This is not guaranteed but was the observed pattern.

## Monitoring Pipelines via DB
Since the Woodpecker API requires authentication (tokens are cached in server memory; DB-only token changes don't work without a server restart), monitor pipelines directly via PostgreSQL:
```bash
# Latest pipelines
PGPASSWORD='<db_password>' psql -h 127.0.0.1 -U woodpecker -d woodpecker -c \
  "SELECT number, status, branch, event, commit FROM pipelines
   WHERE repo_id = (SELECT id FROM repos WHERE full_name = 'johba/harb')
   ORDER BY number DESC LIMIT 5;"

# Step details for a specific pipeline
PGPASSWORD='<db_password>' psql -h 127.0.0.1 -U woodpecker -d woodpecker -c \
  "SELECT s.name, s.state,
    CASE WHEN s.finished > 0 AND s.started > 0 THEN (s.finished - s.started)::int::text || 's'
    ELSE '-' END as duration, s.exit_code
   FROM steps s WHERE s.pipeline_id = (
     SELECT id FROM pipelines WHERE number = <N>
     AND repo_id = (SELECT id FROM repos WHERE full_name = 'johba/harb'))
   ORDER BY s.started NULLS LAST;"
```

## Triggering Pipelines
- **Normal flow**: Push to Codeberg → Codeberg fires webhook to `https://ci.sovraigns.network/api/hook` → Woodpecker creates pipeline.
- **Known issue**: Codeberg webhooks can stop firing if `ci.sovraigns.network` becomes unreachable (DNS/connectivity). Check Codeberg repo settings → Webhooks to verify delivery history and re-trigger.
- **Manual trigger via API** (requires valid token — see known issues):
  ```bash
  WOODPECKER_SERVER=http://127.0.0.1:8000 WOODPECKER_TOKEN=<token> \
    /tmp/woodpecker-cli pipeline create --branch feature/ci johba/harb
  ```
- **API auth limitation**: The server caches user token hashes in memory. Inserting a token directly into the DB does not work without restarting the server (`sudo systemctl restart woodpecker-server`).

## CI Docker Images
- `docker/Dockerfile.service-ci` — Unified parameterized Dockerfile for all service CI images (ponder, webapp, landing, txnBot). Uses `--build-arg` for service-specific configuration (SERVICE_DIR, SERVICE_PORT, ENTRYPOINT_SCRIPT, NEEDS_SYMLINKS, etc.).
  - **sync-tax-rates**: Builder stage runs `scripts/sync-tax-rates.mjs` to sync tax rates from `Stake.sol` into kraiken-lib before TypeScript compilation.
  - **Symlinks fix** (webapp only, `NEEDS_SYMLINKS=true`): Creates `/web-app`, `/kraiken-lib`, `/onchain` symlinks to work around Vite's `removeBase()` stripping `/app/` prefix from filesystem paths.
  - **CI env detection** (`CI=true`): Disables Vue DevTools plugin in `vite.config.ts` to prevent 500 errors caused by path resolution issues with `/app/` base path.
  - **HEALTHCHECK**: Configurable via build args; webapp uses `--retries=84 --interval=5s` = 420s (7 min), aligned with `wait-for-stack` step timeout.
- **Shared entrypoints**: Each service uses a unified entrypoint script (`containers/<service>-entrypoint.sh`) that branches on `CI=true` env var for CI vs local dev paths. Common helpers in `containers/entrypoint-common.sh`.
- **Shared bootstrap**: `scripts/bootstrap-common.sh` contains shared contract deployment, seeding, and funding functions used by both `containers/bootstrap.sh` (local dev) and `.woodpecker/e2e.yml` (CI).
- CI images are tagged with git SHA and `latest`, pushed to a local registry.

## CI Agent & Registry Auth
- **Agent**: Runs as user `ci` (uid 1001) on `harb-staging`, same host as the dev environment. Binary at `/usr/local/bin/woodpecker-agent`.
- **Registry credentials**: The `ci` user must have Docker auth configured at `/home/ci/.docker/config.json` to pull private images from `registry.niovi.voyage`. If images fail to pull with "no basic auth credentials", fix with:
  ```bash
  sudo mkdir -p /home/ci/.docker
  sudo cp /home/debian/.docker/config.json /home/ci/.docker/config.json
  sudo chown -R ci:ci /home/ci/.docker
  sudo chmod 600 /home/ci/.docker/config.json
  ```
- **Shared Docker daemon**: The `ci` and `debian` users share the same Docker daemon. Running `docker system prune` as `debian` removes images cached for CI pipelines. If CI image pulls fail after a prune, either fix registry auth (above) or pre-pull images as `debian`: `docker pull registry.niovi.voyage/harb/ponder-ci:latest` etc.

## Debugging Tips
- If pipelines aren't being created after a push, check Codeberg webhook delivery logs first.
- The Woodpecker server needs `sudo` to restart. Without it, you cannot: refresh API tokens, clear cached state, or recover from webhook auth issues.
- E2E pipeline failures often come from `wait-for-stack` timing out. Check the webapp HEALTHCHECK alignment and Ponder indexing time.
- The `web-app/vite.config.ts` `allowedHosts` array must include container hostnames (`webapp`, `caddy`) for health checks to succeed inside Docker networks.
- **Never use `bash -lc`** in Woodpecker pipeline commands — login shell resets PATH via `/etc/profile`, losing Foundry and other tools set by Docker ENV. Use `bash -c` instead.
fix: refactor AGENTS.md into progressive-disclosure structure (#184) - Root AGENTS.md: 350+ lines → 68 lines (map, not encyclopedia) - New docs/dev-environment.md (67 lines): Docker, dev.sh, ports, pitfalls - New docs/ci-pipeline.md (73 lines): Woodpecker setup, monitoring, debugging - New docs/testing.md (41 lines): Foundry, E2E, version validation - New docs/codeberg-api.md (32 lines): .netrc auth, API usage - Updated stale model refs in .claude-code-supervisor.yml files - Sub-component AGENTS.md files unchanged - Context docs (PRODUCT-TRUTH, ARCHITECTURE, UX-DECISIONS) unchanged 2026-02-23 09:46:35 +00:00			`# Woodpecker CI`

			`## Infrastructure`
			- Server: Woodpecker 3.10.0 runs as a systemd service (`woodpecker-server.service`), NOT a Docker container. Binary at `/usr/local/bin/woodpecker-server`.
			- Host: `https://ci.sovraigns.network` (port 8000 locally at `http://127.0.0.1:8000`)
			- Forge: Codeberg (Gitea-compatible) — repo `johba/harb`, forge remote ID `800173`
			- Database: PostgreSQL at `127.0.0.1:5432`, database `woodpecker`, user `woodpecker`
			- Config: `/etc/woodpecker/server.env` (contains secrets — agent secret, Gitea OAuth secret, DB credentials)
			- CLI: Downloaded to `/tmp/woodpecker-cli` (v3.10.0). Requires `WOODPECKER_SERVER` and `WOODPECKER_TOKEN` env vars.
			- Logs: `journalctl -u woodpecker-server -f` (NOT `docker logs`)

			`## Pipeline Configs`
			- `.woodpecker/build-ci-images.yml` — Builds Docker CI images using unified `docker/Dockerfile.service-ci`. Triggers on push to `master` or `feature/ci` when files in `docker/`, `.woodpecker/`, `containers/`, `kraiken-lib/`, `onchain/`, `services/`, `web-app/`, or `landing/` change.
			- `.woodpecker/e2e.yml` — Runs Playwright E2E tests. Bootstrap step sources `scripts/bootstrap-common.sh` for shared deploy/seed logic. Health checks use `scripts/wait-for-service.sh`. Triggers on pull_request to `master`.
			`- Pipeline numbering: even = build-ci-images (push events), odd = E2E (pull_request events). This is not guaranteed but was the observed pattern.`

			`## Monitoring Pipelines via DB`
			`Since the Woodpecker API requires authentication (tokens are cached in server memory; DB-only token changes don't work without a server restart), monitor pipelines directly via PostgreSQL:`
			```bash
			`# Latest pipelines`
			`PGPASSWORD='<db_password>' psql -h 127.0.0.1 -U woodpecker -d woodpecker -c \`
			`"SELECT number, status, branch, event, commit FROM pipelines`
			`WHERE repo_id = (SELECT id FROM repos WHERE full_name = 'johba/harb')`
			`ORDER BY number DESC LIMIT 5;"`

			`# Step details for a specific pipeline`
			`PGPASSWORD='<db_password>' psql -h 127.0.0.1 -U woodpecker -d woodpecker -c \`
			`"SELECT s.name, s.state,`
			`CASE WHEN s.finished > 0 AND s.started > 0 THEN (s.finished - s.started)::int::text \|\| 's'`
			`ELSE '-' END as duration, s.exit_code`
			`FROM steps s WHERE s.pipeline_id = (`
			`SELECT id FROM pipelines WHERE number = <N>`
			`AND repo_id = (SELECT id FROM repos WHERE full_name = 'johba/harb'))`
			`ORDER BY s.started NULLS LAST;"`
			```

			`## Triggering Pipelines`
			- Normal flow: Push to Codeberg → Codeberg fires webhook to `https://ci.sovraigns.network/api/hook` → Woodpecker creates pipeline.
			- Known issue: Codeberg webhooks can stop firing if `ci.sovraigns.network` becomes unreachable (DNS/connectivity). Check Codeberg repo settings → Webhooks to verify delivery history and re-trigger.
			`- Manual trigger via API (requires valid token — see known issues):`
			```bash
			`WOODPECKER_SERVER=http://127.0.0.1:8000 WOODPECKER_TOKEN=<token> \`
			`/tmp/woodpecker-cli pipeline create --branch feature/ci johba/harb`
			```
			- API auth limitation: The server caches user token hashes in memory. Inserting a token directly into the DB does not work without restarting the server (`sudo systemctl restart woodpecker-server`).

			`## CI Docker Images`
			- `docker/Dockerfile.service-ci` — Unified parameterized Dockerfile for all service CI images (ponder, webapp, landing, txnBot). Uses `--build-arg` for service-specific configuration (SERVICE_DIR, SERVICE_PORT, ENTRYPOINT_SCRIPT, NEEDS_SYMLINKS, etc.).
			- sync-tax-rates: Builder stage runs `scripts/sync-tax-rates.mjs` to sync tax rates from `Stake.sol` into kraiken-lib before TypeScript compilation.
			- Symlinks fix (webapp only, `NEEDS_SYMLINKS=true`): Creates `/web-app`, `/kraiken-lib`, `/onchain` symlinks to work around Vite's `removeBase()` stripping `/app/` prefix from filesystem paths.
			- CI env detection (`CI=true`): Disables Vue DevTools plugin in `vite.config.ts` to prevent 500 errors caused by path resolution issues with `/app/` base path.
			- HEALTHCHECK: Configurable via build args; webapp uses `--retries=84 --interval=5s` = 420s (7 min), aligned with `wait-for-stack` step timeout.
			- Shared entrypoints: Each service uses a unified entrypoint script (`containers/<service>-entrypoint.sh`) that branches on `CI=true` env var for CI vs local dev paths. Common helpers in `containers/entrypoint-common.sh`.
			- Shared bootstrap: `scripts/bootstrap-common.sh` contains shared contract deployment, seeding, and funding functions used by both `containers/bootstrap.sh` (local dev) and `.woodpecker/e2e.yml` (CI).
			- CI images are tagged with git SHA and `latest`, pushed to a local registry.

			`## CI Agent & Registry Auth`
			- Agent: Runs as user `ci` (uid 1001) on `harb-staging`, same host as the dev environment. Binary at `/usr/local/bin/woodpecker-agent`.
			- Registry credentials: The `ci` user must have Docker auth configured at `/home/ci/.docker/config.json` to pull private images from `registry.niovi.voyage`. If images fail to pull with "no basic auth credentials", fix with:
			```bash
			`sudo mkdir -p /home/ci/.docker`
			`sudo cp /home/debian/.docker/config.json /home/ci/.docker/config.json`
			`sudo chown -R ci:ci /home/ci/.docker`
			`sudo chmod 600 /home/ci/.docker/config.json`
			```
			- Shared Docker daemon: The `ci` and `debian` users share the same Docker daemon. Running `docker system prune` as `debian` removes images cached for CI pipelines. If CI image pulls fail after a prune, either fix registry auth (above) or pre-pull images as `debian`: `docker pull registry.niovi.voyage/harb/ponder-ci:latest` etc.

			`## Debugging Tips`
			`- If pipelines aren't being created after a push, check Codeberg webhook delivery logs first.`
			- The Woodpecker server needs `sudo` to restart. Without it, you cannot: refresh API tokens, clear cached state, or recover from webhook auth issues.
			- E2E pipeline failures often come from `wait-for-stack` timing out. Check the webapp HEALTHCHECK alignment and Ponder indexing time.
			- The `web-app/vite.config.ts` `allowedHosts` array must include container hostnames (`webapp`, `caddy`) for health checks to succeed inside Docker networks.
			- Never use `bash -lc` in Woodpecker pipeline commands — login shell resets PATH via `/etc/profile`, losing Foundry and other tools set by Docker ENV. Use `bash -c` instead.